A Wireguard comparison DB

you can manually add it to every device, or perhaps create a wiki page.

Before adding information from this 'benchmark' to the wiki it'd probably be worth considering whether it actually tells users anything of use.

As far as I can tell the benchmark creates a new virtual interface to allow a wireguard tunnel to be set up entirely within the single device. It then uses iperf3 to measure data transfer over that tunnel. This would appear to add a significant amount of overhead that wouldn't exist in a real world setup. There doesn't appear to be anything that tries to take that into account or normalise the results to make them comparable in any meaningful way.

5 Likes

I am not going to say this is an "accurate measurement", but a kind of order of magnitude to compare their encryption power, as the GitHub page mentioned: It doesn't take account into something like network driver (that's why you can see RPi 3 can have a speed faster than the max. of USB2 bus speed), however this can help some users to have some numbers that can be used to compared between devices.

And frankly speaking, trying to get real world numbers for all devices is not possible so that's why I am thinking of this "kind of simplified" way.

One very important downside is that it runs iperf3 client and server and severly affects performance of single core mips and legacy arm (like rpi1) most likely more than network driver stack

Yes I do notice that for those very old devices it's showing extraordinary low numbers.

1 Like

And during my tests against MT7620/MT7628 devices, due to very little system resources the test itself often crashes! Also in real world usage those routers can't really handle too much work, and that Wireguard + iperf setup is over the limit already, so yeah those devices should be simple router only.

For RPi 1B, it doesn't crash, however in previous few weeks when I was trying to plug a USB WiFi to make it simple OpenWrt AP I already noticed that CPU usage also kind of high, the VPN setup didn't crash but I strongly believe it has serious performance hit to affect overall result. Well......who is going to use this 2012 Pi for OpenWrt now? lol......

1 Like

I hate to go out of topic but there are still scenarios where can be usefull if it's already around like you said some portable AP or iot vpn gw where performance is not important etc

But that's not what the benchmark appears to be measuring. It's measuring a number of metrics, some of which are relevant to WG performance in a real setup and some which are less (or not at all) relevant. Without any way to separate those metrics how are you fairly comparing devices WG performance against each other?

You can already see in this relatively short thread that there are several 'inconsistencies'. That in itself should be sufficient to render this as a fun thing for people to try out, but rule it out as any sort of meaningful performance marker.

I'd argue it's so 'simplified' it's not measuring the thing you're actually interested in.

1 Like

This test gives correct results. That:

  • mt7981 ~3.5 times faster than mt7621 in wireguard;
  • mt7986 2 times faster than mt7981.

And it’s great that now we have rough estimation which we can use choosing a router. If you can suggest an equally simple, but more accurate test method, please share it.

1 Like

How are you determining correct? Do you get the same results if the router is only acting as one end of the tunnel and isn't involved in generating any of the traffic? What about other CPUs/SoCs? Does the performance differential measured by this test remain the same in a real world situation across all devices?

I can't. Because doing it properly involves removing as many variables as possible from the test. So you need access to several other devices that are plenty powerful enough to not create any bottlenecks during testing.

However, just because it's complicated/resource intensive to do it properly doesn't mean we should start adding unverified numbers, which can't actually be shown to be comparable across devices, to the wiki. Or offering them out to other users as something to consider when making purchasing decisions.

I have mt7981, mt7986, mt7621 devices. Therefore, I know approximately how much faster which one is in wireguard.

No. This test should be considered only as wireguard performance index.

No test can guarantee this. With pppoe / without pppoe, with wifi / without wifi, different mtu, ipv4 / ipv6, NAT, internet speed, offload settings etc. These factors will influence the results. You can check "wireguard speed" field in ToH across different devices. We may notice very strange and irrelevant results possibly due to factors mentioned above. Test proposed by @fakemanhk minimizes the influence of external factors.

1 Like

If you don't get the same results (not necessarily speeds, but performance differentials) as when the devices are running wireguard in the real world then the test is useless as any sort of performance indicator.

Of course not, but that's why you do controlled testing. As I said, it's complicated/resource intensive to do it properly.

And then introduces a whole load of different factors that can impact just as much, but in ways that are much more difficult to pinpoint as the 'issue'. For example, how much impact is packet generation having on the result across different cpu/soc architecture and speeds? It's not going to impact all devices equally or be easily ascertainable as to how much it effects the headline performance figure. It's things like this (and other factors) that mean serious questions should be asked about the real world usefulness of this 'benchmark' and how much merit should be given to it's outputs. Especially if there's a suggestion they should be put in a place where they might impact on purchases people make.

As I said earlier, it looks like a fun script for people to run on their devices if they want (and if they want to also share it then fair enough), but it appears to be a long way from a tool that should be given the perception of OpenWRT approval (which is what people will think if results end up next to devices on the wiki).

Do you also consider this test completely useless?

openssl speed -evp chacha20

It depends on what you're using it for. Not sure what it's got to do with the 'benchmark' being discussed in this thread though.

Ok, this is your personal opinion. Useless for you doesn't mean useless for everyone. You can just ignore such results. Let everyone decide useful or not on one's own.

I think we can stop this discussion here, just leave this post in the forum but not getting it into Wiki, then it should be fine for everyone.

2 Likes

Quite so!

| Netgear Nighthawk X4S R7800 | IPQ8065 (Dual-core ARMv7, 1.7 GHz) | 23.05.2 | 291 Mbps |

Details
 -----------------------------------------------------
 OpenWrt 23.05.2, r23630-842932a63d
 -----------------------------------------------------
root@ng7800:~# ubus call system board
{
        "kernel": "5.15.137",
        "hostname": "ng7800",
        "system": "ARMv7 Processor rev 0 (v7l)",
        "model": "Netgear Nighthawk X4S R7800",
        "board_name": "netgear,r7800",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.2",
                "revision": "r23630-842932a63d",
                "target": "ipq806x/generic",
                "description": "OpenWrt 23.05.2 r23630-842932a63d"
        }
}

root@ng7800:/tmp/wg-bench# ./benchmark.sh
Connecting to host 169.254.200.2, port 5201
[  5] local 169.254.200.1 port 41518 connected to 169.254.200.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  36.8 MBytes   308 Mbits/sec    0    806 KBytes
[  5]   1.00-2.00   sec  34.9 MBytes   293 Mbits/sec    0   1.31 MBytes
[  5]   2.00-3.00   sec  35.0 MBytes   294 Mbits/sec   45   1.10 MBytes
[  5]   3.00-4.00   sec  35.1 MBytes   294 Mbits/sec    0   1.19 MBytes
[  5]   4.00-5.00   sec  35.2 MBytes   296 Mbits/sec    0   1.26 MBytes
[  5]   5.00-6.00   sec  33.8 MBytes   283 Mbits/sec   12    844 KBytes
[  5]   6.00-7.00   sec  35.0 MBytes   294 Mbits/sec    0   1.01 MBytes
[  5]   7.00-8.00   sec  35.0 MBytes   294 Mbits/sec    0   1.06 MBytes
[  5]   8.00-9.00   sec  33.8 MBytes   283 Mbits/sec    0   1.10 MBytes
[  5]   9.00-10.00  sec  35.2 MBytes   295 Mbits/sec    0   1.11 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   350 MBytes   293 Mbits/sec   57             sender
[  5]   0.00-10.00  sec   347 MBytes   291 Mbits/sec                  receiver

iperf Done.

root@GL-MT3000:~/wg-bench# ubus call system board
{
"kernel": "5.4.211",
"hostname": "GL-MT3000",
"system": "ARMv8 Processor rev 4",
"model": "GL.iNet GL-MT3000",
"board_name": "glinet,mt3000-snand",
"release": {
"distribution": "OpenWrt",
"version": "21.02-SNAPSHOT",
"revision": "r15812+885-46b6ee7ffc",
"target": "mediatek/mt7981",
"description": "OpenWrt 21.02-SNAPSHOT r15812+885-46b6ee7ffc"
}
}
root@GL-MT3000:~/wg-bench# ./benchmark.sh
Connecting to host 169.254.200.2, port 5201
[ 5] local 169.254.200.1 port 40010 connected to 169.254.200.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 45.4 MBytes 381 Mbits/sec 0 1010 KBytes
[ 5] 1.00-2.00 sec 45.2 MBytes 379 Mbits/sec 8 906 KBytes
[ 5] 2.00-3.00 sec 43.5 MBytes 365 Mbits/sec 0 1015 KBytes
[ 5] 3.00-4.00 sec 44.1 MBytes 370 Mbits/sec 0 1.06 MBytes
[ 5] 4.00-5.00 sec 43.9 MBytes 368 Mbits/sec 0 1.12 MBytes
[ 5] 5.00-6.00 sec 44.5 MBytes 373 Mbits/sec 0 1.15 MBytes
[ 5] 6.00-7.00 sec 43.4 MBytes 364 Mbits/sec 0 1.18 MBytes
[ 5] 7.00-8.00 sec 43.4 MBytes 364 Mbits/sec 0 1.19 MBytes
[ 5] 8.00-9.00 sec 44.2 MBytes 371 Mbits/sec 0 1.19 MBytes
[ 5] 9.00-10.00 sec 44.6 MBytes 374 Mbits/sec 0 1.19 MBytes


[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 442 MBytes 371 Mbits/sec 8 sender
[ 5] 0.00-10.00 sec 440 MBytes 369 Mbits/sec receiver

iperf Done.

Hey.....you are still on a very old version firmware? Now even OpenWrt stable release for this one is on 23.05.2 already, not to mention snapshot, will you upgrade and try again?

It seems that this is GL.iNet stock firmware (based OpenWrt).