Before adding information from this 'benchmark' to the wiki it'd probably be worth considering whether it actually tells users anything of use.
As far as I can tell the benchmark creates a new virtual interface to allow a wireguard tunnel to be set up entirely within the single device. It then uses iperf3 to measure data transfer over that tunnel. This would appear to add a significant amount of overhead that wouldn't exist in a real world setup. There doesn't appear to be anything that tries to take that into account or normalise the results to make them comparable in any meaningful way.
I am not going to say this is an "accurate measurement", but a kind of order of magnitude to compare their encryption power, as the GitHub page mentioned: It doesn't take account into something like network driver (that's why you can see RPi 3 can have a speed faster than the max. of USB2 bus speed), however this can help some users to have some numbers that can be used to compared between devices.
And frankly speaking, trying to get real world numbers for all devices is not possible so that's why I am thinking of this "kind of simplified" way.
One very important downside is that it runs iperf3 client and server and severly affects performance of single core mips and legacy arm (like rpi1) most likely more than network driver stack
And during my tests against MT7620/MT7628 devices, due to very little system resources the test itself often crashes! Also in real world usage those routers can't really handle too much work, and that Wireguard + iperf setup is over the limit already, so yeah those devices should be simple router only.
For RPi 1B, it doesn't crash, however in previous few weeks when I was trying to plug a USB WiFi to make it simple OpenWrt AP I already noticed that CPU usage also kind of high, the VPN setup didn't crash but I strongly believe it has serious performance hit to affect overall result. Well......who is going to use this 2012 Pi for OpenWrt now? lol......
I hate to go out of topic but there are still scenarios where can be usefull if it's already around like you said some portable AP or iot vpn gw where performance is not important etc
But that's not what the benchmark appears to be measuring. It's measuring a number of metrics, some of which are relevant to WG performance in a real setup and some which are less (or not at all) relevant. Without any way to separate those metrics how are you fairly comparing devices WG performance against each other?
You can already see in this relatively short thread that there are several 'inconsistencies'. That in itself should be sufficient to render this as a fun thing for people to try out, but rule it out as any sort of meaningful performance marker.
I'd argue it's so 'simplified' it's not measuring the thing you're actually interested in.
mt7981 ~3.5 times faster than mt7621 in wireguard;
mt7986 2 times faster than mt7981.
And it’s great that now we have rough estimation which we can use choosing a router. If you can suggest an equally simple, but more accurate test method, please share it.
How are you determining correct? Do you get the same results if the router is only acting as one end of the tunnel and isn't involved in generating any of the traffic? What about other CPUs/SoCs? Does the performance differential measured by this test remain the same in a real world situation across all devices?
I can't. Because doing it properly involves removing as many variables as possible from the test. So you need access to several other devices that are plenty powerful enough to not create any bottlenecks during testing.
However, just because it's complicated/resource intensive to do it properly doesn't mean we should start adding unverified numbers, which can't actually be shown to be comparable across devices, to the wiki. Or offering them out to other users as something to consider when making purchasing decisions.
I have mt7981, mt7986, mt7621 devices. Therefore, I know approximately how much faster which one is in wireguard.
No. This test should be considered only as wireguard performance index.
No test can guarantee this. With pppoe / without pppoe, with wifi / without wifi, different mtu, ipv4 / ipv6, NAT, internet speed, offload settings etc. These factors will influence the results. You can check "wireguard speed" field in ToH across different devices. We may notice very strange and irrelevant results possibly due to factors mentioned above. Test proposed by @fakemanhk minimizes the influence of external factors.
If you don't get the same results (not necessarily speeds, but performance differentials) as when the devices are running wireguard in the real world then the test is useless as any sort of performance indicator.
Of course not, but that's why you do controlled testing. As I said, it's complicated/resource intensive to do it properly.
And then introduces a whole load of different factors that can impact just as much, but in ways that are much more difficult to pinpoint as the 'issue'. For example, how much impact is packet generation having on the result across different cpu/soc architecture and speeds? It's not going to impact all devices equally or be easily ascertainable as to how much it effects the headline performance figure. It's things like this (and other factors) that mean serious questions should be asked about the real world usefulness of this 'benchmark' and how much merit should be given to it's outputs. Especially if there's a suggestion they should be put in a place where they might impact on purchases people make.
As I said earlier, it looks like a fun script for people to run on their devices if they want (and if they want to also share it then fair enough), but it appears to be a long way from a tool that should be given the perception of OpenWRT approval (which is what people will think if results end up next to devices on the wiki).
Ok, this is your personal opinion. Useless for you doesn't mean useless for everyone. You can just ignore such results. Let everyone decide useful or not on one's own.
Hey.....you are still on a very old version firmware? Now even OpenWrt stable release for this one is on 23.05.2 already, not to mention snapshot, will you upgrade and try again?
Like csharper2005 above said, this is stock GLi.net firmware and it's completely up to date. I understand it's not up to date with openwrt. It is what it is.