Let's talk about VPN performance

I replaced OpenVPN for WireGuard and it was huuge difference. I am not going back.
Throughput with WireGuard was x4 compared to OpenVPN. It was also much easier to set up and connection time is much quicker.

I am running N5105 x86 router and WireGuard is basically transparent up to 1Gbit as long as client keeps up with router. (I benchmarked it with iperf3)

1 Like

Wireguard on Zyxel Multy M1 (WSM20)

# iperf3 -c 10.99.66.40
Connecting to host 10.99.66.40, port 5201
[  5] local 192.168.1.1 port 46526 connected to 10.99.66.40 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  16.3 MBytes   137 Mbits/sec    0    221 KBytes       
[  5]   1.00-2.00   sec  15.5 MBytes   130 Mbits/sec    0    221 KBytes       
[  5]   2.00-3.00   sec  15.5 MBytes   130 Mbits/sec    0    234 KBytes       
[  5]   3.00-4.00   sec  15.0 MBytes   126 Mbits/sec    0    260 KBytes       
[  5]   4.00-5.00   sec  15.0 MBytes   126 Mbits/sec    0    260 KBytes       
[  5]   5.00-6.00   sec  15.2 MBytes   128 Mbits/sec    0    260 KBytes       
[  5]   6.00-7.00   sec  14.9 MBytes   125 Mbits/sec    0    273 KBytes       
[  5]   7.00-8.00   sec  15.0 MBytes   126 Mbits/sec    0    273 KBytes       
[  5]   8.00-9.00   sec  16.3 MBytes   136 Mbits/sec    0    428 KBytes       
[  5]   9.00-10.00  sec  16.8 MBytes   141 Mbits/sec    0    428 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   155 MBytes   130 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   154 MBytes   129 Mbits/sec                  receiver

The Wireguard peer is running on a Fritzbox 7530 which might limit the throughput. I will re-run some tests later with a more potent peer. iperf3 server and client were not running on the Wireguard peers.

Great deal for the performance and features given (used but sealed units are sold for 30€ to 50€). Ideal router for travelling for me.

Here is a little update with a connection to a more potent peer (Ryzen 4300).
While sending data is limited to the same performance as in the quote above, receiving data is a bit improved:

# iperf3 -c 10.77.0.1 -R
Connecting to host 10.77.0.1, port 5201
Reverse mode, remote host 10.77.0.1 is sending
[  5] local 10.77.0.2 port 38858 connected to 10.77.0.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  27.9 MBytes   234 Mbits/sec                  
[  5]   1.00-2.00   sec  28.2 MBytes   237 Mbits/sec                  
[  5]   2.00-3.00   sec  28.3 MBytes   237 Mbits/sec                  
[  5]   3.00-4.00   sec  27.9 MBytes   234 Mbits/sec                  
[  5]   4.00-5.00   sec  28.6 MBytes   240 Mbits/sec                  
[  5]   5.00-6.00   sec  27.8 MBytes   233 Mbits/sec                  
[  5]   6.00-7.00   sec  27.8 MBytes   234 Mbits/sec                  
[  5]   7.00-8.00   sec  26.9 MBytes   226 Mbits/sec                  
[  5]   8.00-9.00   sec  26.9 MBytes   225 Mbits/sec                  
[  5]   9.00-10.00  sec  27.5 MBytes   231 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec   283 MBytes   237 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   278 MBytes   233 Mbits/sec                  receiver

So the router is not providing top class performance, but it's a huge improvement over my previous WNDR 3800.

2 Likes

Measured between 2 modern multicore PC's running iperf3 one wired on the LAN side the other wired on the WAN side
Router doing nothing else no Wireless.

{
"kernel": "5.15.123",
"hostname": "EA8500",
"system": "ARMv7 Processor rev 0 (v7l)",
"model": "Linksys EA8500 WiFi Router",
"board_name": "linksys,ea8500",
"rootfs_type": "squashfs",
"release": {
"distribution": "OpenWrt",
"version": "23.05-SNAPSHOT",
"revision": "r23314-7efec0acca",
"target": "ipq806x/generic",
"description": "OpenWrt 23.05-SNAPSHOT r23314-7efec0acca"
}
}

No irqbalance

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.01 sec 1.71 GBytes 490 Mbits/sec sender
[ 4] 0.00-30.01 sec 1.71 GBytes 490 Mbits/sec receiver

iperf Done.

irqbalance


[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.00 sec 2.03 GBytes 561 Mbits/sec sender
[ 4] 0.00-30.00 sec 2.03 GBytes 561 Mbits/sec receiver

iperf Done.

WireGuard


[ ID] Interval Transfer Bandwidth
[ 4] 0.00-20.00 sec 323 MBytes 136 Mbits/sec sender
[ 4] 0.00-20.00 sec 323 MBytes 135 Mbits/sec receiver

iperf Done.

PATCHED

root@EA8500:~# ubus call system board
{
"kernel": "5.15.127",
"hostname": "EA8500",
"system": "ARMv7 Processor rev 0 (v7l)",
"model": "Linksys EA8500 WiFi Router",
"board_name": "linksys,ea8500",
"rootfs_type": "squashfs",
"release": {
"distribution": "OpenWrt",
"version": "23.05-SNAPSHOT",
"revision": "r23399-e74a4b509f",
"target": "ipq806x/generic",
"description": "OpenWrt 23.05-SNAPSHOT r23399-e74a4b509f"
}
}

Normal LAN<>WAN throughput with irqbalance


[ ID] Interval Transfer Bandwidth
[ 4] 0.00-30.00 sec 3.14 GBytes 898 Mbits/sec sender
[ 4] 0.00-30.00 sec 3.14 GBytes 898 Mbits/sec receiver

iperf Done.

WireGuard


[ ID] Interval Transfer Bandwidth
[ 4] 0.00-20.00 sec 596 MBytes 250 Mbits/sec sender
[ 4] 0.00-20.00 sec 596 MBytes 250 Mbits/sec receiver

iperf Done.

I don't know if this thread, as it is, it is worth it. Normally, the right way to compare VPN performance is between clean wire throughput and VPN, without any other subprocess as NAT or firewall, generating traffic outside the router (with powerful devices) and connecting to another powerful device without CPU constraints. We do have a clear relation between CPU capacity and VPN performance. So, the real test would be related to the CPU (or SoC) and not exactly device model. We would have much less variants to test and much more useable info.

There any many internal and external factors that also affect the performance. A nice wiki page listing these most common factors would be great. I see interesting tests about irqbalance, the fact that OpenVPN is single thread (except when it used DCO). I would also add link latency, use of TCP (for OpenVPN), MTU size, that can both reduce the throughput or generate fragmentation. The default wg interface MTU, for example, fragments the encrypted packets when the peers uses IPv6 and anything with less than 1500 MTU (like PPPoE).

I think there is already a wiki article about VPN performance. We just need to expand that.

I understand your point but I doubt that many people are interested in such a theoretical VPN performance. In real life VPN is used to connect two private networks (or a device and a private network) via the Internet. So the routers firewall and NAT is always involved.

My initial idea was a topic where people can easily post the VPN performances of there devices so that other people can get an rough idea what the maximum is that they can expect from a device. The more complex this process is the less people will be willing to do that. Therefore, I would prefer simplicity to precision.

Other factors (like the external network infrastructure) are anyways much more relevant for the real VPN performance that you will finally get e.g. when you are travelling.

2 Likes

I think that indeed what people (like me) would like to know, what kind of performance to expect under 'normal' router operating mode.

2 Likes

OpenVPN has 2 isolated benchmark options, to test the raw CPU encryption throughput: https://x3mtek.com/openvpn-performance/

1.aes-centric encryption throughput:

openssl speed –evp aes-128-cbc

this only reflects part of the calc work, but might show, how good the router is capable of AES hw acceleration.

2.broader coverage of the total typical OpenVPN enc calc effort:

prep:

openvpn --genkey --secret /tmp/secret

bench:

time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-cbc

Result: e.g. real 0m18.405s

throughput calculation:
( 3200 / execution_time_seconds ) = theo. max perf in Mbit/s

Both benchmark options have the cipher as parameter, so you could also try different ciphers.

If you are mostly into comparing HW for OpenVPN, those 2 commands might be easier verifiable/more helpful for other users. I guess, a typical single home user OpenVPN scenario might not look that much like a massive parallel request flood as caused by iperf3, so including end to end network IO is probably not that important for real world SOHO use cases to measure, compared to the OpenVPN encryption part.

I respectfully disagree. Synthetic benchmarks have their use cases, but for assessing VPN performance of a router it seems odd to benchmark without actually routing....

On the other hand, "routing" does not imply the same overhead for everyone. I would prefer a synthetic but comparable benchmark, even if it comes with an asterisk that says "your actual performance will certainly be lower, depending on your router's other tasks".

So here's my thought, and if it's a stupid one please clobber me: Why not use OpenSSL's benchmarking capabilities? We have done that in the past and the AES benchmarks give a pretty good (even if rough) indication of what OpenVPN would be able to achieve with AES encryption. OpenSSL (edit: and wolfssl) now supports the ChaCha20-Poly1305 cipher, wouldn't a benchmark of that give us a rough estimate of what WireGuard would be able to push?

Simply because not everyone uses OpenSSL. E.g. Wireguard and IPsec use in-kernel encryption. OpenVPN compiled with OpenSSL marked experimental in OpenWRT...

I'm talking about using it for benchmarking with the respective ciphers, not everyday use.

Interesting thread. I have recently started to use wireguard and im loving it over ovpn for basic home VPN networking. However for larger networks I am very curious how Intel QAT v3 will work since Chachacha is now supported by QAT. Intel claims some really high VPN numbers in their testing.

I believe it was not why this thread was created :slight_smile:

No offense, but what are you talking about? "VPN performance", the topic of this here thread, hinges on the performance of the CPU with the respective ciphers. Are you thrown off by me suggesting an SSL tool, not a VPN binary, to benchmark the ciphers?

The point is "certainly be lower" can end up being much lower, or low enough to not be worthwhile anymore... because tight synthetic tests tend to not stress the memory and chache system of a SoC sufficiently. I would rather have a test with say PPPoE/NAT/firewalling and a note that on lighter encapsulations VPN throughput might actually increase a bit :wink: but I get your point there is a big "you mileage will vary" disclaimer needed.

I do not think this is a bad idea, but personally I would still only use such a page as starting point and then still try to find some evidence of how things perform in real life on the desired router.

At least that is what I think I WOULD like to dodo. What I actually did, was to simply install OpenVPN on my router and simply use it, my use-case is to occasionally do some administrative work and an occasional (smallish) file transfer so the actual VPN speed never really mattered all that much.

Absolutely, but we can only give a starting point, a rough estimate anyway. We have no way of estimating what else the users do with/on their router, what line they are on, what their latencies are. VPN performance depends on so many variables that, I'm arguing, anything else but a synthetic benchmark is pointless.

For example, I just benched my router using libwolfssl-benchmark. And the relevant lines it spits out are:

AES-128-CBC-enc             55 MiB took 1.004 seconds,   54.804 MiB/s Cycles per byte =  22.39
AES-128-CBC-dec             60 MiB took 1.077 seconds,   55.686 MiB/s Cycles per byte =  22.03
AES-256-CBC-enc             45 MiB took 1.035 seconds,   43.481 MiB/s Cycles per byte =  28.21
AES-256-CBC-dec             45 MiB took 1.018 seconds,   44.192 MiB/s Cycles per byte =  27.76
CHA-POLY                    70 MiB took 1.045 seconds,   66.967 MiB/s Cycles per byte =  18.32

So that's a baseline for me to compare my router with others, AES-128 or -256 for "traditional" OpenVPN, ChaCha20-Poly1305 for WireGuard and newer OpenVPN VPNs. (Never mind that I have no reason to change my router with these kinds of numbers. :wink: )

What I really would like to know is how tightly do SSL cipher benchmarks correlate with actual VPN performance? Are the SSL benchmarking tools way off in any direction, or do they deliver something close to what VPN binaries deliver?

1 Like

That hardly could be used for estimation if others provide iperf3 results. WolfSSL results, unlike openssl speed, do not provide information about block size. It is crucial for VPN performance estimation. VPN payload is less than 1500 and if you compare it with blocksize 8192 result... well, you will always complain about poor perfomance of your VPN :slight_smile:

I am very aware that SSL results are not identical to real-world VPN performance. What I am arguing is that benchmarking ciphers in SSL delivers a synthetic benchmark and is directly comparable, while VPN+iperf3 delivers an empirical result that is highly dependent on the individual setup of the tester -- and, especially with high performance routers, their ability to even create the necessary and proper setup.

... then it should be possible to qualify and quantify the correlation.

(Edit: I just tried wolfssl-benchmark 1500 for a benchmark with a block size of 1500 bytes, which consistently results in about 5% lower benchmarks for CHA-POLY and virtually identical marks for AES, but for some reason it fails to benchmark AES-CBC decoding with a "Bad function argument provided" error.)

But hey, I don't claim to have it all figured out. Is there a better synthetic benchmarking tool that tests the relevant ciphers?

1 Like

I kind of agree with your disagreement.
But OpenVPN is so much CPU-intensive that routing should not add up that much more to the table in a benchmark.

Intel QAT v3 is only available in Intel® Xeon® D-2700 processor and above. Quite pricey piece of hardware just to bring OpenVPN on-par with WireGuard performance-wise.

I tested OpenVPN vs WireGuard with proper iperf3, etheenet only, with both server and client on separate machines. WireGuard is at least 3 times faster, ceterus-paribus. On top of it, it is more robust (initial handshake is much much quicker) and do not get me started on ease of configuration.

While I see the advantage of OpenVPN modular nature and being able to replace encryption algorithms, it is really a mess of layer-upon-layer that does not scale well. Besides, you can replace WireGuard ChaCha20 for something else if you really want to.

1 Like