Wireguard dropping connection speed by almost 80%

kowrt · December 31, 2022, 5:58am

Wireguard (VPN) is dropping connection speed by almost 80%. I read some earlier threads and tried the below already:
• Enabled packet steering in Global network options
• Enabled software/hardware offloading
• Modified br-lan MTU to 1300
• Modified Wifi operating frequency width to 80HZ in 5GHZ

I have tested that same wireguard VPN location without openwrt and it doesn't reduce my connection speed by no where near that much.

Router Model: TP-Link Archer A7 v5
Firmware Version: OpenWrt 22.03.2

Below is my /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix '<IPV6>::/48'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        option mtu '1300'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option ipaddr '192.168.33.1'

config device
        option name 'eth0.2'
        option macaddr '<MacAddress>'

config interface 'wan'
        option device 'eth0.2'
        option proto 'dhcp'
        option peerdns '0'
        list dns '<VPN's Public DNS Server>'

config interface 'wan6'
        option device 'eth0.2'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix 'auto'
        option peerdns '0'
        list dns '<VPN's Public DNS Server>'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0t 1'
        option vid '2'

config interface 'wg'
        option proto 'wireguard'
        list addresses '10.64.90.154/32'
        option force_link '1'
        option peerdns '0'
        list dns '<VPN's Public DNS Server>'
        option private_key '<MyPrivateKey1>'

config wireguard_wg
        option public_key '<MyPublicKey1>'
        list allowed_ips '0.0.0.0/0'
        option route_allowed_ips '1'
        option endpoint_host '<Unique IP Address of this interface that I will connect to>'
        option endpoint_port '51820'
        option description 'Location1'

This is what I see in iperf3 test with
Server: iperf3 -s
Client: iperf3 -c 192.168.33.1

Accepted connection from 192.168.33.118, port 61415
[  5] local 192.168.33.1 port 5201 connected to 192.168.33.118 port 61416
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  22.6 MBytes   189 Mbits/sec
[  5]   1.00-2.00   sec  20.3 MBytes   170 Mbits/sec
[  5]   2.00-3.00   sec  23.9 MBytes   200 Mbits/sec
[  5]   3.00-4.00   sec  22.5 MBytes   189 Mbits/sec
[  5]   4.00-5.01   sec  22.2 MBytes   185 Mbits/sec
[  5]   5.01-6.00   sec  20.9 MBytes   176 Mbits/sec
[  5]   6.00-7.01   sec  22.3 MBytes   187 Mbits/sec
[  5]   7.01-8.00   sec  22.3 MBytes   188 Mbits/sec
[  5]   8.00-9.00   sec  22.5 MBytes   189 Mbits/sec
[  5]   9.00-10.00  sec  19.3 MBytes   161 Mbits/sec
[  5]  10.00-10.01  sec   128 KBytes   161 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec   219 MBytes   183 Mbits/sec                  receiver

On checking speed while connected to Wifi with Wireguard connection:

Thanks and appreciate any help on this.

patrakov · December 31, 2022, 7:07am

This router has a very weak CPU, and any VPN involves some CPU-intensive encryption and decryption. Also your router's WiFi chipset is quite taxing on the CPU. So the speed that you get is more-or-less expected.

You can confirm (e.g. by running top over ssh, or by using Status > Realtime graphs) that the CPU becomes pegged to 100% and thus represents a bottleneck.

For VPN, I can only recommend x86 routers (Qotom fanless mini-pcs).

kowrt · December 31, 2022, 8:24am

This makes sense. I have done some testing though based on what you mentioned but seems like CPU is not that loaded to reduce speed so drastically.

Not sure if I'm looking at right parameters but it shows 81% idle. I tested speed few times when CPU idle was at even 90%, the speed was still dropping by 80%.

root@OpenWrt:~# top -n 1 -b
Mem: 50852K used, 71468K free, 344K shrd, 0K buff, 13268K cached
CPU:   0% usr   4% sys   0% nic  81% idle   0% io   0% irq  12% sirq
Load average: 0.64 0.37 0.30 1/57 15689

Thanks for the suggestion, I'm looking something in the range of $150 if not this router so I will check if this is something that's in that budget.

jav · December 31, 2022, 9:03am

While it's difficult to find a non-x86 router with good performance for VPNs, I'm currently using Xiaomi Redmi AX6000 (Mediatek MT7986A) with quite good VPN performance (>600 Mbps). It's quite cheap at ~$100. With the same chipset, there're more capable systems (Banana Pi-R3, TP-Link TL-XDR6088) but they're also more expensive.

apccv · December 31, 2022, 4:48pm

Have you considered a separate device just for WireGuard? Have any old routers lying around?

We had an old D-Link 825C1 gathering dust. It was discontinued 10 years ago, so you can get an idea of how dull this cutting edge is. With 22.03.2, radios off, and connected through Ethernet to the main router, we get close to 40 down 30 up Mbps on VPN and less than 250ms of ping time loaded according to fast-dot-com. (CPU flatlines at that point so that's as far as it will go). A far cry from the 600Mbps mentioned above but very livable when travelling for basic internet access.

kowrt · December 31, 2022, 6:51pm

This router is being used just for wireguard with just one device connected via Wifi. Not sure why it's still dropping speed by more than 80%.

psherman · December 31, 2022, 6:54pm

What happens if you disable wifi entirely and run via ethernet? Does that improve the speeds? And is there anything else running on this router (any other user-installed packages, obviously not including Wireguard)?

apccv · December 31, 2022, 7:58pm

why it's still dropping speed by more than 80%

As it was said above, our best guess is because the CPU is tiny. These things are streamlined to do routing or WiFi, not for the newer encryption required by WireGuard.

Routing WiFi takes CPU cycles. So does PPPoE. Is the router also handling PPPoE? Everything counts with these small CPUs. I'd be surprised if you can get more than 50Mbps in ideal conditions.

The other common issue w/WireGuard is MTU size mismatches that lower throughput due to fragmentation. I'm no expert w/that... the forum has plenty on it.

Not sure if top would report what the CPU is going through. I usually run a vmstat 1 to look for sys/wait numbers.

Just my 0.02.

Pico · December 31, 2022, 8:40pm

Could be that it is just visually dropping in speed, by incorrectly calculating a much too high throughput at the beginning. Never trust initial segments of benchmarks.

This „too much at the beginning and dropping eventually“ of measurements could have something to do with integer-based division and not calculating diff time in microseconds from the start. While counted bits are quickly in the fine grained 7 digit range right at the beginning, elapsed seconds are quite some time in the below 3 digit range, rather coarse grained.
Also empty IO buffers at beginning of enduring network transfer on client and/or server side could cause initially quicker response times at beginning.

patrakov · December 31, 2022, 9:42pm

Definitely unrelated.

I have set up a test WireGuard VPN server on DigitalOcean, and compared throughput achievable on two different routers.

TP-Link Archer C7 v2 (same hardware as TP-Link Archer A7): 25 Mbps
Linksys E8450: 150 Mbps

Both figures are via WiFi.

apccv · January 1, 2023, 1:10am

On upload the DIR-825C1 with an AR9344 (ath79, mips_24kc, 560MHz) gets 25-30Mbps. Ballpark in line with what you are getting off the Archer A7 (ath79, mips_24kc,720MHz) once you factor in WiFi load. My guess is that is what those SOCs can do.

The E8450 is a completely different, much more modern beast, with a MediaTek MT7622. Nice to see it can do 150Mbps. AFAIK, mvebu (the architecture behind the WRT1900AC) can push about 200Mbps according to a chart somewhere on forums with benchmarks.

kowrt · January 1, 2023, 1:40am

No difference in wireguard speed. I have done multiple rounds of testing over Wifi vs Ethernet and with different settings and almost have same results.

I have policy service running on it but it just has one policy rule. I even tested after Stopping this service and tested over Ethernet and Wifi and still have same results.

Additionally, without wireguard I'm getting speeds up to 250 Mbps even over Wifi.

kowrt · January 1, 2023, 1:45am

Thanks, this does make sense but the CPU is almost idle in my case as there's only wireguard running on it with one device connected and only testing I did was for speed test so that doesn't justify 80% drop in speeds.

No

I have already tried different MTU settings based on different forum threads.

Could you please provide more information on how to check this? I can test that and verify.

kowrt · January 1, 2023, 1:53am

That's some huge difference. How's your experience been with E8450? Do you have any suggested router in range on $150? My main priority is high speed. My main network is Wifi mesh and gets me speed of up to 500Mbps over Wifi so I'm looking only for an additional router to manage few devices over wireguard.

I see same results over Ethernet too if using wireguard. Without Wireguard I'm getting speeds of up to 250 Mbps even over Wifi.

apccv · January 1, 2023, 2:55am

Sure. If your image does not have the vmstat command installed, you can install it via opkg update && opkg install procps-ng-vmstat

This is what a fast-dot-com run looks like on the DIR-825C1:

BusyBox v1.35.0 (2022-10-14 22:44:41 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 22.03.2, r19803-9a599fee93
 -----------------------------------------------------
root@VPNWRT:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0  74644      0  19820    0    0     0     0  495   72  0  1 99  0  0
 0  0      0  74644      0  19820    0    0     0     0  489   51  0  2 98  0  0
 0  0      0  74644      0  19820    0    0     0     0  461   39  0  1 99  0  0
 0  0      0  74644      0  19820    0    0     0     0  497   35  0  0 100  0  0
 0  0      0  74644      0  19820    0    0     0     0  838  344  0  4 96  0  0
 0  0      0  74644      0  19820    0    0     0     0  687   40  1  0 99  0  0
 0  0      0  74644      0  19820    0    0     0     0  597  123  0  3 97  0  0
>>>DOWNLOAD TEST STARTS ABOUT HERE<<<
 1  0      0  74660      0  19820    0    0     0     0 1206  994  3 15 82  0  0
 1  0      0  74660      0  19820    0    0     0     0  969  751  3 12 85  0  0
 0  0      0  74632      0  19820    0    0     0     0 1294 1325  6 12 82  0  0
 2  0      0  73064      0  19820    0    0     0     0 3826 3660  0 100  0  0  0
 3  0      0  74284      0  19820    0    0     0     0 4182 4028  2 98  0  0  0
 1  0      0  72736      0  19820    0    0     0     0 4189 3211  0 100  0  0  0
 3  0      0  72640      0  19820    0    0     0     0 4028 2579  0 100  0  0  0
 2  0      0  72656      0  19820    0    0     0     0 4589 4109  0 100  0  0  0
 3  0      0  72864      0  19820    0    0     0     0 4522 3779  0 100  0  0  0
 3  0      0  73040      0  19820    0    0     0     0 4646 5021  0 100  0  0  0
 3  0      0  72528      0  19820    0    0     0     0 4168 2885  0 100  0  0  0
 1  0      0  72736      0  19820    0    0     0     0 4418 4456  0 100  0  0  0
 1  0      0  72656      0  19820    0    0     0     0 4168 3711  0 100  0  0  0
 3  0      0  73232      0  19820    0    0     0     0 4281 3581  0 100  0  0  0
 1  0      0  72564      0  19820    0    0     0     0 4476 3483  1 99  0  0  0
 2  0      0  72836      0  19820    0    0     0     0 4487 3928  0 100  0  0  0
 1  0      0  72756      0  19820    0    0     0     0 4224 3096  0 100  0  0  0
 1  0      0  72564      0  19820    0    0     0     0 4604 4395  0 100  0  0  0
 1  0      0  74448      0  19820    0    0     0     0 4546 4093  0 72 28  0  0
 0  0      0  74448      0  19820    0    0     0     0 2976 3184  1 47 52  0  0
 1  0      0  74416      0  19820    0    0     0     0 3281 3504  0 42 58  0  0
 2  0      0  74416      0  19820    0    0     0     0 2973 3431  0 45 55  0  0
 3  0      0  74416      0  19820    0    0     0     0 3203 3449  0 42 58  0  0
>>>UPLOAD TEST STARTS ABOUT HERE<<<
 1  0      0  72432      0  19820    0    0     0     0 4144 1838  0 77 23  0  0
 2  0      0  72168      0  19820    0    0     0     0 4949 1922  0 100  0  0  0
 2  0      0  71832      0  19820    0    0     0     0 4791 1873  0 100  0  0  0
 1  0      0  72776      0  19820    0    0     0     0 5266 2053  0 100  0  0  0
 2  0      0  73268      0  19820    0    0     0     0 5208 2161  0 100  0  0  0
 2  0      0  73060      0  19820    0    0     0     0 6186 2088  0 100  0  0  0
 1  0      0  73028      0  19820    0    0     0     0 5260 2171  0 100  0  0  0
 2  0      0  72900      0  19820    0    0     0     0 4918 1994  0 100  0  0  0
 2  0      0  72872      0  19820    0    0     0     0 4795 2191  7 93  0  0  0
 2  0      0  72712      0  19820    0    0     0     0 5112 2180  1 99  0  0  0
 2  0      0  72584      0  19820    0    0     0     0 5050 2227  0 100  0  0  0
 4  0      0  72424      0  19820    0    0     0     0 5054 2170  0 100  0  0  0
 2  0      0  72024      0  19820    0    0     0     0 5266 2341  0 100  0  0  0
 3  0      0  72040      0  19820    0    0     0     0 5114 2138  0 100  0  0  0
 2  0      0  72588      0  19820    0    0     0     0 5227 2253  0 100  0  0  0
 4  0      0  72364      0  19820    0    0     0     0 6254 2159  0 100  0  0  0
 2  0      0  72204      0  19820    0    0     0     0 5242 2247  1 99  0  0  0
 3  0      0  72076      0  19820    0    0     0     0 5184 2176  0 100  0  0  0
 2  0      0  72076      0  19820    0    0     0     0 5132 2355  0 100  0  0  0
 2  0      0  72140      0  19820    0    0     0     0 5048 2458  0 100  0  0  0
 2  0      0  72108      0  19820    0    0     0     0 5032 2435  0 100  0  0  0
 2  0      0  72204      0  19820    0    0     0     0 5161 2290  0 100  0  0  0
 2  0      0  72076      0  19820    0    0     0     0 5034 2354  0 100  0  0  0
 2  0      0  71772      0  19820    0    0     0     0 4962 2467  1 99  0  0  0
 2  0      0  72396      0  19820    0    0     0     0 5314 2358  0 100  0  0  0
 2  0      0  72652      0  19820    0    0     0     0 5071 2256  0 100  0  0  0
 2  0      0  71788      0  19820    0    0     0     0 5081 2303  1 99  0  0  0
 2  0      0  72300      0  19820    0    0     0     0 5300 2397  0 100  0  0  0
 3  0      0  71980      0  19820    0    0     0     0 4992 2315  0 100  0  0  0
 2  0      0  71804      0  19820    0    0     0     0 5010 2277  0 100  0  0  0
 0  0      0  74356      0  19820    0    0     0     0 3978 1688  0 75 25  0  0
>>>TEST ENDS<<<
 0  0      0  74356      0  19820    0    0     0     0  601   36  0  0 100  0  0
 0  0      0  74388      0  19820    0    0     0     0  532   31  0  0 100  0  0
 0  0      0  74388      0  19820    0    0     0     0  516   96  0  1 99  0  0
 0  0      0  74388      0  19820    0    0     0     0  501   46  1  1 98  0  0
 0  0      0  74388      0  19820    0    0     0     0  448   51  1  2 97  0  0
^C
root@VPNWRT:~#

Note how idle (third to last column) idles at near 100 at start. Drops to zero at the same time CPU goes 100% on SYS (kernel). The first chunk is the download test (scored 35Mbps) and the second chunk is for upload (scored 26Mbps). These are typical results for me.

In this case the router is the WG server and an iPhone is the client. All traffic is through Ethernet as there is an access point between the iPhone and the WG server. The router is working on WG and routing and nothing else.

The SOC in the Archer C7 v2 is a little more sophisticated than the DIR-825C2 but not much more. Not sure if it is a dual core (the DIR-825 sure isn't). If you are asking it to do WiFi on top of WG I would expect similar results at best.

Nihilokrat · January 1, 2023, 12:54pm

Best I got out of wireguard on Archer C7v4 and v5, which are listed with the same CPU, was 65 Mbit/s over LAN and roughly 60 over 5G wifi. So in my experience the 25 Mbit/s is definitely on the low side.

However, my tests have been with OpenWrt's latest stable releases of 18 and 19. So there might've been changes since then that impact wireguard speeds negatively.

patrakov · January 1, 2023, 3:18pm

So far, I have no better suggestions below $150 that I have personally tested.

efahl · January 1, 2023, 6:40pm

Well, if you're in the US, Walmart has had the Belkin RT3200 at $63 for the past few days. Depending on how much you're willing to spend it's pretty reasonable to just grab one for experimentation. (Despite identical internals, my searches show the Linksys E8450 is typically priced almost twice as high as the Belkin, and only differs in that it has a black case instead of the Belkin's white one.)

Experience: I've installed two RT3200s (sister's for almost a year and son's for about 6 months, both as their main routers) and they've been rock solid. I have another one sitting on the desk here (next to a couple x86 routers) that I use for configuration and development experiments, never had any issues with it. (The RT3200 was pressed into service for our main WiFi AP for a couple weeks when I screwed up the standalone WAP.)

kowrt · January 3, 2023, 12:43am

Thanks, I did install it and test again.

I did notice similar behavior. SYS goes to 100. So, as even other have mentioned this router won't be able to support wg with any decent outcome.

frollic · January 3, 2023, 10:18am

Reyee RG-E5, same hw as the RT3200/E8450.
Got supported just two or three weeks ago, so only snapshots are currently available.

Open boxes start at around $45 on US eBay.