Slow speeds on wireguard-to-wireguard transfers when wireless

Installing and Using OpenWrt Network and Wireless Configuration
1

I have two openwrt routers located in different cities.
At site A, Router A has a wireguard interface wg1 on it configured to connect to router B's wireguard interface wg0. I am using pbr here to allow traffic through the tunnel to the "vpn" interface. There is also a dumb access point behind this router which is broadcasting the SSID for wireless devices to use to go through the tunnel.

At site B, the setup is more simplistic, just a single router/firewall/access point.

I am finding really slow speed test numbers when connected on the order of 40-50 Mbps when I test tunnel over Wi-Fi, but speeds are great when I test the tunnel directly on router A.

  • Site A has a symmetrical gigabit connection and site B has an asymmetrical 50/500 MBit connection.
  • When connected to network A (dumb access point), speeds are slow (40-50Mbps).
  • When I run an iperf3 test on the wired router, I get fast results.
  • There is no CPU saturation anytime during the tests on either router.

On router A, directing the test to the wireguard interface:

# iperf3 -c 10.100.100.100
Connecting to host 10.100.100.100, port 5201
[  5] local 10.100.100.104 port 52352 connected to 10.100.100.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.50 MBytes  54.5 Mbits/sec    0    234 KBytes       
[  5]   1.00-2.00   sec  20.9 MBytes   175 Mbits/sec    0   1.25 MBytes       
[  5]   2.00-3.00   sec  69.5 MBytes   583 Mbits/sec    0   2.01 MBytes       
[  5]   3.00-4.00   sec  66.8 MBytes   560 Mbits/sec    0   2.01 MBytes       
[  5]   4.00-5.00   sec  71.0 MBytes   596 Mbits/sec    0   2.01 MBytes       
[  5]   5.00-6.00   sec  67.0 MBytes   562 Mbits/sec    0   2.01 MBytes       
[  5]   6.00-7.00   sec  68.8 MBytes   577 Mbits/sec    0   2.01 MBytes       
[  5]   7.00-8.00   sec  71.2 MBytes   598 Mbits/sec    0   2.01 MBytes       
[  5]   8.00-9.00   sec  69.0 MBytes   579 Mbits/sec    0   2.01 MBytes       
[  5]   9.00-10.00  sec  72.6 MBytes   609 Mbits/sec    0   2.01 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   583 MBytes   489 Mbits/sec    0             sender
[  5]   0.00-10.03  sec   579 MBytes   484 Mbits/sec                  receiver

iperf Done.

Now if I repeat this on my phone, (iperf for iOS), I get the slower 40-50 Mbit results.

Here is running iperf3 on the router B at the remote site and conducting the iperf test on the phone:

# iperf3 -s -f m          
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 10.100.100.104, port 50179
[  5] local 192.168.1.1 port 5201 connected to 10.100.100.104 port 50180
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.75 MBytes  56.6 Mbits/sec   29    313 KBytes       
[  5]   1.00-2.00   sec  6.38 MBytes  53.5 Mbits/sec    0    355 KBytes       
[  5]   2.00-3.00   sec  6.38 MBytes  53.5 Mbits/sec    0    381 KBytes       
[  5]   3.00-4.00   sec  6.50 MBytes  54.5 Mbits/sec    5    291 KBytes       
[  5]   4.00-5.00   sec  5.88 MBytes  49.3 Mbits/sec   13    161 KBytes       
[  5]   5.00-6.00   sec  5.38 MBytes  45.1 Mbits/sec    0    182 KBytes       
[  5]   6.00-7.00   sec  5.88 MBytes  49.3 Mbits/sec    0    205 KBytes       
[  5]   7.00-8.00   sec  6.38 MBytes  53.5 Mbits/sec    0    226 KBytes       
[  5]   8.00-9.00   sec  6.38 MBytes  53.5 Mbits/sec    0    246 KBytes       
[  5]   9.00-10.00  sec  6.25 MBytes  52.5 Mbits/sec    4    197 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec  62.1 MBytes  52.0 Mbits/sec   51             sender
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------

I am thinking that the problem must lie with my dumb AP behind router A.

Happy to post more configs but I will start with /etc/config/network from all three devices.

On router A:

/etc/config/network 

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1d:692b:58dc::/48'
	option packet_steering '2'
	option steering_flows '128'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1'
	option ipv6 '0'

config device
	option name 'br-lan.3'
	option type '8021q'
	option ifname 'br-lan'
	option vid '3'
	option ipv6 '0'

config device
	option name 'br-lan.4'
	option type '8021q'
	option ifname 'br-lan'
	option vid '4'
	option ipv6 '0'

config device
	option name 'br-lan.5'
	option type '8021q'
	option ifname 'br-lan'
	option vid '5'
	option ipv6 '0'

config device
	option name 'br-lan.6'
	option type '8021q'
	option ifname 'br-lan'
	option vid '6'
	option ipv6 '0'

config device
	option name 'br-lan.10'
	option type '8021q'
	option ifname 'br-lan'
	option vid '10'
	option ipv6 '0'

config bridge-vlan
	option device 'br-lan'
	option vlan '3'
	list ports 'eth1:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '4'
	list ports 'eth1:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '5'
	list ports 'eth1:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '6'
	list ports 'eth1:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'eth1:t'

config device
	option type 'bridge'
	option name 'lxcbr0'
	option ipv6 '0'
	option bridge_empty '1'

config interface 'wan'
	option device 'eth0'
	option proto 'static'
	option ipaddr 'xxx'
	option netmask '255.255.255.192'
	option gateway 'xxx'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'guest'
	option device 'br-lan.3'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'homeoffice'
	option device 'br-lan.4'
	option proto 'static'
	option ipaddr '10.9.6.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'iot'
	option device 'br-lan.5'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'vpn'
	option device 'br-lan.6'
	option proto 'static'
	option ipaddr '10.9.4.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'lan'
	option device 'br-lan.10'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'lxc'
	option device 'lxcbr0'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'

config interface 'wg0'
	option proto 'wireguard'
	option listen_port '51820'
	list addresses '10.200.200.200/24'
	option private_key 'xxx'
	option delegate '0'

config wireguard_wg0
	option description 'me'
	list allowed_ips '10.200.200.201/32'
	option route_allowed_ips '1'
	option public_key 'xxx'
	option preshared_key 'xxx'

config interface 'wg1'
	option proto 'wireguard'
	option private_key 'xxx'
	list addresses '10.100.100.104/24'
	list dns '10.0.4.250'
	option defaultroute '0'
	option delegate '0'

config wireguard_wg1
	option description 'MT6000'
	option public_key 'xxx'
	option preshared_key 'xxx'
	option endpoint_host 'my.routerb.org'
	option endpoint_port '47100'
	list allowed_ips '0.0.0.0/0'
	option persistent_keepalive '25'
/etc/config/pbr 
config pbr 'config'
	option enabled '1'
	option verbosity '2'
	option strict_enforcement '1'
	option resolver_set 'dnsmasq.nftset'
	list resolver_instance '*'
	option ipv6_enabled '0'
	list ignored_interface 'vpnserver'
	option nft_file_support '0'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_boot_delay '0'
	option procd_reload_delay '1'
	option webui_show_ignore_target '0'
	option nft_set_auto_merge '1'
	option nft_set_counter '1'
	option nft_set_flags_interval '1'
	option nft_set_flags_timeout '0'
	option nft_set_policy 'performance'
	list webui_supported_protocol 'all'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'
	option secure_reload '1'

config include
	option path '/usr/share/pbr/pbr.user.aws'
	option enabled '0'

config include
	option path '/usr/share/pbr/pbr.user.netflix'
	option enabled '0'

config policy
	option name 'Ignore Local Requests'
	option interface 'ignore'
	option dest_addr '10.0.0.0/24 10.0.1.0/24 192.168.100.0/24 192.168.1.0/24'
	option enabled '0'

config policy
	option name 'Plex/Emby Local Server'
	option interface 'wan'
	option src_port '8096 8920 32400'
	option enabled '0'

config policy
	option name 'Plex/Emby Remote Servers'
	option interface 'wan'
	option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'
	option enabled '0'

config policy
	option src_addr '10.9.4.0/24'
	option interface 'wg1'
	option name 'wg1 redirect'

On dumb access point behind router A:

/etc/config/network 

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdb2:3f17:7fe0::/48'
	option packet_steering '2'
	option steering_flows '128'

config interface 'lan'
	option device 'br-lan.10'
	option proto 'dhcp'

config interface 'guest'
	option proto 'none'
	option device 'br-lan.3'
	option delegate '0'

config interface 'homeoffice'
	option device 'br-lan.4'
	option proto 'none'
	option delegate '0'

config interface 'iot'
	option proto 'none'
	option device 'br-lan.5'
	option delegate '0'

config interface 'vpn'
	option proto 'none'
	option device 'br-lan.6'
	option delegate '0'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'lan2:u*'
	list ports 'lan3:u*'
	list ports 'lan4:u*'
	list ports 'wan:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '3'
	list ports 'wan:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '4'
	list ports 'wan:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '5'
	list ports 'wan:t'

config bridge-vlan
	option device 'br-lan'
	option vlan '6'
	list ports 'wan:t'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'
	list ports 'wan'
	option ipv6 '0'

config device
	option name 'br-lan.3'
	option type '8021q'
	option ifname 'br-lan'
	option vid '3'
	option ipv6 '0'

config device
	option name 'br-lan.4'
	option type '8021q'
	option ifname 'br-lan'
	option vid '4'
	option ipv6 '0'

config device
	option name 'br-lan.5'
	option type '8021q'
	option ifname 'br-lan'
	option vid '5'
	option ipv6 '0'

config device
	option name 'br-lan.6'
	option type '8021q'
	option ifname 'br-lan'
	option vid '6'
	option ipv6 '0'

config device
	option name 'br-lan.10'
	option type '8021q'
	option ifname 'br-lan'
	option vid '10'
	option ipv6 '0'

config device
	option name 'lan2'
	option ipv6 '0'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'lan3'
	option ipv6 '0'

config device
	option name 'wan'
	option ipv6 '0'

config device
	option name 'lan4'
	option ipv6 '0'

config device
	option name 'phy1-ap0'
	option ipv6 '0'
/etc/config/wireless 
config wifi-device 'radio0'
	option type 'mac80211'
	option path 'platform/soc/18000000.wifi'
	option band '2g'
	option country 'US'
	option cell_density '3'
	option htmode 'HE40'
	option channel '1'
	option txpower '3'
	option disabled '1'

config wifi-device 'radio1'
	option type 'mac80211'
	option path 'platform/soc/18000000.wifi+1'
	option band '5g'
	option country 'US'
	option cell_density '2'
	option htmode 'HE80'
	option channel '149'
	option txpower '24'

config wifi-iface 'wifinet2'
	option device 'radio1'
	option mode 'ap'
	option network 'lan'
	option ssid 'LexingtonAvenueLine'
	option encryption 'sae'
	option key 'xxx'
	option dtim_period '3'
	option wpa_disable_eapol_key_retries '1'
	option ieee80211w '2'
	option ocv '0'

config wifi-iface 'wifinet1'
	option device 'radio1'
	option mode 'ap'
	option network 'guest'
	option ssid 'CentralLine'
	option key 'xxx'
	option encryption 'sae-mixed'
	option dtim_period '3'
	option ieee80211w '2'
	option wpa_disable_eapol_key_retries '1'
	option ocv '0'

config wifi-iface 'wifinet3'
	option device 'radio1'
	option mode 'ap'
	option network 'homeoffice'
	option ssid 'CrosstownLine'
	option key 'xxx'
	option encryption 'sae-mixed'
	option dtim_period '3'
	option ieee80211w '1'
	option wpa_disable_eapol_key_retries '1'

config wifi-iface 'wifinet4'
	option device 'radio1'
	option mode 'ap'
	option network 'iot'
	option ssid 'lightsout'
	option key 'xxx'
	option encryption 'sae-mixed'
	option dtim_period '3'
	option ieee80211w '1'
	option wpa_disable_eapol_key_retries '1'

config wifi-iface 'wifinet6'
	option device 'radio1'
	option mode 'ap'
	option network 'vpn'
	option ssid 'Amtrack'
	option key 'xxx'
	option encryption 'sae-mixed'
	option dtim_period '3'
	option ieee80211w '1'
	option wpa_disable_eapol_key_retries '1'

config wifi-iface 'wifinet5'
	option device 'radio0'
	option mode 'ap'
	option ssid 'lightsout2.4'
	option encryption 'sae-mixed'
	option key 'xxx'
	option wpa_disable_eapol_key_retries '1'
	option network 'iot'
	option disabled '1'

config wifi-iface 'wifinet7'
	option device 'radio1'
	option mode 'ap'
	option ssid 'GuestNetwork-5GHz'
	option encryption 'sae-mixed'
	option key 'xxx'
	option ocv '0'
	option network 'guest'
	option disabled '1'

On router B:

/etc/config/network 
config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd99:97d2:a14e::/48'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lan'
	option device 'br-lan.10'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'
	list ports 'lan5'
	option ipv6 '0'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'lan1:u*'
	list ports 'lan2:u*'
	list ports 'lan3:u*'
	list ports 'lan4:u*'
	list ports 'lan5:u*'

config bridge-vlan
	option device 'br-lan'
	option vlan '3'

config bridge-vlan
	option device 'br-lan'
	option vlan '4'

config device
	option name 'br-lan.10'
	option type '8021q'
	option ifname 'br-lan'
	option vid '10'
	option ipv6 '0'

config device
	option name 'br-lan.3'
	option type '8021q'
	option ifname 'br-lan'
	option vid '3'
	option ipv6 '0'

config device
	option name 'br-lan.4'
	option type '8021q'
	option ifname 'br-lan'
	option vid '4'
	option ipv6 '0'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'lan1'
	option ipv6 '0'

config device
	option name 'lan2'
	option ipv6 '0'

config device
	option name 'lan3'
	option ipv6 '0'

config device
	option name 'lan4'
	option ipv6 '0'

config device
	option name 'lan5'
	option ipv6 '0'

config interface 'guest'
	option proto 'static'
	option device 'br-lan.3'
	option ipaddr '192.168.3.1'
	option netmask '255.255.255.0'

config interface 'iot'
	option proto 'static'
	option device 'br-lan.4'
	option ipaddr '192.168.4.1'
	option netmask '255.255.255.0'

config interface 'lxc'
	option device 'lxcbr0'
	option proto 'static'
	option netmask '255.255.255.0'
	option ipaddr '10.0.4.1'

config interface 'wg0'
	option proto 'wireguard'
	list addresses '10.100.100.100/24'
	option private_key 'xxx'
	option listen_port '47100'
	option delegate '0'

config wireguard_wg0
	option description 'phone'
	list allowed_ips '10.100.100.103/32'
	option route_allowed_ips '1'
	option public_key 'xxx'
	option preshared_key 'xxx'

config wireguard_wg0
	option description 'amtrack'
	list allowed_ips '10.100.100.104/32'
	option route_allowed_ips '1'
	option public_key 'xxx'
	option preshared_key 'xxx'

config device
	option type 'bridge'
	option name 'lxcbr0'
	option bridge_empty '1'
	option ipv6 '0'
2

Post /etc/config/wireless from the router with wifi.

3

I edited the first post adding it.

4

Seems pretty typical if your SSID is in a heavily radio-congested area, especially 2.4GHz.Your hardware-to-hardware test is indicative of that.

5

No, I’m out in the isle of nowhere. If iperf test from the phone to another SSID that routes through the ISP, I get 700+ Mbps

6

Reduce cell_density to 1 or 0, and restore txppower to default.

7

I dropped the cell_density to 1 and the txppower to driver default but no change. When I connect to another SSID on my network, get 700+ MBps. This is really odd.

EDIT: using iperf3 directly on the dumb AP also gives good throughput so this would seem to isolate the problem to something in the wireless stack, no?

On the dumb AP:

# iperf3 -c 10.100.100.100

Connecting to host 10.100.100.100, port 5201
[  5] local 10.9.8.2 port 57406 connected to 10.100.100.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  6.50 MBytes  54.5 Mbits/sec    0    353 KBytes       
[  5]   1.00-2.00   sec  26.0 MBytes   218 Mbits/sec    0   1.55 MBytes       
[  5]   2.00-3.00   sec  63.1 MBytes   530 Mbits/sec    0   2.02 MBytes       
[  5]   3.00-4.00   sec  64.2 MBytes   539 Mbits/sec    0   2.02 MBytes       
[  5]   4.00-5.00   sec  60.8 MBytes   510 Mbits/sec    0   2.02 MBytes       
[  5]   5.00-6.00   sec  63.4 MBytes   532 Mbits/sec    0   2.02 MBytes       
[  5]   6.00-7.00   sec  61.4 MBytes   515 Mbits/sec    0   2.02 MBytes       
[  5]   7.00-8.00   sec  64.4 MBytes   540 Mbits/sec    0   2.02 MBytes       
[  5]   8.00-9.00   sec  62.4 MBytes   523 Mbits/sec    0   2.02 MBytes       
[  5]   9.00-10.00  sec  60.0 MBytes   503 Mbits/sec    0   2.02 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   532 MBytes   446 Mbits/sec    0             sender
[  5]   0.00-10.03  sec   531 MBytes   444 Mbits/sec                  receiver

iperf Done.

On my phone, running iperf for iOS using 10.100.100.100 as the target, I get around 40-50 MBps.

@egc - you have been extremely helpful, pinging to see if you have any thoughts

8 
ubus call system board

You know that CPU processing speed is not made out of rubber.

9

For the dumb AP:

# ubus call system board
{
	"kernel": "6.6.57",
	"hostname": "redmi-AP",
	"system": "ARMv8 Processor rev 4",
	"model": "Xiaomi Redmi Router AX6000 (OpenWrt U-Boot layout)",
	"board_name": "xiaomi,redmi-router-ax6000-ubootmod",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "SNAPSHOT",
		"revision": "r27796-caf5caae44f5",
		"target": "mediatek/filogic",
		"description": "OpenWrt SNAPSHOT r27796-caf5caae44f5"
	}
}
10

That is very fast device indeed.
Try bridger package to apply offloads outside firewall.

11

I used it in the past but found that it would cause kernel panics so I discontinued using it. For now, I would like to troubleshoot the slow connection described above.

12

To rule out something on the dumb access point, I swapped it out for an older Archer A6 v3 and I got the same slow result when connected over the wireless interface. I am not sure what is to blame.

13

Check MTU-s.

14

Not sure which settings, can you suggest? For the vpn interface?

15

Cannot make a guess regarding 4-5 devices in chain yielding poor performance.

16

I am seeing stuff like this. I will try using alternative values for MTU on one of the peers and see if it improves. Thanks for the suggestion.

17

You need to count exact encapsulation overheads across the path, like vlan=4 or pppoe=8 and so on.

18

Do you have a link that explains the process you’re suggesting?

19

You serious? Subtract two numbers.

20

Just scanning some values between 1200 and 1350 stepping up by 25 (1200,1225,1250...1350) on one peer did not results in anything different.