X86: Wireguard really slow

goldmund · April 7, 2023, 6:03pm

Hi!

I have installed Wireguard on my x86 router (Fujitsu S920) which is the only router at my home. I can connect to Wireguard from inside the network as well as from outside (e.g. LTE), however, network speeds then drop dramatically:

I tested this by connecting to the home Wi-Fi and running two speed tests:
Without Wireguard: 230Mbps
With Wireguard: 50-60Mbps

I should get about 250Mbps from my ISP. The CPU cores are not doing a lot when I connect through Wireguard, htop reports only a 30% load. I get the same slow results when I connect through LTE.

After doing a lot of research, I have enabled Packet Steering, but this hasn't done anything to the speeds.

Here are some technical details:

# /etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix '<redacted>'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth0'
	list ports 'eth1'
	list ports 'eth2'
	list ports 'eth3'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config device
	option name 'pppoe-WAN2'
	option type 'tunnel'

config interface 'WAN'
	option proto 'pppoe'
	option device 'eth4'
	option username '<redacted>'
	option password '<redacted>'
	option ipv6 'auto'

config interface 'wg_lan'
	option proto 'wireguard'
	option private_key '<redacted>'
	option listen_port '989'
	list addresses '10.0.5.1/24'
	option mtu '1420'

config wireguard_wg_lan
	option public_key '<redacted>'
	option preshared_key '<redacted>'
	option description '<redacted>'
	list allowed_ips '10.0.5.2/32'
	option route_allowed_ips '1'
	option persistent_keepalive '25'

config wireguard_wg_lan
	option public_key '<redacted>'
	option preshared_key '<redacted>'
	option description '<redacted>'
	list allowed_ips '10.0.5.6/32'
	option route_allowed_ips '1'
	option persistent_keepalive '25'

config wireguard_wg_lan
	option public_key '<redacted>'
	option preshared_key '<redacted>'
	option description '<redacted>'
	list allowed_ips '10.0.5.7/32'
	option route_allowed_ips '1'
	option persistent_keepalive '25'

# /etc/config/firewall

config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone 'lan'
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'
	list network 'wg_lan'

config zone 'wan'
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'WAN'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule 'wg'
	option name 'Allow-WireGuard-lan'
	option src 'wan'
	option dest_port '989'
	option proto 'udp'
	option target 'ACCEPT'

I really have no idea.
Thank you!

trendy · April 7, 2023, 6:29pm

Could be that the ISP is rate limiting certain ports or traffic. Try to use https or ssh port to see if there is any difference.

goldmund · April 7, 2023, 7:03pm

I have tried port 443, but unfortunately no difference in terms of speed...

frollic · April 7, 2023, 7:17pm

I could imagine the CPU being a bottle neck, install htop, and run it.

goldmund · April 7, 2023, 7:45pm

See my initial post:

I also suspected a slow CPU, but since load is around 30% when data is being transferred, I've had to rule it out...

frollic · April 7, 2023, 7:48pm

Sorry, missed it

Check this out though

goldmund · April 7, 2023, 8:22pm

Thanks, but I don't think that this issue is caused by my ISP limiting the use of certain ports.

I used to have a Raspberry Pi with Wireguard running on it (same ISP provider) and I got full network speeds. I now wanted my router to manage Wireguard instead in order to free up the Raspberry Pi.

Anyone?

vgaetera · April 7, 2023, 8:57pm

Test peak network performance outside the tunnel separately for TCP and UDP with Iperf, since UDP may have a different rate limiting policy.

goldmund · April 7, 2023, 9:47pm

You've hit the nail on the head! There is indeed trouble with UDP in my network, it seems!

I connected my PC to my router via a LAN cable, the router was an iperf3 server and my PC a client. The difference in bitrate is quite shocking:

TCP: 930 Mbits/sec
UDP: 1.05 Mbits/sec

Can someone tell me how to fix this?

_bernd · April 7, 2023, 10:56pm

NO you are measuring wrong. I don't have the option in my head but you have to set a rate limit for UDP with iperf/3.

goldmund · April 8, 2023, 8:13am

Oh, all right, sorry! Now I set:

$ iperf3 -c 192.168.1.1 -u -b 1000m

and got bitrate 932 Mbits/sec.

Weird.

eas · April 8, 2023, 7:42pm

You could still be limited by the speed of a single core.

frollic · April 8, 2023, 8:47pm

Then the CPU usage wouldn't be 30%, but around 25 or 50%, depending on if it's a dual, or quad core CPU.

goldmund · April 8, 2023, 8:54pm

It's a quad core CPU.

goldmund · April 8, 2023, 8:57pm

I use a Dynalink DL-WRX36 as a wireless access point at the moment and I have now somehow lost patience with x86 on the Fujitsu S920. I am going to try running Dynalink as my main router and see if Wireguard improves. Furthermore, upgrades will be MUCH more straightforward.

Thanks a lot for your help!

frollic · April 8, 2023, 9:48pm

Shouldn't be, if you do your home work, and use the squashfs.

eas · April 9, 2023, 4:25pm

That's incorrect. Being limited by the speed of a single core doesn't mean other cores don't have something to do.