Problem with data transfers through wireguard tunnel

PROBLEM: When I transfer a 10K chunk of data to a remote server through a test wireguard tunnel, the ssh connection fails. A 1K transfer works fine. A 10K transfer through the tunnel to either wg router also works fine. The 10K transfer works fine when I don't use the wg tunnel.

Background: I have configured two test routers on my LAN to build a wireguard tunnel between them. They establish the tunnel fine, and routing works well from one router, through the tunnel to the other router, and then out to the real world. Each wireguard interface has it's own firewall zone, and that seems to work well too. I can access the internet through the tunnel. I am using 19.07.3.

This feels like a MTU problem, but I am unclear what I can or should do to fix it. https://www.speedguide.net/analyzer.php says MTU = 1420 and MSS = 1380, which matches what I would expect from wireguard.

test@frooze:~/wireguard$ dd if=/dev/zero count=1 bs=10K | ssh remote dd of=/dev/null
1+0 records in
1+0 records out
10240 bytes (10 kB, 10 KiB) copied, 7.1e-05 s, 144 MB/s
Timeout, server remote not responding.

test@frooze:~/wireguard$ dd if=/dev/zero count=1 bs=1K | ssh remote dd of=/dev/null
1+0 records in
1+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 6.96e-05 s, 14.7 MB/s
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0312847 s, 32.7 kB/s

What is your final goal?
It's unlikely that your real goal is transferring abstract data chunks.
I recommend to utilize some protocol with additional integrity control, e.g. rsync.
See also: Determine MTU size using ping

3 Likes

This is a test case that fails, so I want to fix it. dd is just a convenient test mechanism, available on Openwrt and big Linux.

Testing with ping shows packets up to 1392 bytes pass well.

What, if anything, is failing other than trying a 10K transfer? Is it having an impact on real world usage?

1 Like

Calculate the maximum MTU for your WAN and deduct MTU for the WG interface accordingly.
Repeat the WG tunnel testing when the MTU issue is resolved.

I have not yet found other failures. So far any large ssh upload through the wg tunnel will fail, including scp copies. Large downloads do not seem to fail.

It is having a big impact on my upload speed. Direct uploads to the internet are 30Mb/s against 3 Mb/s when they first go through the wireguard tunnel on my LAN.

What's the output of:

uci export network; uci export firewall; \
head -n -0 /etc/firewall.user; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; wg

on both routers?

How are the two routers connected to each other and the wider network/internet?

1 Like

The two routers are plugged into my normal LAN, since this is a test setup. They connect to the wider internet through the normal LAN's WAN connection.

The router configs are as follows, with no custom firewall.user file. I have blotted out the private keys, but the connection works as you can see from wg.

# router 264e
package network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd8d:6c6a:9a32::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.1'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device 'lan_eth0_1_dev'
        option name 'eth0.1'
        option macaddr 'c4:71:54:3b:5b:44'

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'dhcp'

config device 'wan_eth0_2_dev'
        option name 'eth0.2'
        option macaddr 'c4:71:54:3b:5b:45'

config interface 'wan6'
        option ifname 'eth0.2'
        option proto 'dhcpv6'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '1 2 3 4 6t'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0 6t'

config interface 'wg0'
        option proto 'wireguard'
        option private_key 'xxx='
        option listen_port '51555'
        list addresses '192.168.219.1/24'

config wireguard_wg0 'wgpeer'
        option persistent_keepalive '25'
        list allowed_ips '0.0.0.0/0'
        list allowed_ips '::/0'
        option endpoint_port '51555'
        option endpoint_host '192.168.8.252'
        option public_key 'U3CXLF55rU27j5s7Sh4kO1nKqgflVHeaqbfRyP4COjA='

package firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config include
        option path '/etc/firewall.user'

config zone 'vpn'
        option name 'vpn'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option masq '1'
        list network 'wg0'

config forwarding 'lan_vpn'
        option src 'lan'
        option dest 'vpn'

config forwarding 'vpn_wan'
        option src 'vpn'
        option dest 'wan'

config rule 'wg_inbound'
        option name 'Allow-WG-Inbound'
        option target 'ACCEPT'
        option src '*'
        option proto 'udp'
        option dest_port '51555'
        option limit '5/sec'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
6: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.8.251/24 brd 192.168.8.255 scope global eth0.2
       valid_lft forever preferred_lft forever
8: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.219.1/24 brd 192.168.219.255 scope global wg0
       valid_lft forever preferred_lft forever

default via 192.168.219.2 dev wg0 table LocalWiFi
default via 192.168.8.1 dev eth0.2 proto static src 192.168.8.251
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
192.168.8.0/24 dev eth0.2 proto kernel scope link src 192.168.8.251
192.168.219.0/24 dev wg0 proto kernel scope link src 192.168.219.1
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168.1.1
local 192.168.1.1 dev br-lan table local proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.168.1.1
broadcast 192.168.8.0 dev eth0.2 table local proto kernel scope link src 192.168.8.251
local 192.168.8.251 dev eth0.2 table local proto kernel scope host src 192.168.8.251
broadcast 192.168.8.255 dev eth0.2 table local proto kernel scope link src 192.168.8.251
broadcast 192.168.219.0 dev wg0 table local proto kernel scope link src 192.168.219.1
local 192.168.219.1 dev wg0 table local proto kernel scope host src 192.168.219.1
broadcast 192.168.219.255 dev wg0 table local proto kernel scope link src 192.168.219.1

0:      from all lookup local
32765:  from 192.168.1.1/24 lookup LocalWiFi
32766:  from all lookup main
32767:  from all lookup default

interface: wg0
  public key: DioqUp7BoOx+6dtQe90FKFF6eb82c25noKy5XZvevhY=
  private key: (hidden)
  listening port: 51555

peer: U3CXLF55rU27j5s7Sh4kO1nKqgflVHeaqbfRyP4COjA=
  endpoint: 192.168.8.252:51555
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 42 seconds ago
  transfer: 2.12 GiB received, 230.25 MiB sent
  persistent keepalive: every 25 seconds

Second router with almost identical config

# router 1e8b
package network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdfa:0803:f6ca::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.1'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device 'lan_eth0_1_dev'
        option name 'eth0.1'
        option macaddr '7c:8b:ca:f6:71:d6'

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'dhcp'

config device 'wan_eth0_2_dev'
        option name 'eth0.2'
        option macaddr '7c:8b:ca:f6:71:d7'

config interface 'wan6'
        option ifname 'eth0.2'
        option proto 'dhcpv6'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '1 2 3 4 6t'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0 6t'

config interface 'wg0'
        option proto 'wireguard'
        option private_key '0xxxI0Q='
        option listen_port '51555'
        list addresses '192.168.219.2/24'

config wireguard_wg0 'wgpeer'
        option persistent_keepalive '25'
        list allowed_ips '0.0.0.0/0'
        list allowed_ips '::/0'
        option endpoint_port '51555'
        option endpoint_host '192.168.8.251'
        option public_key 'DioqUp7BoOx+6dtQe90FKFF6eb82c25noKy5XZvevhY='

package firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config include
        option path '/etc/firewall.user'

config zone 'vpn'
        option name 'vpn'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option masq '1'
        list network 'wg0'

config forwarding 'lan_vpn'
        option src 'lan'
        option dest 'vpn'

config forwarding 'vpn_wan'
        option src 'vpn'
        option dest 'wan'

config rule 'wg_inbound'
        option name 'Allow-WG-Inbound'
        option target 'ACCEPT'
        option src '*'
        option proto 'udp'
        option dest_port '51555'
        option limit '5/sec'

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
6: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.8.252/24 brd 192.168.8.255 scope global eth0.2
       valid_lft forever preferred_lft forever
8: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 192.168.219.2/24 brd 192.168.219.255 scope global wg0
       valid_lft forever preferred_lft forever

default via 192.168.8.1 dev eth0.2 proto static src 192.168.8.252
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
192.168.8.0/24 dev eth0.2 proto kernel scope link src 192.168.8.252
192.168.219.0/24 dev wg0 proto kernel scope link src 192.168.219.2
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168.1.1
local 192.168.1.1 dev br-lan table local proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.168.1.1
broadcast 192.168.8.0 dev eth0.2 table local proto kernel scope link src 192.168.8.252
local 192.168.8.252 dev eth0.2 table local proto kernel scope host src 192.168.8.252
broadcast 192.168.8.255 dev eth0.2 table local proto kernel scope link src 192.168.8.252
broadcast 192.168.219.0 dev wg0 table local proto kernel scope link src 192.168.219.2
local 192.168.219.2 dev wg0 table local proto kernel scope host src 192.168.219.2
broadcast 192.168.219.255 dev wg0 table local proto kernel scope link src 192.168.219.2

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

interface: wg0
  public key: U3CXLF55rU27j5s7Sh4kO1nKqgflVHeaqbfRyP4COjA=
  private key: (hidden)
  listening port: 51555

peer: DioqUp7BoOx+6dtQe90FKFF6eb82c25noKy5XZvevhY=
  endpoint: 192.168.8.251:51555
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 1 minute, 54 seconds ago
  transfer: 230.53 MiB received, 2.12 GiB sent
  persistent keepalive: every 25 seconds

Have you tried removing option limit '5/sec' from this rule?

2 Likes

I will try. I think it should have no impact as the WG connection is already established, and this will AFAIK only rate limit unestablished initial UDP "connections".

Limiting a ACCEPT on UDP (stateless, no "establish") to 5 packets a second won't have an impact???

Try removing it.

1 Like

I actually don't think it will have an impact. The rule comes after the ACCEPT target for cstate RELATED,ESTABLISHED connections.

Why do you think it would have an impact?

I have removed the limit and restarted each router. No change.

Before going too far into potential troubleshooting, what is your end use case for WG? Are you planning to run a wireguard setup within your local network? Or is this a prelude to setting up an external tunnel?

2 Likes

Yes, I would eventually run this externally over the internet.

Then I would advise setting up the external tunnel first to see if the problem even exists when going externally. There's little point in trying to work out exactly where/why the issue is cropping up in a temporary setup.

1 Like

I have a similar tunnel running across the internet (two different ISPs) and have the same problem. Also have the same problem with a tunnel within a different test LAN.

Then it sounds like you're setting something up wrong. What are the setups for those other tunnels?

1 Like