PROBLEM: When I transfer a 10K chunk of data to a remote server through a test wireguard tunnel, the ssh connection fails. A 1K transfer works fine. A 10K transfer through the tunnel to either wg router also works fine. The 10K transfer works fine when I don't use the wg tunnel.
Background: I have configured two test routers on my LAN to build a wireguard tunnel between them. They establish the tunnel fine, and routing works well from one router, through the tunnel to the other router, and then out to the real world. Each wireguard interface has it's own firewall zone, and that seems to work well too. I can access the internet through the tunnel. I am using 19.07.3.
This feels like a MTU problem, but I am unclear what I can or should do to fix it. https://www.speedguide.net/analyzer.php says MTU = 1420 and MSS = 1380, which matches what I would expect from wireguard.
test@frooze:~/wireguard$ dd if=/dev/zero count=1 bs=10K | ssh remote dd of=/dev/null
1+0 records in
1+0 records out
10240 bytes (10 kB, 10 KiB) copied, 7.1e-05 s, 144 MB/s
Timeout, server remote not responding.
test@frooze:~/wireguard$ dd if=/dev/zero count=1 bs=1K | ssh remote dd of=/dev/null
1+0 records in
1+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 6.96e-05 s, 14.7 MB/s
2+0 records in
2+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0312847 s, 32.7 kB/s
What is your final goal?
It's unlikely that your real goal is transferring abstract data chunks.
I recommend to utilize some protocol with additional integrity control, e.g. rsync.
See also: Determine MTU size using ping
I have not yet found other failures. So far any large ssh upload through the wg tunnel will fail, including scp copies. Large downloads do not seem to fail.
It is having a big impact on my upload speed. Direct uploads to the internet are 30Mb/s against 3 Mb/s when they first go through the wireguard tunnel on my LAN.
The two routers are plugged into my normal LAN, since this is a test setup. They connect to the wider internet through the normal LAN's WAN connection.
The router configs are as follows, with no custom firewall.user file. I have blotted out the private keys, but the connection works as you can see from wg.
# router 264e
package network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fd8d:6c6a:9a32::/48'
config interface 'lan'
option type 'bridge'
option ifname 'eth0.1'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
config device 'lan_eth0_1_dev'
option name 'eth0.1'
option macaddr 'c4:71:54:3b:5b:44'
config interface 'wan'
option ifname 'eth0.2'
option proto 'dhcp'
config device 'wan_eth0_2_dev'
option name 'eth0.2'
option macaddr 'c4:71:54:3b:5b:45'
config interface 'wan6'
option ifname 'eth0.2'
option proto 'dhcpv6'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '1 2 3 4 6t'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '0 6t'
config interface 'wg0'
option proto 'wireguard'
option private_key 'xxx='
option listen_port '51555'
list addresses '192.168.219.1/24'
config wireguard_wg0 'wgpeer'
option persistent_keepalive '25'
list allowed_ips '0.0.0.0/0'
list allowed_ips '::/0'
option endpoint_port '51555'
option endpoint_host '192.168.8.252'
option public_key 'U3CXLF55rU27j5s7Sh4kO1nKqgflVHeaqbfRyP4COjA='
package firewall
config defaults
option syn_flood '1'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
config zone
option name 'lan'
list network 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
config zone
option name 'wan'
list network 'wan'
list network 'wan6'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option src_ip 'fc00::/6'
option dest_ip 'fc00::/6'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config include
option path '/etc/firewall.user'
config zone 'vpn'
option name 'vpn'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
option masq '1'
list network 'wg0'
config forwarding 'lan_vpn'
option src 'lan'
option dest 'vpn'
config forwarding 'vpn_wan'
option src 'vpn'
option dest 'wan'
config rule 'wg_inbound'
option name 'Allow-WG-Inbound'
option target 'ACCEPT'
option src '*'
option proto 'udp'
option dest_port '51555'
option limit '5/sec'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
valid_lft forever preferred_lft forever
6: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.8.251/24 brd 192.168.8.255 scope global eth0.2
valid_lft forever preferred_lft forever
8: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
inet 192.168.219.1/24 brd 192.168.219.255 scope global wg0
valid_lft forever preferred_lft forever
default via 192.168.219.2 dev wg0 table LocalWiFi
default via 192.168.8.1 dev eth0.2 proto static src 192.168.8.251
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
192.168.8.0/24 dev eth0.2 proto kernel scope link src 192.168.8.251
192.168.219.0/24 dev wg0 proto kernel scope link src 192.168.219.1
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168.1.1
local 192.168.1.1 dev br-lan table local proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.168.1.1
broadcast 192.168.8.0 dev eth0.2 table local proto kernel scope link src 192.168.8.251
local 192.168.8.251 dev eth0.2 table local proto kernel scope host src 192.168.8.251
broadcast 192.168.8.255 dev eth0.2 table local proto kernel scope link src 192.168.8.251
broadcast 192.168.219.0 dev wg0 table local proto kernel scope link src 192.168.219.1
local 192.168.219.1 dev wg0 table local proto kernel scope host src 192.168.219.1
broadcast 192.168.219.255 dev wg0 table local proto kernel scope link src 192.168.219.1
0: from all lookup local
32765: from 192.168.1.1/24 lookup LocalWiFi
32766: from all lookup main
32767: from all lookup default
interface: wg0
public key: DioqUp7BoOx+6dtQe90FKFF6eb82c25noKy5XZvevhY=
private key: (hidden)
listening port: 51555
peer: U3CXLF55rU27j5s7Sh4kO1nKqgflVHeaqbfRyP4COjA=
endpoint: 192.168.8.252:51555
allowed ips: 0.0.0.0/0, ::/0
latest handshake: 42 seconds ago
transfer: 2.12 GiB received, 230.25 MiB sent
persistent keepalive: every 25 seconds
Second router with almost identical config
# router 1e8b
package network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fdfa:0803:f6ca::/48'
config interface 'lan'
option type 'bridge'
option ifname 'eth0.1'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
config device 'lan_eth0_1_dev'
option name 'eth0.1'
option macaddr '7c:8b:ca:f6:71:d6'
config interface 'wan'
option ifname 'eth0.2'
option proto 'dhcp'
config device 'wan_eth0_2_dev'
option name 'eth0.2'
option macaddr '7c:8b:ca:f6:71:d7'
config interface 'wan6'
option ifname 'eth0.2'
option proto 'dhcpv6'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '1 2 3 4 6t'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '0 6t'
config interface 'wg0'
option proto 'wireguard'
option private_key '0xxxI0Q='
option listen_port '51555'
list addresses '192.168.219.2/24'
config wireguard_wg0 'wgpeer'
option persistent_keepalive '25'
list allowed_ips '0.0.0.0/0'
list allowed_ips '::/0'
option endpoint_port '51555'
option endpoint_host '192.168.8.251'
option public_key 'DioqUp7BoOx+6dtQe90FKFF6eb82c25noKy5XZvevhY='
package firewall
config defaults
option syn_flood '1'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
config zone
option name 'lan'
list network 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
config zone
option name 'wan'
list network 'wan'
list network 'wan6'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option src_ip 'fc00::/6'
option dest_ip 'fc00::/6'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config include
option path '/etc/firewall.user'
config zone 'vpn'
option name 'vpn'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
option masq '1'
list network 'wg0'
config forwarding 'lan_vpn'
option src 'lan'
option dest 'vpn'
config forwarding 'vpn_wan'
option src 'vpn'
option dest 'wan'
config rule 'wg_inbound'
option name 'Allow-WG-Inbound'
option target 'ACCEPT'
option src '*'
option proto 'udp'
option dest_port '51555'
option limit '5/sec'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
4: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
valid_lft forever preferred_lft forever
6: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
inet 192.168.8.252/24 brd 192.168.8.255 scope global eth0.2
valid_lft forever preferred_lft forever
8: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
inet 192.168.219.2/24 brd 192.168.219.255 scope global wg0
valid_lft forever preferred_lft forever
default via 192.168.8.1 dev eth0.2 proto static src 192.168.8.252
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1
192.168.8.0/24 dev eth0.2 proto kernel scope link src 192.168.8.252
192.168.219.0/24 dev wg0 proto kernel scope link src 192.168.219.2
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168.1.1
local 192.168.1.1 dev br-lan table local proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.168.1.1
broadcast 192.168.8.0 dev eth0.2 table local proto kernel scope link src 192.168.8.252
local 192.168.8.252 dev eth0.2 table local proto kernel scope host src 192.168.8.252
broadcast 192.168.8.255 dev eth0.2 table local proto kernel scope link src 192.168.8.252
broadcast 192.168.219.0 dev wg0 table local proto kernel scope link src 192.168.219.2
local 192.168.219.2 dev wg0 table local proto kernel scope host src 192.168.219.2
broadcast 192.168.219.255 dev wg0 table local proto kernel scope link src 192.168.219.2
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
interface: wg0
public key: U3CXLF55rU27j5s7Sh4kO1nKqgflVHeaqbfRyP4COjA=
private key: (hidden)
listening port: 51555
peer: DioqUp7BoOx+6dtQe90FKFF6eb82c25noKy5XZvevhY=
endpoint: 192.168.8.251:51555
allowed ips: 0.0.0.0/0, ::/0
latest handshake: 1 minute, 54 seconds ago
transfer: 230.53 MiB received, 2.12 GiB sent
persistent keepalive: every 25 seconds
I will try. I think it should have no impact as the WG connection is already established, and this will AFAIK only rate limit unestablished initial UDP "connections".
Before going too far into potential troubleshooting, what is your end use case for WG? Are you planning to run a wireguard setup within your local network? Or is this a prelude to setting up an external tunnel?
Then I would advise setting up the external tunnel first to see if the problem even exists when going externally. There's little point in trying to work out exactly where/why the issue is cropping up in a temporary setup.
I have a similar tunnel running across the internet (two different ISPs) and have the same problem. Also have the same problem with a tunnel within a different test LAN.