Help understanding wireshark log

Installing and Using OpenWrt
1

Hello, I have an interesting case with a dropped packet.

I have a setup with two subnets (site A 192.168.1.0/24 and site B 192.168.2.0/24) connected over Wireguard. Hosts on both sites can ping each other and send small mqtt messages back and forth.

That makes me think it’s not a routing or firewall issue.

Hosts on site B can browse servers on site A no problem. But hosts on site A can not browse servers on site B. I’ve tracked down the TCP connection so I have all packets in the conversation, I’m not sure how to interpret them though.

MTU on both tunnel interfaces is 1280 (I had 1420 before and the behaviour was the same), both interfaces on both routers live in their own firewall zone and MSS clamping is enabled on both zones in both routers.

This is what the connection attempt looks like captured on the lan interface on site B router.

image
image1350×101 10.5 KB

All packets except the last http response (1410 length) reach their destination.But this one is dropped and after that there are some retransmissions that never go anywhere either.

Does this indicate that the size of the response packet is to big for the tunnel (the caller announced MSS 1240 which sounds reasonable). One thing I don’t understand is that if I trace the exact same connection attempt from the host on site B the response packet is smaller (1210 if I remember correctly). So I’m not entirely sure why it shows up bigger when it reaches the router?

Any help on how to troubleshoot or make sense of this is greatly appreciated.

2

About MTU for PPPoE side substract 8 but do not go lower than 1280 if you have IPv6

Enable MSS clamping (MTU fix ) on the WG firewall zone.

But it could well be a config problem.

For setup instructions see:
WireGuard Server Setup Guide
There is also a paragraph about site-to-site setup

It will help if you share your configs on both sides, so please connect to your OpenWRT device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button

Remember to redact keys, passwords, MAC addresses and any public IP addresses you may have but do not redact private RFC 1918 IP addresses as that is not needed:

ubus call system board
cat /etc/config/network
cat /etc/config/dhcp
cat /etc/config/firewall
ip route show
ip -6 route show
ip route show table all
ip rule show
wg show
3

Thanks, ok, will do, wall of text incoming. :grinning_face:

4

site A router

ubus call system board
{
        "kernel": "6.6.119",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 4",
        "model": "GL.iNet GL-MT6000",
        "board_name": "glinet,gl-mt6000",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "24.10.5",
                "revision": "r29087-d9c5716d1d",
                "target": "mediatek/filogic",
                "description": "OpenWrt 24.10.5 r29087-d9c5716d1d",
                "builddate": "1766005702"
        }
}

cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix '<redacted>'
        option packet_steering '1'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'
        list ports 'lan5'
        option ipv6 '0'

config interface 'lan'
        option device 'br-lan.10'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        list dns '192.168.1.2'
        list dns_search 'lan'
        option delegate '0'

config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth1'
        option proto 'dhcpv6'

config interface 'wg0'
        option proto 'wireguard'
        option private_key '<redacted>'
        option listen_port '<redacted>'
        option delegate '0'
        list addresses '10.0.0.1/24'
        option defaultroute '0'

config wireguard_wg0
        option description 'beryl ax'
        option public_key '<redacted>'
        option private_key '<redacted>'
        option persistent_keepalive '25'
        list allowed_ips '10.0.0.2'

config wireguard_wg0
        option description 's25'
        option public_key '<redacted>'
        option private_key '<redacted>'
        list allowed_ips '10.0.0.3'
        option persistent_keepalive '25'

config wireguard_wg0
        option description 'laptop'
        option public_key '<redacted>'
        option private_key '<redacted>'
        list allowed_ips '10.0.0.4'
        option persistent_keepalive '25'

config interface 'wgclient1'
        option proto 'wireguard'
        option private_key '<redacted>'
        list addresses '<redacted>'
        list addresses '<redacted>'
        list dns '<redacted>'
        list dns '<redacted>'
        list dns '<redacted>'
        list dns '<redacted>'

config wireguard_wgclient1
        option description 'wgclient1.conf'
        option public_key '<redacted>'
        list allowed_ips '0.0.0.0/0'
        list allowed_ips '::/0'
        option endpoint_host '<redacted>'
        option endpoint_port '<redacted>'

config bridge-vlan
        option device 'br-lan'
        option vlan '10'
        list ports 'lan1:u*'
        list ports 'lan2:u*'
        list ports 'lan3:u*'
        list ports 'lan4:u*'
        list ports 'lan5'

config bridge-vlan
        option device 'br-lan'
        option vlan '20'
        list ports 'lan1:t'
        list ports 'lan5:u*'

config interface 'vlan20'
        option proto 'static'
        option device 'br-lan.20'
        option ipaddr '192.168.0.1'
        option netmask '255.255.255.0'
        option delegate '0'

config interface 'wg1'
        option proto 'wireguard'
        option private_key '<redacted>'
        option listen_port '<redacted>'
        list addresses '10.0.100.1/29'
        option delegate '0'
        option defaultroute '0'
        option mtu '1280'

config wireguard_wg1
        option description 'spitz ax'
        option public_key '<redacted>'
        option private_key '<redacted>'
        list allowed_ips '10.0.100.2/32'
        list allowed_ips '192.168.2.0/24'
        option endpoint_host '<redacted>'
        option endpoint_port '<redacted>'
        option persistent_keepalive '25'

config wireguard_wg0
        option description 'iphone'
        option public_key '<redacted>'
        option private_key '<redacted>'
        list allowed_ips '10.0.0.5'

config wireguard_wg0
        option description 'galaxy'
        option public_key '<redacted>'
        option private_key '<redacted>'
        list allowed_ips '10.0.0.6'
        option endpoint_host '<redacted>'
        option endpoint_port '<redacted>'

config wireguard_wg1
        option description '<redacted>'
        option public_key '<redacted>'
        option private_key '<redacted>'
        option route_allowed_ips '1'
        option endpoint_host '<redacted>'
        option endpoint_port '<redacted>'
        option persistent_keepalive '25'
        list allowed_ips '10.0.100.3'

config route
        option interface 'wg1'
        option target '192.168.2.0/24'

cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/home/'
        option domain 'home'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        list dhcp_option '6,192.168.1.2'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'
        option piofolder '/tmp/odhcpd-piofolder'

config dhcp 'vlan20'
        option interface 'vlan20'
        option start '100'
        option limit '150'
        option leasetime '12h'

cat /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'
        list network 'vlan20'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'wg0'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'wg0'
        option mtu_fix '1'
        option masq '1'

config rule
        option src 'wan'
        option name 'Allow-Wireguard'
        list proto 'udp'
        option dest_port '<redacted> <redacted>'
        option target 'ACCEPT'

config forwarding
        option src 'wg0'
        option dest 'lan'

config forwarding
        option src 'wg0'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'wg0'

config zone
        option name 'wgwan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'wgclient1'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wgwan'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'Allow service'
        option src 'wgwan'
        option src_dport '<redacted>'
        option dest_ip '192.168.1.50'
        option dest_port '<redacted>'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'Allow service2'
        list proto 'tcp'
        option src 'wan'
        option src_dport '<redacted>'
        option dest_ip '192.168.1.50'
        option dest_port '<redacted>'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'Allow service3'
        option src 'wan'
        option src_dport '<redacted>'
        option dest_ip '192.168.1.50'
        option dest_port '<redacted>'

config zone
        option name 'wg1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'wg1'
        option mtu_fix '1'

config forwarding
        option src 'wg1'
        option dest 'lan'

config forwarding
        option src 'wg1'
        option dest 'wan'

config forwarding
        option src 'wg1'
        option dest 'wg0'

config forwarding
        option src 'lan'
        option dest 'wg1'

config forwarding
        option src 'wg0'
        option dest 'wg1'

ip route show

default via <redacted> dev eth1 proto static src <redacted>
10.0.0.0/24 dev wg0 proto kernel scope link src 10.0.0.1
10.0.100.0/29 dev wg1 proto kernel scope link src 10.0.100.1
10.0.100.3 dev wg1 proto static scope link
<redacted> dev eth1 proto kernel scope link src <redacted>
192.168.0.0/24 dev br-lan.20 proto kernel scope link src 192.168.0.1
192.168.1.0/24 dev br-lan.10 proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev wg1 proto static scope link
<redacted> via <redacted> dev eth1 proto static

ip -6 route show

fd00:0:1337:cafe:1111:1111:1a2d:460 dev wgclient1 proto kernel metric 256 pref medium
fd46:609:777d::/64 dev br-lan.10 proto static metric 1024 pref medium
unreachable fd46:609:777d::/48 dev lo proto static metric 2147483647 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev br-lan.10 proto kernel metric 256 pref medium
fe80::/64 dev br-lan.20 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium

ip route show table all

default via <redacted> dev eth1 table pbr_wan
10.0.0.0/24 dev wg0 table pbr_wan proto kernel scope link src 10.0.0.1
10.0.100.0/29 dev wg1 table pbr_wan proto kernel scope link src 10.0.100.1
10.0.100.3 dev wg1 table pbr_wan proto static scope link
192.168.0.0/24 dev br-lan.20 table pbr_wan proto kernel scope link src 192.168.0.1
192.168.1.0/24 dev br-lan.10 table pbr_wan proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev wg1 table pbr_wan proto static scope link
default via <redacted> dev wgclient1 table pbr_wgclient1
10.0.0.0/24 dev wg0 table pbr_wgclient1 proto kernel scope link src 10.0.0.1
10.0.100.0/29 dev wg1 table pbr_wgclient1 proto kernel scope link src 10.0.100.1
10.0.100.3 dev wg1 table pbr_wgclient1 proto static scope link
192.168.0.0/24 dev br-lan.20 table pbr_wgclient1 proto kernel scope link src 192.168.0.1
192.168.1.0/24 dev br-lan.10 table pbr_wgclient1 proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev wg1 table pbr_wgclient1 proto static scope link
default via <redacted> dev eth1 proto static src <redacted>
10.0.0.0/24 dev wg0 proto kernel scope link src 10.0.0.1
10.0.100.0/29 dev wg1 proto kernel scope link src 10.0.100.1
10.0.100.3 dev wg1 proto static scope link
<redacted> dev eth1 proto kernel scope link src <redacted>
192.168.0.0/24 dev br-lan.20 proto kernel scope link src 192.168.0.1
192.168.1.0/24 dev br-lan.10 proto kernel scope link src 192.168.1.1
192.168.2.0/24 dev wg1 proto static scope link
<redacted> via <redacted> dev eth1 proto static
local 10.0.0.1 dev wg0 table local proto kernel scope host src 10.0.0.1
broadcast 10.0.0.255 dev wg0 table local proto kernel scope link src 10.0.0.1
local 10.0.100.1 dev wg1 table local proto kernel scope host src 10.0.100.1
broadcast 10.0.100.7 dev wg1 table local proto kernel scope link src 10.0.100.1
local <redacted> dev eth1 table local proto kernel scope host src <redacted>
broadcast <redacted> dev eth1 table local proto kernel scope link src <redacted>
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local <redacted> dev wgclient1 table local proto kernel scope host src <redacted>
local 192.168.0.1 dev br-lan.20 table local proto kernel scope host src 192.168.0.1
broadcast 192.168.0.255 dev br-lan.20 table local proto kernel scope link src 192.168.0.1
local 192.168.1.1 dev br-lan.10 table local proto kernel scope host src 192.168.1.1
broadcast 192.168.1.255 dev br-lan.10 table local proto kernel scope link src 192.168.1.1
fd00:0:1337:cafe:1111:1111:1a2d:460 dev wgclient1 proto kernel metric 256 pref medium
fd46:609:777d::/64 dev br-lan.10 proto static metric 1024 pref medium
unreachable fd46:609:777d::/48 dev lo proto static metric 2147483647 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev br-lan.10 proto kernel metric 256 pref medium
fe80::/64 dev br-lan.20 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fd00:0:1337:cafe:1111:1111:1a2d:460 dev wgclient1 table local proto kernel metric 0 pref medium
anycast fd46:609:777d:: dev br-lan.10 table local proto kernel metric 0 pref medium
local fd46:609:777d::1 dev br-lan.10 table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev br-lan.20 table local proto kernel metric 0 pref medium
anycast fe80:: dev br-lan.10 table local proto kernel metric 0 pref medium
anycast fe80:: dev eth1 table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fea2:203f dev eth1 table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fea2:2041 dev eth0 table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fea2:2041 dev br-lan.20 table local proto kernel metric 0 pref medium
local fe80::9683:c4ff:fea2:2041 dev br-lan.10 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev br-lan.10 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev br-lan.20 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eth1 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wgclient1 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wg0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wg1 table local proto kernel metric 256 pref medium

ip rule show

0:      from all lookup local
29996:  from all sport <redacted> lookup pbr_wan
29997:  from all sport <redacted> lookup pbr_wan
29998:  from all fwmark 0x20000/0xff0000 lookup pbr_wgclient1
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
32766:  from all lookup main
32767:  from all lookup default

wg show

interface: wg0
  public key: <redacted>
  private key: (hidden)
  listening port: <redacted>

peer: <redacted>
  endpoint: <redacted>
  allowed ips: 10.0.0.3/32
  latest handshake: 1 day, 17 hours, 20 minutes, 35 seconds ago
  transfer: 33.60 MiB received, 242.39 MiB sent
  persistent keepalive: every 25 seconds

peer: <redacted>
  allowed ips: 10.0.0.2/32
  persistent keepalive: every 25 seconds

peer: <redacted>
  allowed ips: 10.0.0.4/32
  persistent keepalive: every 25 seconds

peer: <redacted>
  allowed ips: 10.0.0.5/32

peer: <redacted>
  endpoint: <redacted>
  allowed ips: 10.0.0.6/32

interface: wgclient1
  public key: <redacted>
  private key: (hidden)
  listening port: <redacted>

peer: <redacted>
  endpoint: <redacted>
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 1 minute, 41 seconds ago
  transfer: 13.89 GiB received, 26.31 GiB sent

interface: wg1
  public key: <redacted>
  private key: (hidden)
  listening port: <redacted>

peer: <redacted>
  endpoint: <redacted>
  allowed ips: 10.0.100.2/32, 192.168.2.0/24
  latest handshake: 21 seconds ago
  transfer: 2.83 GiB received, 161.37 MiB sent
  persistent keepalive: every 25 seconds

peer: <redacted>
  endpoint: <redacted>
  allowed ips: 10.0.100.3/32
  transfer: 0 B received, 4.51 MiB sent
  persistent keepalive: every 25 seconds
5

site B router

ubus call system board

{
        "kernel": "5.4.211",
        "hostname": "GL-X3000",
        "system": "ARMv8 Processor rev 4",
        "model": "GL.iNet GL-X3000",
        "board_name": "glinet,x3000-emmc",
        "release": {
                "distribution": "OpenWrt",
                "version": "21.02-SNAPSHOT",
                "revision": "",
                "target": "mediatek/mt7981",
                "description": "OpenWrt 21.02-SNAPSHOT "
        }
}

cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix '<redacted>'

config device
        option name 'br-lan'
        option type 'bridge'
        option macaddr '<redacted>'
        list ports 'eth1'
        list ports 'eth0'

config device
        option name 'eth1'
        option macaddr '<redacted>'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option isolate '0'
        option ipaddr '192.168.2.1'

config device
        option name 'eth0'
        option macaddr '<redacted>'

config interface 'wan'
        option proto 'dhcp'
        option force_link '0'
        option ipv6 '0'
        option classlessroute '0'
        option metric '10'

config interface 'wan6'
        option proto 'dhcpv6'
        option disabled '1'
        option device '@wan'

config interface 'tethering6'
        option proto 'dhcpv6'
        option disabled '1'
        option device '@tethering'

config interface 'wwan6'
        option proto 'dhcpv6'
        option disabled '1'
        option device '@wwan'

config interface 'guest'
        option force_link '1'
        option type 'bridge'
        option proto 'static'
        option ipaddr '192.168.9.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option multicast_querier '1'
        option igmp_snooping '0'
        option isolate '0'
        option bridge_empty '1'
        option disabled '1'

config interface 'wwan'
        option proto 'dhcp'
        option classlessroute '0'
        option metric '20'

config interface 'secondwan'
        option ipv6 '0'
        option proto 'dhcp'
        option metric '15'
        option force_link '0'
        option classlessroute '0'

config interface 'secondwan6'
        option proto 'dhcpv6'
        option disabled '1'
        option metric '15'
        option device '@secondwan'

config interface 'modem_0001_6'
        option proto 'dhcpv6'
        option disabled '1'
        option metric '40'
        option device '@modem_0001'

config rule 'policy_relay_lo_rt_lan'
        option lookup '16800'
        option in 'loopback'
        option priority '1'

config rule 'novpn_to_main'
        option gl_vpn_rules '1'
        option mark '0x8000/0xf000'
        option priority '6000'
        option lookup 'main'
        option disabled '0'

config rule 'vpn_to_main'
        option gl_vpn_rules '1'
        option mark '0x0/0xf000'
        option priority '9000'
        option lookup 'main'
        option invert '1'
        option disabled '0'

config rule 'vpn_leak_block'
        option gl_vpn_rules '1'
        option mark '0x0/0xf000'
        option priority '9910'
        option action 'blackhole'
        option invert '1'
        option disabled '0'

config rule 'vpn_block_lan_leak'
        option gl_vpn_rules '1'
        option in 'lan'
        option priority '9920'
        option action 'blackhole'
        option disabled '0'

config rule 'vpn_block_guest_leak'
        option gl_vpn_rules '1'
        option in 'guest'
        option priority '9920'
        option action 'blackhole'
        option disabled '0'

config rule 'vpn_block_wgserver_leak'
        option gl_vpn_rules '1'
        option in 'wgserver'
        option priority '9920'
        option disabled '0'
        option action 'blackhole'

config rule 'vpn_block_ovpnserver_leak'
        option gl_vpn_rules '1'
        option in 'ovpnserver'
        option priority '9920'
        option action 'blackhole'
        option disabled '0'

config interface 'modem_0001'
        option apn '<redacted>'
        option ip_type 'IPV4V6'
        option proto 'qcm'
        option device '/dev/mhi_QMI0'
        option metric '40'
        option roaming '1'
        option band_enable '0'
        option iccid '<redacted>'
        option disabled '0'

config interface 'wgclient1'
        option proto 'wgclient'
        option config 'peer_2001'
        option ip4table '1001'
        option ip6table '1001'
        option disabled '0'
        option mtu '1280'

config rule 'rt_wgclient1'
        option mark '0x1000/0xf000'
        option lookup '1001'
        option priority '6000'

config route 'wgclient1_blackhole'
        option target '0.0.0.0'
        option netmask '0.0.0.0'
        option interface 'loopback'
        option metric '254'
        option type 'blackhole'
        option table '1001'

config rule6 'rt_wgclient1_6'
        option mark '0x1000/0xf000'
        option lookup '1001'
        option priority '6000'

config route6 'wgclient1_blackhole_6'
        option target '::0/0'
        option interface 'loopback'
        option metric '25400'
        option type 'blackhole'
        option table '1001'

config rule 'main_static_net'
        option gl_vpn_rules '1'
        option suppress_prefixlength '0'
        option priority '800'
        option lookup '9910'
        option disabled '0'

config route
        option interface 'wgclient1'
        option target '192.168.1.0'
        option netmask '255.255.255.0'
        option table '1001'

cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'
        option ednspacket_max '1232'
        option rebind_protection '0'
        option filter_aaaa '1'
        option confdir '/tmp/dnsmasq.d'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option ra_slaac '1'
        option force '1'
        option dhcpv6 'disabled'
        option ra 'disabled'
        option ignore '0'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config domain
        option name '<redacted>'
        option ip '192.168.2.1'

config domain
        option name '<redacted>'
        option ip '::ffff:192.168.2.1'

config dhcp 'guest'
        option interface 'guest'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv6 'disabled'
        option ra 'disabled'

config dhcp 'secondwan'
        option interface 'secondwan'
        option ignore '1'

config dnsmasq 'wgclient1'
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '0'
        option boguspriv '0'
        option expandhosts '1'
        option cachesize '1000'
        option authoritative '1'
        option readethers '1'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.wgclient1'
        option ednspacket_max '1232'
        option confdir '/tmp/dnsmasq.d.wgclient1'
        option port '2153'
        option filter_aaaa '1'
        option user 'dnsmasq_vpn'
        list domain 'lan'
        list domain 'guest'
        list domain 'wan'
        list addnhosts '/tmp/hosts.vpn'
        option ignore_hosts_dir '1'
        option disabled '0'

config host
        option mac '<redacted>'
        option ip '192.168.2.135'
        option tag 'raspberrypi'

config host
        option mac '<redacted>'
        option ip '192.168.2.219'
        option tag 'SLZB-06'

cat /etc/config/firewall

config defaults
        option syn_flood '1'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'

config rule 'wan_drop_leaked_adgdns'
        option name 'wan_drop_leaked_adgdns'
        option src 'wan'
        option proto 'udp'
        option dest_port '3053'
        option mark '0x0/0xf000'
        option target 'DROP'
        option enabled '0'

config rule 'wan_drop_leaked_dns'
        option name 'wan_drop_leaked_dns'
        option src 'wan'
        option proto 'udp'
        option dest_port '53'
        option mark '!0x8000/0xf000'
        option target 'DROP'
        option enabled '0'

config rule 'guest_drop_leaked_dns'
        option name 'guest_drop_leaked_dns'
        option src 'guest'
        option proto 'udp'
        option dest_port '53'
        option mark '!0x8000/0xf000'
        option target 'DROP'
        option enabled '0'

config rule 'guest_drop_leak_adgdns'
        option name 'guest_drop_leak_adgdns'
        option src 'guest'
        option proto 'udp'
        option dest_port '3053'
        option mark '0x0/0xf000'
        option target 'DROP'
        option enabled '0'

config zone
        option name 'lan'
        list network 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        list network 'wwan'
        list network 'secondwan'
        list network 'modem_0001'
        option output 'ACCEPT'
        option forward 'REJECT'
        option mtu_fix '1'
        option input 'DROP'
        option masq '1'

config forwarding
        option src 'lan'
        option dest 'wan'
        option enabled '1'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Support-UDP-Traceroute'
        option src 'wan'
        option dest_port '33434:33689'
        option proto 'udp'
        option family 'ipv4'
        option target 'REJECT'
        option enabled 'false'

config include
        option path '/etc/firewall.user'

config include 'nat6'
        option path '/etc/firewall.nat6'
        option reload '1'

config zone
        option name 'guest'
        option network 'guest'
        option forward 'REJECT'
        option output 'ACCEPT'
        option input 'REJECT'

config forwarding
        option src 'guest'
        option dest 'wan'
        option enabled '1'

config rule
        option name 'Allow-DHCP'
        option src 'guest'
        option target 'ACCEPT'
        option proto 'udp'
        option dest_port '67-68'

config rule
        option name 'Allow-DNS'
        option src 'guest'
        option target 'ACCEPT'
        option proto 'tcp udp'
        option dest_port '53'

config forwarding 'wgclient2lan'
        option src 'wgclient'
        option dest 'lan'
        option enabled '1'

config include 'dns_order'
        option type 'script'
        option path '/etc/firewall.dns_order'
        option reload '1'
        option enabled '1'

config include 'vpnclient'
        option type 'script'
        option path '/usr/bin/rtp2.sh'
        option reload '0'

config include 'glblock'
        option type 'script'
        option path '/usr/bin/gl_block.sh'
        option reload '1'

config include 'dmz_exclude'
        option type 'script'
        option path '/etc/firewall.dmz.exclude'
        option reload '1'

config include 'security'
        option type 'script'
        option path '/etc/firewall.security'
        option reload '0'

config rule 'lan_drop_leaked_dns'
        option name 'lan_drop_leaked_dns'
        option src 'lan'
        option proto 'udp'
        option dest_port '53'
        option mark '!0x8000/0xf000'
        option target 'DROP'
        option enabled '1'

config rule 'lan_drop_leak_adgdns'
        option name 'lan_drop_leak_adgdns'
        option src 'lan'
        option proto 'udp'
        option dest_port '3053'
        option mark '0x0/0xf000'
        option target 'DROP'
        option enabled '1'

config rule 'wgserver_drop_leaked_dns'
        option name 'wgserver_drop_leaked_dns'
        option src 'wgserver'
        option proto 'udp'
        option dest_port '53'
        option mark '!0x8000/0xf000'
        option target 'DROP'
        option enabled '1'

config rule 'ovpnserver_drop_leaked_dns'
        option name 'ovpnserver_drop_leaked_dns'
        option src 'ovpnserver'
        option proto 'udp'
        option dest_port '53'
        option mark '!0x8000/0xf000'
        option target 'DROP'
        option enabled '1'

config rule 'wgserver_drop_leaked_adgdns'
        option name 'wgserver_drop_leaked_adgdns'
        option src 'wgserver'
        option proto 'udp'
        option dest_port '3053'
        option mark '0x0/0xf000'
        option target 'DROP'
        option enabled '1'

config rule 'ovpnserver_drop_leaked_adgdns'
        option name 'ovpnserver_drop_leaked_adgdns'
        option src 'ovpnserver'
        option proto 'udp'
        option dest_port '3053'
        option mark '0x0/0xf000'
        option target 'DROP'
        option enabled '1'

config rule 'port3456'
        option name 'Block External Port 3456'
        option dest_port '3456'
        option src 'lan'
        option proto 'tcp'
        option target 'DROP'
        option enabled '1'

config zone 'wgclient1'
        option name 'wgclient1'
        option forward 'ACCEPT'
        option output 'ACCEPT'
        option mtu_fix '1'
        option network 'wgclient1'
        option enabled '1'
        option input 'ACCEPT'
        option masq '1'
        option masq6 '1'

config forwarding 'lan2wgclient1'
        option src 'lan'
        option dest 'wgclient1'

config forwarding 'guest2wgclient1'
        option src 'guest'
        option dest 'wgclient1'

config forwarding 'wan2wgclient1'
        option src 'wan'
        option dest 'wgclient1'

config forwarding 'ovpnserver2wgclient1'
        option src 'ovpnserver'
        option dest 'wgclient1'

config forwarding 'wgserver2wgclient1'
        option src 'wgserver'
        option dest 'wgclient1'

config forwarding 'wgclient12lan'
        option src 'wgclient1'
        option dest 'lan'
        option enabled '1'

ip route show

default via <redacted> dev rmnet_mhi0 proto static src <redacted> metric 40
<redacted> dev rmnet_mhi0 proto static scope link metric 40
192.168.2.0/24 dev br-lan proto kernel scope link src 192.168.2.1

ip -6 route show

unreachable <redacted> dev lo proto static metric 2147483647 pref medium

ip route show table all

192.168.2.0/24 dev br-lan table 9910 proto kernel scope link src 192.168.2.1
default dev wgclient1 table 1001 proto static scope link
blackhole default table 1001 proto static metric 254
10.0.100.2 dev wgclient1 table 1001 proto static scope link
default via <redacted> dev rmnet_mhi0 proto static src <redacted> metric 40
<redacted> dev rmnet_mhi0 proto static scope link metric 40
192.168.2.0/24 dev br-lan proto kernel scope link src 192.168.2.1
local 10.0.100.2 dev wgclient1 table local proto kernel scope host src 10.0.100.2
broadcast <redacted> dev rmnet_mhi0 table local proto kernel scope link src <redacted>
local <redacted> dev rmnet_mhi0 table local proto kernel scope host src <redacted>
broadcast <redacted> dev rmnet_mhi0 table local proto kernel scope link src <redacted>
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.2.0 dev br-lan table local proto kernel scope link src 192.168.2.1
local 192.168.2.1 dev br-lan table local proto kernel scope host src 192.168.2.1
broadcast 192.168.2.255 dev br-lan table local proto kernel scope link src 192.168.2.1
blackhole default dev lo table 1001 proto static metric 25400 pref medium
unreachable <redacted> dev lo proto static metric 2147483647 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium

ip rule show

0:      from all lookup local
1:      from all iif lo lookup 16800
800:    from all lookup 9910 suppress_prefixlength 0
6000:   from all fwmark 0x8000/0xf000 lookup main
6000:   from all fwmark 0x1000/0xf000 lookup 1001
9000:   not from all fwmark 0/0xf000 lookup main
9910:   not from all fwmark 0/0xf000 blackhole
9920:   from all iif br-lan blackhole
10000:  from 10.0.100.2 lookup 1001
20000:  from all to 10.0.100.2 lookup 1001
32766:  from all lookup main
32767:  from all lookup default
90014:  from all iif lo lookup 1001

wg show

interface: wgclient1
  public key: <redacted>
  private key: (hidden)
  listening port: <redacted>
  fwmark: 0x8000

peer: <redacted>
  endpoint: <redacted>
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 1 minute, 39 seconds ago
  transfer: 162.16 MiB received, 2.84 GiB sent
  persistent keepalive: every 25 seconds
6

First something else, port lan5 is untagged in another bridge-vlan so it looks like it needs to be tagged here

About WireGuard , there seems to be a standard WG client 'wgclient1` no default route so that should not matter

Then there is a standard WG server wg0.
My advice add for clarity to the Allowed IPs /32, it is default so will work anyway.
Furthermore enable Route Allowed IPs for all peers, it works even without it because you have used /24 as mask for the address of the wg0 interface so this is more for clarity and redundancy

For the wg0 firewall remove the Masquerading, that is not necessary

Now on to wg1 which is the site-to-site setup:
It has a listen port set so I assume it can handle incoming connections.
Note that this listen port needs to be different from the listen port of wg0 and that both have a firewall rule to allow this incoming port, you did redact the ports so I cannot tell.
For the subnet use a /24 mask

For the peers enable Route Allowed IPs so that you can remove the following :

Note that the connection to the peer 10.0.100.3 is not working, not sure what is on that side but there is no one home

Unfortunately I do not think this will help as the problem is likely to be on the other side.
But that is using Software from GL.iNET, although it is an offshoot of a four year old OpenWRT fork, it is heavily modified so you have to ask at the GL.iNet forum we can only help if you use a current genuine OpenWRT build

7

First something else, port lan5 is untagged in another bridge-vlan so it looks like it needs to be tagged here

Thanks, missed that one.

Then there is a standard WG server wg0.
My advice add for clarity to the Allowed IPs /32, it is default so will work anyway.

I’m not sure what that means?

Furthermore enable Route Allowed IPs for all peers, it works even without it because you have used /24 as mask for the address of the wg0 interface so this is more for clarity and redundancy

I was contemplating between having a /24 or /32 on the wg interface vs having Route Allowed IPs checked or not. Is there any particular reason to choose one over the other. Personally I think it’s easier with fewer routes that’s why I chose /24 on the wg interface and unchecked Route Allowed IPs, but maybe I should rethink this, is there a reason to have both (i.e. /24 AND Route Allowed IPs).

For the wg0 firewall remove the Masquerading, that is not necessary

Was a bit unsure. Fixed it and seems to work just fine.

Now on to wg1 which is the site-to-site setup:
It has a listen port set so I assume it can handle incoming connections.
Note that this listen port needs to be different from the listen port of wg0 and that both have a firewall rule to allow this incoming port, you did redact the ports so I cannot tell.
For the subnet use a /24 mask

Yes, incoming for wg0 and wg1 is on different ports and it seems to work fine. Use /24 because it’s more standard I reckon? I was thinking a bit about having both site to site and road warrior (wg0) on the same interface but I’m not sure if that’s a good or bad idea so I split them. Any input on that?

For the peers enable Route Allowed IPs so that you can remove the following :

Will do.

I have a question about the peer config. All allowed IP:s are added to the peer configuration Interface/Addresses attribute, should all of them be there or just the peers wireguard tunnel interface IP? I.e. should I manually remove the other subnets before I use it on the peer or do they serve a purpose being there? I don’t understand why the peer Interface should have more than one address?

Unfortunately I do not think this will help as the problem is likely to be on the other side.
But that is using Software from GL.iNET, although it is an offshoot of a four year old OpenWRT fork, it is heavily modified so you have to ask at the GL.iNet forum we can only help if you use a current genuine OpenWRT build

I think you are right. Thanks alot for your effort anyway. My site A router is a Flint 2 I flashed with the latest vanilla OpenWRT and my Spitz AX (site B) also supports vanilla so I will flash that as soon as I can and hope that will fix the oddities I experience now.

I was just curious about how to troubleshoot this, would be interesting to learn how to do it, I find networking really hard to really understand correctly.

8

Look at the MSS sizes on the SYN and SYN,ACK response

192.168.1.186 sends an MSS of 1240
192.168.2.135 responds with an MSS of 1460 .. which is a problem if the tunnel MTU is 1280

9

I thought it could have something to do with MTU too but I’m not sure how to test it.

I have two different web servers on the remote LAN and the behaviour when trying to reach them is the same, I can ping them, but when I browse the first response packet is blocked.

The 192.168.2.135 response has the Don’t Fragment flag set so I would expect the packet to be dropped, but the other one (.219) hasn’t. But it doesn’t seem to make any difference. Not sure why neither adapts to the callers MSS though. Both responses are bigger than the tunnel MTU.

Any idea on how to test if it’s the MTU?

10

I should also add that the packet sizes are all over the place. I get different size if I measure using Wireshark (sshdump) on the router vs tcpdump on the host interface. Which is weird I guess, might be that they measure different envelopes. But nothing lines up.

This is what leaves the web host according to tcpdump (with the -e flag)
11:29:53.439596 <redacted> (oui Unknown) > <redacted> (oui Unknown), ethertype IPv4 (0x0800), length 1294: 192.168.2.135.http-alt > 192.168.1.186.61208: Flags [.], seq 1:1229, ack 83, win 509, options [nop,nop,TS val 774854861 ecr 1688151018], length 1228: HTTP: HTTP/1.1 200 OK

And this is what hits the router interface:

image
image842×105 40.4 KB

11

I also tried adding TRACE targets for the connection. But that output is pure gibberish for someone like me. Tried to get an LLM to make sense of it to no avail.

Is there any tool out there that can parse the dmesg TRACE log and make sense of the information?

12

At least on Windows, what you capture on the sending host might not be what goes out on the wire. I’ve seen >30KB packets in wireshark on the sending host, so obviously that isn’t going out as is ‘over the wire’. So yes, you might see “strange” results when capturing on the same host that is sending the packets.

For checking if you’ve got MTU problems, a simple test on a windows machine is ‘ping -f -l [length] destination’

-f : do not fragment

-l length

I’m on a linux box, so it’s a bit different but the same idea:

$ ping -c 5 -M do -s 1472 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 1472(1500) bytes of data.
1480 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=21.0 ms
1480 bytes from 8.8.8.8: icmp_seq=2 ttl=119 time=24.1 ms
1480 bytes from 8.8.8.8: icmp_seq=3 ttl=119 time=26.6 ms
1480 bytes from 8.8.8.8: icmp_seq=4 ttl=119 time=31.1 ms
1480 bytes from 8.8.8.8: icmp_seq=5 ttl=119 time=23.3 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 20.953/25.200/31.091/3.453 ms


$ ping -c 5 -M do -s 1473 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 1473(1501) bytes of data.
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4093ms
1 Like
13

Tunnel MTU 1280 - ipheader 20 - icmp header 8 = 1252.

If I ping with the size of 1252 it works, 1253 and no returns.

Lets say this is the issue, why isn’t the packet fragmented? One of the hosts doesn’t set the Don’t Fragment header, shouldn’t that traffic be fragmented and passed along?

And how would I fix this? It seems weird all hosts (I only have two right now) on the other LAN have the same issue.