Strange DNS behavior with a particular machine connected to Raspberry Pi 4 router

[daigennki]

Hi. Got some more Raspberry Pi 4 weirdness.

I have a RPi4 running OpenWrt 24.10.5 as a router/firewall, and on the LAN side (connected to the RPi4's built-in gigabit NIC) I have a ThinkCentre M720q running Alpine Linux.

When doing some DNS queries from the M720q, it never receives a reply from dnsmasq on the RPi4. After investigating using tcpdump and Wireshark, I found that while there is nothing wrong with the DNS queries going out of the M720q, the queries appear to get corrupted right when it reaches the RPi4. The last 26 bytes of the 68-byte "AAAA" query packet gets stuck onto the end of the 68-byte "A" query packet, and the AAAA query packet never shows up to tcpdump on the RPi4. This causes the checksum for the A query to fail, thus dropping the packet. If RX checksum offloading is disabled with ethtool on the RPi4's eth0, the packets no longer get dropped. The packets still appear corrupted to tcpdump, but dnsmasq is able to reply anyways.

I've tested this with various combinations of hardware, such as eliminating all network switches in between the M720q and the RPi4 (leaving just a short ethernet cable between the two), and connecting a USB NIC to the M720q, and it still happens. This consistently occurs as long as any interface on the Alpine M720q is connected to the RPi4's built-in NIC; if using a USB NIC connected to a USB port on the RPi4 instead, this issue does not occur.

I've also tested connecting the M720q to a gaming laptop running Bazzite, then using the "Shared to other computers" option in KDE network settings for its wired ethernet interface (RTL8125), thus making the laptop act as a router (it's connected to the internet via Wi-Fi). That works just fine for any DNS query, so it's almost certainly an issue with the RPi4's built-in NIC.

I should also elaborate on only "some" queries from the M720q failing. By the looks of it, UDP DNS query packets with the "don't fragment" flag set consistently get corrupted on the RPi4. This is the case when I try to use pip install on the M720q, which queries for pypi.org. Meanwhile, this is not the case when I do nslookup pypi.org on the M720q; those do not have "don't fragment" set, and work just fine.

What in the world is going on here?

[anon63541380]

Likely you need to check "mss fixup" in firewall on the wan side.

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button (red circle; this works best in the 'Markdown' composer view in the blue oval):

Remember to redact passwords, VPN keys, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

[daigennki]

How do I check "mss fixup" in firewall? Also, I'd like to emphasize that this seems to be an issue within the LAN, between the RPi4 router and the M720q running Alpine.

Here are the configs you asked for.

# ubus call system board
{
        "kernel": "6.6.119",
        "hostname": "openwrt-rpi4",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 4 Model B Rev 1.5",
        "board_name": "raspberrypi,4-model-b",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "24.10.5",
                "revision": "r29087-d9c5716d1d",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 24.10.5 r29087-d9c5716d1d",
                "builddate": "1766005702"
        }
}

# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option packet_steering '2'
        option ula_prefix 'fd01::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        option bridge_empty '1'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.10.1'
        option netmask '255.255.255.0'
        option delegate '0'
        option ip6assign '64'
        list ip6class 'wan6'

config interface 'wan'
        option proto 'dhcp'
        option device 'eth1'
        option hostname '*'
        option peerdns '0'

config device
        option name 'eth1'

config interface 'wg0'
        option proto 'wireguard'
        option private_key '[redacted]'
        option listen_port '[redacted]'
        option delegate '0'
        list addresses '[redacted]'

config interface 'wan6'
        option proto 'dhcpv6'
        option device 'eth1'
        option reqaddress 'none'
        option reqprefix 'auto'
        option norelease '1'
        option peerdns '0'

config wireguard_wg0
        option description '[redacted]'
        option public_key '[redacted]'
        option private_key '[redacted]'
        option preshared_key '[redacted]'
        list allowed_ips '[redacted]'

config wireguard_wg0
        option public_key '[redacted]'
        option private_key '[redacted]'
        option description '[redacted]'
        option preshared_key '[redacted]'
        list allowed_ips '[redacted]'

# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option localservice '1'
        option ednspacket_max '1232'
        option confdir '/tmp/dnsmasq.d'
        list server '127.0.0.1#5053'
        list interface 'lan'
        list interface 'wg0'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        list dhcp_option '43,01:04:00:00:00:02'
        option ra 'server'
        list ra_flags 'none'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'
        option start '100'
        option limit '150'
        option leasetime '12h'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'
        option piofolder '/tmp/odhcpd-piofolder'

config dhcp 'wan6'
        option interface 'wan6'
        option ignore '1'
        option master '1'

# cat /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option drop_invalid '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wg0'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'wg0'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config redirect 'adblock_lan53'
        option name 'Adblock DNS (lan, 53)'
        option src 'lan'
        option proto 'tcp udp'
        option src_dport '53'
        option dest_port '53'
        option target 'DNAT'
        option family 'any'

config redirect 'adblock_lan853'
        option name 'Adblock DNS (lan, 853)'
        option src 'lan'
        option proto 'tcp udp'
        option src_dport '853'
        option dest_port '853'
        option target 'DNAT'
        option family 'any'

config redirect 'adblock_lan5353'
        option name 'Adblock DNS (lan, 5353)'
        option src 'lan'
        option proto 'tcp udp'
        option src_dport '5353'
        option dest_port '5353'
        option target 'DNAT'
        option family 'any'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'SRCDS'
        option family 'ipv4'
        option src 'wan'
        option src_dport '27016'
        option dest_ip '192.168.10.30'
        option dest_port '27016'
        option enabled '0'

config rule
        option name 'Allow-Wireguard'
        list proto 'udp'
        option src 'wan'
        option dest_port '[redacted]'
        option target 'ACCEPT'
        option family 'ipv4'

config forwarding
        option src 'wg0'
        option dest 'wan'

Wireless should be irrelevant as I use separate APs and don't use the RPi4's Wi-Fi, but here it is anyways:

# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1'
        option band '2g'
        option channel '1'
        option htmode 'HT20'
        option cell_density '0'

[anon63541380]

@daigennki

config zone
        option name 'wg0'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'wg0'

locate wg0 zone and find the checkbox. (press "EDIT" on it.

[daigennki]

Do you mean the "MSS clamping" checkbox? I checked it and it didn't change anything. Also, this is a UDP DNS issue, which MSS clamping seems unrelated to.

Also, once again, this seems very much like an issue specific to the LAN zone. I don't see how my Wireguard setup has anything to do with this. I only use Wireguard as a VPN to connect to this firewall from outside, and don't use it within LAN.

[daigennki]

For the record, just now I upgraded the RPi4 to the recently released OpenWrt 25.12.0, but this issue still persists in the same way.

[anon32365022]

Have you tried using another OS on the Raspberry Pi in an attempt to rule out the hardware?

[daigennki]

Not yet, since it's a router actively in use. What's a good way to quickly test this with another OS? If there's anything more I can try to rule out the hardware from within OpenWrt then I would prefer to do that first.

Also, as a workaround (or a legit fix?), turns out it works fine if I set nameserver in Alpine's /etc/resolv.conf to the router's IPv6 address. Even a link-local address (e.g. fe80::1234:56ff:fe78:9abc%eth0) works fine. I guess IPv4 UDP DNS queries are problematic somehow.

[anon32365022]

config redirect 'adblock_lan53'
        option name 'Adblock DNS (lan, 53)'
        option src 'lan'
        option proto 'tcp udp'
        option src_dport '53'
        option dest_port '53'
        option target 'DNAT'
        option family 'any'

config redirect 'adblock_lan853'
        option name 'Adblock DNS (lan, 853)'
        option src 'lan'
        option proto 'tcp udp'
        option src_dport '853'
        option dest_port '853'
        option target 'DNAT'
        option family 'any'

config redirect 'adblock_lan5353'
        option name 'Adblock DNS (lan, 5353)'
        option src 'lan'
        option proto 'tcp udp'
        option src_dport '5353'
        option dest_port '5353'
        option target 'DNAT'
        option family 'any'

What adblock software are you using? These rules, remove them (at least for now). Any better?

[daigennki]

Those rules actually disappeared when I upgraded to 25.12, but it hasn't gotten any better.

[anon32365022]

Post your current configs please.

Edit:

Does disabling adblock have any affect?

[daigennki]

Here are the current configs:

# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option packet_steering '2'
        option ula_prefix 'fd01::/48'
        option dhcp_default_duid '[redacted]'

config device
        option name 'br-lan'
        option type 'bridge'
        option bridge_empty '1'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.10.1'
        option netmask '255.255.255.0'
        option delegate '0'
        option ip6assign '64'
        list ip6class 'wan6'

config interface 'wan'
        option proto 'dhcp'
        option device 'eth1'
        option hostname '*'
        option peerdns '0'

config device
        option name 'eth1'

config interface 'wg0'
        option proto 'wireguard'
        option private_key '[redacted]'
        option listen_port '[redacted]'
        option delegate '0'
        list addresses '[redacted]'

config interface 'wan6'
        option proto 'dhcpv6'
        option device 'eth1'
        option reqaddress 'none'
        option reqprefix 'auto'
        option norelease '1'
        option peerdns '0'

config wireguard_wg0
        option description '[redacted]'
        option public_key '[redacted]'
        option private_key '[redacted]'
        option preshared_key '[redacted]'
        list allowed_ips '[redacted]'

config wireguard_wg0
        option public_key '[redacted]'
        option private_key '[redacted]'
        option description '[redacted]'
        option preshared_key '[redacted]'
        list allowed_ips '[redacted]'

# cat /etc/config/dhcp

config dnsmasq
        option domainneeded '1'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option localservice '1'
        option ednspacket_max '1232'
        option confdir '/tmp/dnsmasq.d'
        list server '127.0.0.1#5053'
        list interface 'lan'
        list interface 'wg0'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        list dhcp_option '43,01:04:00:00:00:02'
        option ra 'server'
        list ra_flags 'none'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'
        option start '100'
        option limit '150'
        option leasetime '12h'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/odhcpd.leases'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'
        option piodir '/tmp/odhcpd-piodir'
        option hostsdir '/tmp/hosts'

# cat /etc/config/firewall

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option drop_invalid '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'
        option mtu_fix '1'

config zone
        option name 'wg0'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'wg0'
        option mtu_fix '1'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'SRCDS'
        option family 'ipv4'
        option src 'wan'
        option src_dport '27016'
        option dest_ip '192.168.10.30'
        option dest_port '27016'
        option enabled '0'

config rule
        option name 'Allow-Wireguard'
        list proto 'udp'
        option src 'wan'
        option dest_port '20158'
        option target 'ACCEPT'
        option family 'ipv4'

config forwarding
        option src 'wg0'
        option dest 'wan'

Disabling the Adblock service (from the adblock OpenWrt package) had no effect.

[dave14305]

@daigennki

list dhcp_option '43,01:04:00:00:00:02'

What is this for?

[anon32365022]

@daigennki

    list dhcp_option '43,01:04:00:00:00:02'

Vendor code??

[daigennki]

Yeah, code to tell Windows machines to disable NetBIOS.

[dibdot]

@daigennki

option confdir '/tmp/dnsmasq.d'

Remove this option from your dnsmasq section, it's no longer used by current adblock and may cause problems ... restart your router afterwards.

[daigennki]

Hmm, removed that and the problem still occurs.

Also, I tested this with a fresh 25.12 image on a different SD card and the problem occurs there too. At this point I'm somewhat convinced it's a hardware issue. I did also test this with my laptop by booting the Alpine install media and running nslookup pypi.org; somehow that particular command is broken there even though it isn't on the proper install. Meanwhile nslookup www.alpinelinux.org works fine.

Unfortunately, I don't quite have either the time or motivation to go after this right now, especially now that I found the IPv6 workaround to work. If anybody else wants to try and reproduce it (thus definitely proving my RPi4 hardware being or not being defective), here's what you'll need:

• Raspberry Pi 4B running OpenWrt (my unit has 2GB RAM, though I'm not sure it matters)
  • I used an old 8GB microSD, but different SD cards don't seem to make a difference
• Any x86 machine running Alpine (it doesn't need to be installed; just a USB install media is enough)
  • Somehow this doesn't affect a RPi3B running Alpine

Connect an ethernet cable to the RPi4's built-in ethernet port, and the other end can be connected to any ethernet port on the x86 machine. Boot the Alpine install media on the x86 machine, and run either nslookup pypi.org or nslookup pool.ntp.org (NTP seems to be failing in the same way as pip install).

[daigennki]

Update to this: It appears that there is a serious amount of packet loss on the RPi4 when eth0 receives data, but not when it sends data. Confirmed with iperf3 between the RPi4 and my Fedora desktop. It can reliably do full gigabit when sending from the RPi4 to my Fedora desktop, but it can barely even do 20 Mbps in the other direction, and it does a lot of retries (about 30 retries per second) while doing so. I've tried disabling RX offloading with ethtool -K eth0 rx off, but that didn't help.

As such, this is very unlikely to be DNS-related, and more to do with the RPi4's eth0 being unreliable for receiving data. Perhaps it's the real reason why I was having this issue earlier: Raspberry Pi eth0 instability due to EEE being enabled by default - #5 by daigennki

[anon32365022]

eth0 - is that the Pi’s built in Broadcom port or the USB dongle?

[daigennki]

That's the built-in Broadcom interface.