WAN router repeatedly crashing, nf_conntrack table full

Problem first occurred with a TP-Link Archer AX23 v1. I thought perhaps it had something to do with the hardware, so I replaced it with a TP-Link Archer C6 v3, but it had same problem.

After reboot it seems to run for about 24 hours, give or take, then suddenly the gateway router hangs. When I am able to watch the log, it appears to be full of lines like these:

[   28.334531] mt7530-mdio mdio-bus:1f wan: Link is Up - 1Gbps/Full - flow control off
[   28.901244] br-lan: port 6(phy1-ap0) entered blocking state
[   28.906891] br-lan: port 6(phy1-ap0) entered forwarding state
[   60.553756] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.560298] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.566668] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.572957] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.579288] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.585804] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.592178] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.598560] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.604830] nf_conntrack: nf_conntrack: table full, dropping packet
[   60.611415] nf_conntrack: nf_conntrack: table full, dropping packet
[   65.825287] net_ratelimit: 131 callbacks suppressed
[   65.825320] nf_conntrack: nf_conntrack: table full, dropping packet

I have two AX23 and one C6. One is used as wan gateway, the other two configured as "dumb" access points, as per https://openwrt.org/docs/guide-user/network/wifi/wifiextenders/bridgedap

root@OpenWrt:~# ubus call system board
{
	"kernel": "6.6.73",
	"hostname": "OpenWrt",
	"system": "MediaTek MT7621 ver:1 eco:3",
	"model": "TP-Link Archer C6 v3",
	"board_name": "tplink,archer-c6-v3",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "24.10.0",
		"revision": "r28427-6df0e3d02a",
		"target": "ramips/mt7621",
		"description": "OpenWrt 24.10.0 r28427-6df0e3d02a",
		"builddate": "1738624177"
	}
}
root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd2f:187a:e7ed::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan1'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.10.1'
	option netmask '255.255.255.0'
	option ip6assign '60'
	list dns '192.168.10.10'

config interface 'wan'
	option device 'wan'
	option proto 'dhcp'

config interface 'wan6'
	option device 'wan'
	option proto 'dhcpv6'

root@OpenWrt:~# cat /etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option localservice '1'
	option ednspacket_max '1232'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

root@OpenWrt:~# cat /etc/config/firewall
config defaults
	option syn_flood	1
	option input		REJECT
	option output		ACCEPT
	option forward		REJECT
# Uncomment this line to disable ipv6 rules
#	option disable_ipv6	1

config zone
	option name		lan
	list   network		'lan'
	option input		ACCEPT
	option output		ACCEPT
	option forward		ACCEPT

config zone
	option name		wan
	list   network		'wan'
	list   network		'wan6'
	option input		REJECT
	option output		ACCEPT
	option forward		REJECT
	option masq		1
	option mtu_fix		1

config forwarding
	option src		lan
	option dest		wan

# We need to accept udp packets on port 68,
# see https://dev.openwrt.org/ticket/4108
config rule
	option name		Allow-DHCP-Renew
	option src		wan
	option proto		udp
	option dest_port	68
	option target		ACCEPT
	option family		ipv4

# Allow IPv4 ping
config rule
	option name		Allow-Ping
	option src		wan
	option proto		icmp
	option icmp_type	echo-request
	option family		ipv4
	option target		ACCEPT

config rule
	option name		Allow-IGMP
	option src		wan
	option proto		igmp
	option family		ipv4
	option target		ACCEPT

# Allow DHCPv6 replies
# see https://github.com/openwrt/openwrt/issues/5066
config rule
	option name		Allow-DHCPv6
	option src		wan
	option proto		udp
	option dest_port	546
	option family		ipv6
	option target		ACCEPT

config rule
	option name		Allow-MLD
	option src		wan
	option proto		icmp
	option src_ip		fe80::/10
	list icmp_type		'130/0'
	list icmp_type		'131/0'
	list icmp_type		'132/0'
	list icmp_type		'143/0'
	option family		ipv6
	option target		ACCEPT

# Allow essential incoming IPv6 ICMP traffic
config rule
	option name		Allow-ICMPv6-Input
	option src		wan
	option proto	icmp
	list icmp_type		echo-request
	list icmp_type		echo-reply
	list icmp_type		destination-unreachable
	list icmp_type		packet-too-big
	list icmp_type		time-exceeded
	list icmp_type		bad-header
	list icmp_type		unknown-header-type
	list icmp_type		router-solicitation
	list icmp_type		neighbour-solicitation
	list icmp_type		router-advertisement
	list icmp_type		neighbour-advertisement
	option limit		1000/sec
	option family		ipv6
	option target		ACCEPT

# Allow essential forwarded IPv6 ICMP traffic
config rule
	option name		Allow-ICMPv6-Forward
	option src		wan
	option dest		*
	option proto		icmp
	list icmp_type		echo-request
	list icmp_type		echo-reply
	list icmp_type		destination-unreachable
	list icmp_type		packet-too-big
	list icmp_type		time-exceeded
	list icmp_type		bad-header
	list icmp_type		unknown-header-type
	option limit		1000/sec
	option family		ipv6
	option target		ACCEPT

config rule
	option name		Allow-IPSec-ESP
	option src		wan
	option dest		lan
	option proto		esp
	option target		ACCEPT

config rule
	option name		Allow-ISAKMP
	option src		wan
	option dest		lan
	option dest_port	500
	option proto		udp
	option target		ACCEPT

The wan connection is a FTTH-DHCP router from the provider, 1gb link up/down.

https://www.suse.com/support/kb/doc/?id=000020149 might be useful.

Thanks!!

root@OpenWrt:/etc/sysctl.d# cat /proc/sys/net/netfilter/nf_conntrack_max
15360
root@OpenWrt:/etc/sysctl.d# cat /proc/sys/net/netfilter/nf_conntrack_count
15360

I bumped it up by 10x, hopefully that helps.

What is a normal value? I don't run Kubernetes or microservices, just the usual home network environment + a homelab with a handful of docker services.

10x nooo do calculation, it takes 200+some bytes fixed resident slXb per slot.
64MB (half of ram?)/250 approx 250000 entries.

To be continued

Do you have public ip or kind of cgnat?
Ip6?
Gamer?

Sorry I don't really understand what you mean. 153600 is not enough? Should I make it 250000?

Nominally it was supposed to be CGNAT, but as the provided was unable to provide IPv6 they gave me a standard dynamic NAT IP.

I play a few games, but nothing serious and nothing hosted.

Just that 250k is upper limit.
You csn save your limit in /etc/sysctl.conf

net.netfilter.nf_conntrack_max = 150000

If it is stable with bigger number do

conntrack -L | tee -a /tmp/conntrack.out

(takes some time to list 100k entries)
And use some office spreadsheet to understand what is dominant state. Id guess udp unreplied.

Just to close off the story. Turns out I had a faulty script for a persistent reverse SSH tunnel, that kept creating new connections. With that fixed, /proc/sys/net/netfilter/nf_conntrack_count is now 96 :wink:
Thanks for help and hints!

3 Likes

You dosed yourself? Nice :sunglasses:

1 Like

I guess you could say that - unintentionally though :rofl:

1 Like