LuCI session instantly expiring with bridged ZeroTier interface

CamoYoshi · February 9, 2024, 4:32am

Hello all,

I've got a particular use-case and possible bug I've found, maybe?

Here's the scenario:

I have a device that is very old - So old it doesn't even use TCP/IP; it uses Ethernet II Broadcast frames. I am using ZeroTier to connect to another like device at another location over the internet, using their Ethernet Bridging function. Hence, I bridge the ZeroTier and eth0.1 interfaces together and plug my device into the switch ports on my router, which is a Netgear WNDR4300, running OpenWrt 23.05.2. Everything is fine until I commit the config change to bridge the ZeroTier interface with eth0.1. After that, every 5-30 seconds I get kicked out of LuCI with a message saying my session has expired. Load on the router according to LuCI shows around 0.35-0.55 - Perhaps a bit high but nothing too crazy I'd feel like. Additionally, another symptom is that pinging the router's local IP via a Gigabit Ethernet connection on the switch side gets me about 20-40 packets with sub millisecond latency as one would expect, but then randomly I get anywhere from 50-1000ms, before going back to normal. These older devices are only 10Mbit Half Duplex, if that is of any relevance. I am happy to share my config backup to see if anyone can replicate. So far I have tried:

Different locations for upstream connections to internet (WiFi, two separate physical locations with different internet providers)

Different devices of the same model (both experience the same behavior when using the same config)

Different Ethernet cabling to connect to the endpoint devices

Different endpoint devices

Different router power bricks for each of the routers

Based on the fact that I can replicate this on multiple devices I am guessing it could be a bug? Or I am doing something terrible that OpenWrt or the device model I have selected just simply cannot cope; if so I would love suggestions for very cheap (~20-40USD range) devices that could handle this usage.

Any suggestions on what to try are welcome - and as mentioned I am happy to share my config here.

Thank you in advance!

LilRedDog · February 9, 2024, 5:30am

Half duplex induce, a lot of, collisions (even more when a protocol is not set up to deal with them) and while 1-10MBs networks expected them and just dealt with them (by establishing a take a turn approach) modern networks may not play nice with collisions/half duplex.

Packets that are resent need to be buffered and put back in order.

I can imagine this could get packets so far out of the correct time/order something could believe more time has passed than actually does. Or just , flat out, confuse modern networks.

It was the first thing that comes to mind.

LilRedDog · February 9, 2024, 5:57am

On second reading:

You are bridging wan and lan?
eth0 is usually wan and eth1 is usually on the lan.

Do I have that right?

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:
ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

CamoYoshi · February 9, 2024, 7:19am

BusyBox v1.36.1 (2023-11-14 13:38:11 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 23.05.2, r23630-842932a63d
 -----------------------------------------------------
root@CS-SFRTR-VPN-8441:~# ubus call system board
{
	"kernel": "5.15.137",
	"hostname": "CS-SFRTR-VPN-8441",
	"system": "Atheros AR9344 rev 2",
	"model": "Netgear WNDR4300",
	"board_name": "netgear,wndr4300",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.2",
		"revision": "r23630-842932a63d",
		"target": "ath79/nand",
		"description": "OpenWrt 23.05.2 r23630-842932a63d"
	}
}
root@CS-SFRTR-VPN-8441:~# cat /etc/config/wireless 

config wifi-device 'radio0'
	option type 'mac80211'
	option path 'platform/ahb/18100000.wmac'
	option channel '1'
	option band '2g'
	option htmode 'HT20'
	option cell_density '0'
	option disabled '1'

config wifi-device 'radio1'
	option type 'mac80211'
	option path 'pci0000:00/0000:00:00.0'
	option channel '36'
	option band '5g'
	option htmode 'HT20'
	option cell_density '0'

config wifi-iface 'wifinet1'
	option device 'radio0'
	option mode 'sta'
	option network 'wwan'
	option ssid 'WIRELESS-SSID'
	option encryption 'psk2'
	option key 'PASSWORD'
	option disabled '1'

config wifi-iface 'wifinet2'
	option device 'radio1'
	option mode 'ap'
	option ssid 'WIRELESS-SSID'
	option encryption 'sae-mixed'
	option key 'PASSWORD'
	option network 'lan'

root@CS-SFRTR-VPN-8441:~# cat /etc/config/network 

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd08:82e0:ee11::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	option bridge_empty '1'
	list ports 'eth0.1'
	list ports 'ztppiu5pmj'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '10.19.97.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config device
	option name 'eth0.2'
	option macaddr '44:94:fc:90:a2:31'

config interface 'wan'
	option device 'eth0.2'
	option proto 'dhcp'

config interface 'wan6'
	option device 'eth0.2'
	option proto 'dhcpv6'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '1 2 3 4 0t'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '5 0t'

config interface 'wwan'
	option proto 'dhcp'

root@CS-SFRTR-VPN-8441:~# cat /etc/config/dhcp 

config dnsmasq
	option domainneeded '1'
	option boguspriv '1'
	option filterwin2k '0'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/lan/'
	option domain 'lan'
	option expandhosts '1'
	option nonegcache '0'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option nonwildcard '1'
	option localservice '1'
	option ednspacket_max '1232'
	option filter_aaaa '0'
	option filter_a '0'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

root@CS-SFRTR-VPN-8441:~# cat /etc/config/firewall 

config defaults
	option syn_flood '1'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'
	list network 'wan6'
	list network 'wwan'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

LilRedDog · February 9, 2024, 7:40am

I thought you were, maybe, using eth0.1 as shorthand for wth0 and eth1.

So, I'm going back to "I can see half duplex causing issues" and add "especially when bridged with a full duplex network."

I would ask if you can you set a port to half duplex and force the router to half duplex? But you are using vlans so I don't see that working.

CamoYoshi · February 9, 2024, 8:36am

The VLAN config is what came with the default install of OpenWrt - I did not change anything. I can certainly try to remove it and bridge directly to the switch - as well as try to set the ports to run at half duplex where the endpoint devices are connected. If you have any other thoughts of things to try I'm all ears.

LilRedDog · February 9, 2024, 1:16pm

I'd try setting the ports to half duplex first; that is the easiest thing to undo.

jow · February 9, 2024, 1:23pm

LuCI should not care at all about your underlying network setup. The behavior sounds more like Browser side confusion to me. Maybe you do have two OpenWrt installations with the same IP or DNS name in your network and your browser is flapping between those? When you present LuCI with an invalid authentication cookie (e.g. one which was created by another LuCI instance on another router), you will see that "session expired" message.