Issues with routing in mesh network

fireburner · August 18, 2023, 9:03am

Hi community,

I finally was able to set up a mesh network between my Archer C50 and a Fritz Repeater 1200.
The Archer acts as router and has internet access on its wan port.
The Repeater is in another room and there is no wired connection.

Both devices have 2 SSIDs, one for 2.4 and one for 5. In addition the mesh is on the 2.4 for both of them.
All are bound to the lan bridge.
The router has an additional SSID in a different subnet.

I followed this documentation for setup: https://openwrt.org/docs/guide-user/network/wifi/mesh/80211s
(I also installed the mesh11sd package)

When booting up all seems to work fine. After a while (an hour or longer?), I can't ping/access the Repeater anymore from a client on the Archer. If I send the ping from the Archer itself, I am able to reach the repeater.
Same applies for the Clients on the Repeater, they can access the internet just fine, but can't ping/access clients on the Archer.
If I then ssh into the Archer and from there into the Repeater, run a ping to the Archer and a Client, all starts working again for a while until the same happens again.
I tried disabling firewall, odhcpd and dnsmasq, but that did not change anything.

Has anyone an idea, what could be wrong with the routing here?

/etc/config/network from Archer:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'xxxxx::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0.1'
        option stp '1'
        option ipv6 '1'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ip6assign '64'
        list dns '185.150.99.255'
        list dns '5.9.164.112'
        list dns '2001:678:ed0:f000::'
        list dns '2a01:4f8:251:554::2'
        list ipaddr '192.168.10.1/24'
        option ip6hint '10'
        option delegate '0'

config device
        option name 'eth0.2'
        option macaddr 'xxxxx'

config interface 'wan'
        option device 'eth0.2'
        option proto 'dhcp'
        option peerdns '0'
        list dns '185.150.99.255'
        list dns '5.9.164.112'

config interface 'wan6'
        option device 'eth0.2'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix '60'
        option peerdns '0'
        list dns '2001:678:ed0:f000::'
        list dns '2a01:4f8:251:554::2'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '1 2 3 4 6t'
        option vid '1'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0 6t'
        option vid '2'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option vid '41'
        option ports '6t'

config interface 'IOT'
        option proto 'static'
        option ipaddr '192.168.41.1'
        option netmask '255.255.255.0'
        option type 'bridge'
        option device 'iot-br'
        option delegate '0'

config device
        option type 'bridge'
        option name 'iot-br'
        list ports 'eth0.41'
        option stp '1'

/etc/config/wireless of Archer:

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'pci0000:00/0000:00:00.0/0000:01:00.0'
        option channel '36'
        option band '5g'
        option htmode 'VHT80'
        option txpower '22'
        option country 'DE'
        option cell_density '0'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option ssid 'home-5'
        option ifname 'wlan5'
        option encryption 'psk2'
        option key 'xxxxx'
        option wpa_disable_eapol_key_retries '1'
        option ieee80211r '1'
        option mobility_domain '7195'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option dtim_period '3'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'platform/10180000.wmac'
        option channel '1'
        option band '2g'
        option htmode 'HT20'
        option country 'DE'
        option cell_density '0'
        option txpower '20'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option ssid 'home-2.4'
        option ifname 'wlan2'
        option encryption 'psk2+ccmp'
        option key 'xxxxx'
        option wpa_disable_eapol_key_retries '1'
        option wds '1'
        option ieee80211r '1'
        option mobility_domain '7196'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option dtim_period '3'

config wifi-iface 'mesh2'
        option device 'radio1'
        option ifname 'mesh2'
        option mode 'mesh'
        option encryption 'sae'
        option mesh_fwding '1'
        option mesh_rssi_threshold '0'
        option key 'xxxxx'
        option mesh_id 'home-mesh'
        option network 'lan'

config wifi-iface 'wifinet5'
        option device 'radio1'
        option mode 'ap'
        option ssid 'IOT'
        option hidden '1'
        option isolate '1'
        option ieee80211w '0'
        option wpa_disable_eapol_key_retries '1'
        option key 'xxxxx'
        option encryption 'psk2'
        option network 'IOT'

/etc/config/network of the Repeater

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'xxxxx::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'
        option stp '1'
        option ipv6 '1'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.10.5'
        option netmask '255.255.255.0'
        option gateway '192.168.10.1'
        list dns '192.168.10.1'

config interface 'wwan'
        option proto 'dhcp'

config interface 'lan6'
        option proto 'dhcpv6'
        option device '@lan'
        option reqaddress 'try'
        option reqprefix 'no'

/etc/config/wireless of the Repeater:

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'platform/soc/a000000.wifi'
        option channel '1'
        option band '2g'
        option txpower '20'
        option country 'DE'
        option cell_density '0'
        option htmode 'HT20'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option ssid 'home-2.4'
        option ifname 'wlan2'
        option encryption 'psk2+ccmp'
        option key 'xxxxx'
        option wpa_disable_eapol_key_retries '1'
        option ieee80211r '1'
        option mobility_domain '7196'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option dtim_period '3'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'platform/soc/a800000.wifi'
        option channel '36'
        option band '5g'
        option htmode 'VHT80'
        option country 'DE'
        option cell_density '0'
        option txpower '23'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option ssid 'home-5'
        option ifname 'wlan5'
        option encryption 'psk2+ccmp'
        option key 'xxxxx'
        option wpa_disable_eapol_key_retries '1'
        option ieee80211r '1'
        option mobility_domain '7195'
        option ft_over_ds '0'
        option ft_psk_generate_local '1'
        option dtim_period '3'

config wifi-iface 'mesh2'
        option device 'radio0'
        option ifname 'mesh2'
        option mode 'mesh'
        option encryption 'sae'
        option mesh_rssi_threshold '0'
        option key 'xxxxx'
        option network 'lan'
        option mesh_id 'home-mesh'
        option mesh_fwding '1'

/etc/config/mesh11sd on both:

config mesh11sd 'setup'
        option enabled '1'
        option debuglevel '1'
        option checkinterval '10'
        option interface_timeout '10'

config mesh11sd 'mesh_params'
        option mesh_fwding '1'
        option mesh_rssi_threshold '-90'
        option mesh_gate_announcements '1'
        option mesh_hwmp_rootmode '3'
        option mesh_max_peer_links '8'

bluewavenet · August 18, 2023, 10:27am

-90dbm is far too low, if the signal gets anywhere near that low, the link quality will be diabolical. That's not the problem though.

Show the output of mesh11sd status

fireburner · August 18, 2023, 10:36am

The two devices are a bit a way from each other and they usually report a signal between -65 and -80. I took -90 to be sure they stay connected. There is not much noise around, so that shouldn#t be a big problem I guess.

mesh11sd status on the Repeater:

{
  "setup":{
    "version":"2.0.0",
    "enabled":"1",
    "procd_status":"running",
    "portal_detect":"1",
    "mesh_basename":"m-11s-",
    "checkinterval":"10",
    "interface_timeout":"10",
    "debuglevel":"1"
  }
  "interfaces":{
    "m-11s-0":{
      "mesh_retry_timeout":"100",
      "mesh_confirm_timeout":"100",
      "mesh_holding_timeout":"100",
      "mesh_max_peer_links":"8",
      "mesh_max_retries":"3",
      "mesh_ttl":"31",
      "mesh_element_ttl":"31",
      "mesh_auto_open_plinks":"0",
      "mesh_hwmp_max_preq_retries":"4",
      "mesh_path_refresh_time":"1000",
      "mesh_min_discovery_timeout":"100",
      "mesh_hwmp_active_path_timeout":"5000",
      "mesh_hwmp_preq_min_interval":"10",
      "mesh_hwmp_net_diameter_traversal_time":"50",
      "mesh_hwmp_rootmode":"3",
      "mesh_hwmp_rann_interval":"5000",
      "mesh_gate_announcements":"1",
      "mesh_fwding":"1",
      "mesh_sync_offset_max_neighor":"50",
      "mesh_rssi_threshold":"-90",
      "mesh_hwmp_active_path_to_root_timeout":"6000",
      "mesh_hwmp_root_interval":"5000",
      "mesh_hwmp_confirmation_interval":"2000",
      "mesh_power_mode":"active",
      "mesh_awake_window":"10",
      "mesh_plink_timeout":"0",
      "mesh_connected_to_gate":"1",
      "mesh_nolearn":"0",
      "mesh_connected_to_as":"0",
      "mesh_id":"home-mesh",
      "device":"radio0",
      "channel":"1",
      "tx_packets":"20092",
      "tx_bytes":"7757975",
      "rx_packets":"10458",
      "rx_bytes":"4497034",
      "active_peers":"1",
      "peers":{
        "macaddr1":{
          "next_hop":"macaddr1"
        }
      }
      "active_stations":"4",
      "stations":{
        "macaddr2":{
          "proxy_node":"macaddr1"
        },
        "macaddr3":{
          "proxy_node":"macaddr1"
        },
        "macaddr4":{
          "proxy_node":"macaddr1"
        },
        "macaddr5":{
          "proxy_node":"macaddr1"
        }
      }
    }
  }
}

and on the Router:

{
  "setup":{
    "version":"2.0.0",
    "enabled":"1",
    "procd_status":"running",
    "portal_detect":"1",
    "mesh_basename":"m-11s-",
    "checkinterval":"10",
    "interface_timeout":"10",
    "debuglevel":"1"
  }
  "interfaces":{
    "m-11s-0":{
      "mesh_retry_timeout":"100",
      "mesh_confirm_timeout":"100",
      "mesh_holding_timeout":"100",
      "mesh_max_peer_links":"8",
      "mesh_max_retries":"3",
      "mesh_ttl":"31",
      "mesh_element_ttl":"31",
      "mesh_auto_open_plinks":"0",
      "mesh_hwmp_max_preq_retries":"4",
      "mesh_path_refresh_time":"1000",
      "mesh_min_discovery_timeout":"100",
      "mesh_hwmp_active_path_timeout":"5000",
      "mesh_hwmp_preq_min_interval":"10",
      "mesh_hwmp_net_diameter_traversal_time":"50",
      "mesh_hwmp_rootmode":"3",
      "mesh_hwmp_rann_interval":"5000",
      "mesh_gate_announcements":"1",
      "mesh_fwding":"1",
      "mesh_sync_offset_max_neighor":"50",
      "mesh_rssi_threshold":"-90",
      "mesh_hwmp_active_path_to_root_timeout":"6000",
      "mesh_hwmp_root_interval":"5000",
      "mesh_hwmp_confirmation_interval":"2000",
      "mesh_power_mode":"active",
      "mesh_awake_window":"10",
      "mesh_plink_timeout":"0",
      "mesh_connected_to_gate":"1",
      "mesh_nolearn":"0",
      "mesh_connected_to_as":"0",
      "mesh_id":"home-mesh",
      "device":"radio1",
      "channel":"1",
      "tx_packets":"177144",
      "tx_bytes":"74061387",
      "rx_packets":"155702",
      "rx_bytes":"33914758",
      "active_peers":"1",
      "peers":{
        "macaddr6":{
          "next_hop":"macaddr6"
        }
      }
      "active_stations":"2",
      "stations":{
        "macaddr7":{
          "proxy_node":"macaddr6"
        },
        "macaddr8":{
          "proxy_node":"macaddr6"
        }
      }
    }
  }
}

bluewavenet · August 18, 2023, 11:01am

Put these back to default ie enabled - mesh11sd handles this.

Once done, show the output, for both devices, of:
uci show dhcp

and

mesh11sd connect

Are you pinging using ipv4 or ipv6? It sounds like an ipv4 ARP timeout or ipv6 NDP glitch ...

fireburner · August 18, 2023, 12:54pm

I am pinging via ipv4

Repeater:

uci show dhcp
dhcp.@dnsmasq[0]=dnsmasq
dhcp.@dnsmasq[0].domainneeded='1'
dhcp.@dnsmasq[0].localise_queries='1'
dhcp.@dnsmasq[0].rebind_protection='1'
dhcp.@dnsmasq[0].rebind_localhost='1'
dhcp.@dnsmasq[0].expandhosts='1'
dhcp.@dnsmasq[0].authoritative='1'
dhcp.@dnsmasq[0].readethers='1'
dhcp.@dnsmasq[0].leasefile='/tmp/dhcp.leases'
dhcp.@dnsmasq[0].resolvfile='/tmp/resolv.conf.d/resolv.conf.auto'
dhcp.@dnsmasq[0].localservice='1'
dhcp.@dnsmasq[0].ednspacket_max='1232'
dhcp.@dnsmasq[0].local='/lan/'
dhcp.@dnsmasq[0].domain='lan'
dhcp.@dnsmasq[0].server='192.168.10.1'
dhcp.lan=dhcp
dhcp.lan.interface='lan'
dhcp.lan.start='100'
dhcp.lan.limit='150'
dhcp.lan.leasetime='12h'
dhcp.lan.dhcpv4='server'
dhcp.lan.ignore='1'
dhcp.wan=dhcp
dhcp.wan.interface='wan'
dhcp.wan.ignore='1'
dhcp.odhcpd=odhcpd
dhcp.odhcpd.maindhcp='0'
dhcp.odhcpd.leasefile='/tmp/hosts/odhcpd'
dhcp.odhcpd.leasetrigger='/usr/sbin/odhcpd-update'
dhcp.odhcpd.loglevel='4

Router:

dhcp.@dnsmasq[0]=dnsmasq
dhcp.@dnsmasq[0].domainneeded='1'
dhcp.@dnsmasq[0].localise_queries='1'
dhcp.@dnsmasq[0].rebind_protection='1'
dhcp.@dnsmasq[0].rebind_localhost='1'
dhcp.@dnsmasq[0].expandhosts='1'
dhcp.@dnsmasq[0].authoritative='1'
dhcp.@dnsmasq[0].readethers='1'
dhcp.@dnsmasq[0].leasefile='/tmp/dhcp.leases'
dhcp.@dnsmasq[0].resolvfile='/tmp/resolv.conf.d/resolv.conf.auto'
dhcp.@dnsmasq[0].localservice='1'
dhcp.@dnsmasq[0].ednspacket_max='1232'
dhcp.@dnsmasq[0].local='/lan/'
dhcp.@dnsmasq[0].domain='lan'
dhcp.@dnsmasq[0].strictorder='1'
dhcp.lan=dhcp
dhcp.lan.interface='lan'
dhcp.lan.start='100'
dhcp.lan.limit='150'
dhcp.lan.dhcpv4='server'
dhcp.lan.leasetime='12h'
dhcp.lan.dhcp_option='6,192.168.10.23,192.168.10.1'
dhcp.lan.ra='server'
dhcp.lan.dhcpv6='server'
dhcp.lan.dns_service='0'
dhcp.wan=dhcp
dhcp.wan.interface='wan'
dhcp.wan.ignore='1'
dhcp.odhcpd=odhcpd
dhcp.odhcpd.maindhcp='0'
dhcp.odhcpd.leasefile='/tmp/hosts/odhcpd'
dhcp.odhcpd.leasetrigger='/usr/sbin/odhcpd-update'
dhcp.odhcpd.loglevel='4'
dhcp.@domain[0]=domain
dhcp.@domain[0].ip='192.168.10.1'
dhcp.@domain[0].name='openwrt-router.lan'
dhcp.@domain[1]=domain
dhcp.@domain[1].name='openwrt-repeater.lan'
dhcp.@domain[1].ip='192.168.10.5'
dhcp.@domain[2]=domain
dhcp.@domain[2].name='fritz.box'
dhcp.@domain[2].ip='192.168.178.1'
dhcp.@host[0]=host
dhcp.@host[0].name='xxxxx'
dhcp.@host[0].dns='1'
dhcp.@host[0].mac='xxxxx'
dhcp.@host[0].ip='192.168.10.21'
dhcp.@host[1]=host
dhcp.@host[1].name='xxxxx'
dhcp.@host[1].dns='1'
dhcp.@host[1].mac='xxxxx'
dhcp.@host[1].ip='192.168.10.50'
dhcp.@host[2]=host
dhcp.@host[2].name='xxxxx'
dhcp.@host[2].dns='1'
dhcp.@host[2].mac='xxxxx'
dhcp.@host[2].ip='192.168.10.23'
dhcp.IOT=dhcp
dhcp.IOT.interface='IOT'
dhcp.IOT.start='100'
dhcp.IOT.limit='150'
dhcp.IOT.leasetime='12h'
dhcp.@host[3]=host
dhcp.@host[3].name='tasmota'
dhcp.@host[3].dns='1'
dhcp.@host[3].mac='xxxxx'
dhcp.@host[3].ip='192.168.41.60'

Repeater and Router:

=========================================================
The following meshnodes are available for remote connection:

=========================================================

bluewavenet · August 18, 2023, 1:33pm

I noticed mesh11sd status returned "mesh_connected_to_as":"0" on the router.
This implies there is, or was, no upstream ipv4 connectivity....

The lan sections of the Dnsmasq config are not what I would have expected for the defaults.
Have you changed them?

On both the repeater and router do the following:

service mesh11sd stop
uci revert dhcp

Then edit /etc/config/dhcp and replace the lan section with:

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '12h'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option ra_default '2'

Then reboot both.

Now give things a few minutes to stabilise then on the router do:
mesh11sd connect

It should now return the mac of the repeater as available for connection.
If it does, try your pings again.

fireburner · August 18, 2023, 3:27pm

seems to be a default now, see luci:
Screenshot_20230818_171953

I might have a played around a bit with some DHCP settings.
As mentioned, all devices have internet access, there are just issues with routing within the lan subnet over the mesh.

I assume that

is supposed to reset the dhcp settings to factory? This did not change anything in my /etc/config/dhcp

I have adjusted the lan section on the router as suggested, but mesh11sd connect unfortuntaly still doesn't show any nodes.

With the pinging, that is working for a while as mentioned, then breaks and as soon as I ping from the Repeater to a client on the router, all works again for a while.

I forgot to mention, that I run 23.05.0-rc2 on the router for testing and 22.03.5 on the repeater.
I will received a second repeater (same model) on Monday and then try 23.05-rc2 on that one too. (Upgrade introduces DSA, which means I would need to redo the config manually, which is why I was waiting for the second one first.)
Not sure if these issues might be induced by the beta version?

bluewavenet · August 18, 2023, 6:21pm

No. It reverts any dynamic changes that uci may have made (by request of mesh11sd)

Indeed it will not. /etc/config/dhcp contains the startup static config.

Yes. This is because the repeater is not generating any ipv4 traffic so its ARP times out. As soon as you ping from it, ARP is refreshed.

To be clear, this is the lan section of /etc/config/dhcp.

My instructions were:

On both the repeater and router do the following:

802.11s mesh and mesh11sd both work on both versions of OpenWrt.

bluewavenet · August 18, 2023, 6:36pm

I notice you have this on the br-lan device configuration.
This will have the effect of breaking NDP on br-lan. This should be left out of the configuration as it is relevant only for a PPP wan device.

fireburner · August 18, 2023, 11:03pm

So you are saying I should set dhcp on both devices? Won't they have conflicts?

That makes total sense. Do you have any idea, why this is the case?

I have removed option ipv6 '1' now.

bluewavenet · August 19, 2023, 6:04am

Mesh11sd handles it by dynamically turning off dhcp thus preventing conflicts, thereby very much simplifying the config.
In fact mesh11sd v2 onwards supports rollout of meshnodes without any special configuration other than just enabling the wireless mesh interface ie keeping the default wan/lan router config.

To be honest, I have not yet had time to update the user guide. So of course you have done things recommended from the pre-mesh11sd-v2 days. As you have found, this has no effect on the actual mesh network as all client devices can get to the router and then out to the Internet.

For clarification, an 802.11s mesh is a layer 2 mac-routing network. As such it "routes" packets at layer 2 and plays no part in layer 3 routing protocols. By default a meshnode will not generate any layer 3 traffic at all unless you access it in some way, eg logging in or pinging it. This means that any arp table entries for it will age and expire.

For remote administration, mesh11sd uses ipv6 link-local connections, but you have disabled ipv6 so that does not work.

The Fritz repeater 1200 flashing to OpenWrt looks like a complicated process.
What is the default OpenWrt config?
Usually in OpenWrt it would be ethernet as wan and wireless disabled making it very difficult to get going

In cases like this I would use imagebulder to make an image with wireless enabled......

fireburner · August 19, 2023, 6:45am

Actually, ipv6 is not disabled. See on Repeater

config interface 'lan6'
        option proto 'dhcpv6'
        option device '@lan'
        option reqaddress 'try'
        option reqprefix 'no'

This however does not restart automatically in case the router is restarted or so.
Does this mean, mesh will not work in an ipv4 only network?

If I recall correctly, the Repeater 1200 only had a lan bridge with both the ethernet port and the default (disabled) WIFI.

bluewavenet · August 23, 2023, 7:02pm

802.11s mesh is a level 2 mac-routing network. It does not know or care what happens at layer 3 or above, it just carries the higher level traffic.