Mesh11sd & 802.11s Issue

Hoping someone can help here - I have test devices (UI Rocket M5) that will be forming a mesh network. I have followed all the info on both these pages:

[OpenWrt Wiki] 802.11s Based Wireless Mesh Networking
[OpenWrt Wiki] The Mesh11sd Project

The commands used to setup the mesh and interfaces and a diagram for context are below. The issue is that I seem to have the nodes connected as they appear when running iw dev m-11s-0 station dump command but I cannot ping/ access devices on the other side of the mesh (in either direction). When running mesh11sd status I have no active peers.

I want to be able to have the local clients to each node (mostly connected over eth - later some using node as WAP). Each node's LAN is set as a /16 so that broadcast traffic will work and additional routes are not required to be added.

Assumptions/ Notes

  • latest possible OpenWRT Build (19.07)
  • wpad-mesh-wolfssl and mesh11sd packagaes installed
  • anywhere x is below would be replaced with node id (1,2,3 etc)
  • currently no "gateway" node, all peer to peer tarffic

Commands for Node Setup

# Set system settings
uci set system.@system[0].zonename='Europe/London'
uci set system.@system[0].timezone='GMT0BST,M3.5.0/1,M10.5.0'
uci set system.@system[0].hostname='kn0x'
uci commit system
/etc/init.d/system reload

# Set mesh11sd defaults
uci set mesh11sd.mesh_params.mesh_rssi_threshold='-80'
uci set mesh11sd.mesh_params.mesh_gate_announcements='1'
uci set mesh11sd.mesh_params.mesh_hwmp_rootmode='3'
uci set mesh11sd.mesh_params.mesh_fwding='1'
uci set mesh11sd.mesh_params.mesh_max_peer_links='150'
uci commit mesh11sd

# Setup Mesh (same on all nodes)
uci set wireless.mesh0=wifi-iface
uci set wireless.mesh0.device='radio0'
uci set wireless.mesh0.ifname='mesh0'
uci set wireless.mesh0.network='lan'
uci set wireless.mesh0.mode='mesh'
uci set wireless.mesh0.mesh_id='kestrel_mesh'
uci set wireless.mesh0.encryption='sae'
uci set wireless.mesh0.key='KEY_HERE'
uci set wireless.mesh0.disabled=0
uci commit wireless
wifi

# Set LAN for node (different on each node, 10.80.x.1/16)
uci set network.lan.ipaddr='10.80.x.1'
uci set network.lan.netmask='255.255.0.0'
uci set dhcp.lan.start='5220'
uci set dhcp.lan.limit='50'
uci commit dhcp
uci commit network

#Reboot
reboot

Diagram

There are definitely better people to help with wireless/mesh here, but I'd venture a guess that your /etc/config/network file would be helpful.

It's super simple - have isolated any other changes and reset to defaults then just built as mentioned above. Contents of /etc/config/network is:

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd28:d325:8483::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0'
        option proto 'static'
        option ip6assign '60'
        option ipaddr '10.80.20.1'
        option netmask '255.255.0.0'

I'm now thinking this might be an issue with the way I have the bridging/ DHCP setup. Hopefully someone can clarify for me but I think I can't have DHCP running on LAN on each node since LAN A is bridged with mesh which is bridged with LAN B leading to DHCP conflicts?

Mesh11sd has not been backported as 19.07 is eol. It might work....

What version are you using?

Note that an 802.11s mesh is a layer 2 mac-routing backhaul. Any layer 3 on top should be a single subnet. But before we fix that we need to see what version of mesh11sd you are running.

Running mesh11sd 2.0.0-1. I've done some more testing in the meantime. I've got the nodes communicating now - i.e. showing up in the mesh11sd status command and able to ping each other.

As for client devices on the bridged LANs I can't get them talking to each other.

I think maybe you are overthinking the setup. It sounds like the layer 2 mac-routing is working but your ip/dhcp setup is very over-complicated.

The rocket-m is supported in OpenWrt 23.05.x, why do you think 19.7 is the latest possible build?

I would suggest upgrading to a custom image with everything in place as a good starting point.

https://firmware-selector.openwrt.org/?version=23.05.0&target=ath79%2Fgeneric&id=ubnt_rocket-m

In the "customise" section, edit "Installed Packages" by changing
wpad-basic-mbedtls

to

-wpad-basic-mbedtls wpad-mesh-mbedtls mesh11sd

And in "Script to Run on First Boot", add:

uci set wireless.radio0.disabled='0'
uci set wireless.mesh0=wifi-iface
uci set wireless.mesh0.device='radio0'
uci set wireless.mesh0.mode='mesh'
uci set wireless.mesh0.encryption='sae'
uci set wireless.mesh0.key='KEY_HERE'
uci set wireless.mesh0.disabled='0'
uci set wireless.mesh0.network='lan'
uci set wireless.mesh0.mesh_id='kestrel_mesh'
uci commit wireless

Use the same image on all rockets. It does not matter about ip settings yet.

Your diagram does not show an upstream Internet feed, so all meshnodes will be in gateway mode with the lan port unusable for the moment.

With only one rocket powered on, connect the lan port of that rocket to your Internet router, then connect to the OpenWrt ssid.

When you get this far we can start configuring.
Note: The upcoming mesh11sd v3 release will automate most of the config, but for now this is the way to do it.

You are probably right - here is the use case for extra context: 3 vehicles not always in range of each other. Each node providing local network within the vehicle at all times then when vehicles are within mesh range to allow routing between the client devices behind each mesh node. The reason for the single /16 and different DHCP scopes is I need multicast traffic to work and wasn't sure it would work cross subnet.

See this post - Rocket M5 XW Firmware - It seems only the XM is supported which is odd as it's an older hardware revision.

That's by design - this is for off-grid network where internet access can't be guaranteed so making sure all the mesh side is working first before dealing with internet access.

A very interesting use case. It reminds me of a similar one involving agricultural vehicles...

Having multiple dhcp servers on the same subnet will not work here. For ip traffic you need a default gateway, ie just one reachable gateway, for the entire subnet.

What you will have to do is layer 3 route, with nat, at each meshnode so that you have routed gateway nodes. This can be handled later, but first you need to get the basic mesh working properly.

If you would prefer we can go into in depth detail by private messaging then come back here with a solution..... Up to you.

Checking my notes from a few years ago, I think the Bullet M XW firmware works on the Rocket XW....... everything but the leds, those had to be fixed with a script.
It also worked in a similar fashion on the Nanostation...
Worth a try? You can unbrick easily via tftp if all else fails.

Thanks, will message you later. I have made some more progress I think in the meantime.

Will give this a go too - LEDs will not be visible in custom housing so shouldn't be a massive issue.

I appreciate the help, thank you.