Mesh Network Not Working on Raspberry Pi 4

alexrebell · September 20, 2024, 11:16am

I've decided to build a mesh network using Raspberry Pi 4 devices.

I have 4 Raspberry Pi 4 units, each with a TP-Link TL-WN721N Wi-Fi adapter connected via USB.

Here's what I've done:

Removed the wpad-mbedtls package.
Installed the wpad-mesh-openssl package.
Installed the full driver package for the Wi-Fi adapter:
- ath9k-htc-firmware
- kmod-ath9k
- kmod-ath9k-common
- kmod-ath9k-htc

That's all regarding the software setup.

Next, I proceeded to configure the mesh network itself.

My wireless configuration: /etc/config/wireless

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.1/1-1.1:1.0'
        option channel '13'
        option band '2g'
        option cell_density '0'

config wifi-iface 'wifinet0'
        option device 'radio1'
        option mode 'mesh'
        option mesh_fwding '1'
        option encryption 'sae'
        option mesh_id 'mymeshvini'
        option mesh_fwding '1'
        option mesh_rssi_threshold '0'
        option network 'lan'
        option key '123456789'

My network configuration: /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd68:861a:cb06::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'dhcp'

My firewall configuration: /etc/config/firewall

config zone
        option name             lan
        list   network          'lan'
        option input            ACCEPT
        option output           ACCEPT
        option forward          ACCEPT

config zone
        option name             wan
        list   network          'wan'
        list   network          'wan6'
        option input            REJECT
        option output           ACCEPT
        option forward          REJECT
        option masq             0
        option mtu_fix          0

When the devices are arranged as shown in Picture 1, all devices are within direct line of sight. They can all see each other, and I can connect to each of them via SSH from my laptop, and so on.

However, when I simulate the situation shown in Picture 2, Rpi_1 (acting as the gateway) doesn't see Rpi_3 and Rpi_4 through Rpi_2. But Rpi_2 sees Rpi_3 and Rpi_4.

If I place each of them at maximum distance in a line, it looks like this:
Rpi_1 ---- Rpi_2 ---- Rpi_3 ---- Rpi_4

Rpi_1 only sees Rpi_2
Rpi_2 sees Rpi_1 and Rpi_3
Rpi_3 sees Rpi_2 and Rpi_4
Rpi_4 only sees Rpi_3

I can't figure out what the problem is. Where did I misconfigure something? Please advise what I need to do to make the mesh network work as shown in Picture 2, so that I can access Rpi_4 from my laptop.
Additional Information:

I'm aiming for a fully connected mesh network where data can be relayed through intermediary nodes.
From Rpi_1, I want to be able to communicate with Rpi_4 even if they are not in direct range, using Rpi_2 and Rpi_3 as intermediate nodes.

Any help or guidance would be greatly appreciated!

brada4 · September 20, 2024, 12:01pm

wpad-mbedtls is bigger than -mesh- package, though by irrelevant for your setup radius support
You have to set country code on radio before using ch13
If you ever draw cable to other raspberry or need guest/iot in the barn maybe start with batman now?

Start troubleshooting by setting 2nd node in direct line between antennas, then install it in place, lets troubleshoot first pair first.

bluewavenet · September 20, 2024, 6:12pm

This config with simple 802.11s mesh will result in a single hop network.

Single hop only - so it works.

Only the single hop links work.

For multiple hops and a stable backhaul network you need to set numerous mesh parameters. Most of these cannot be reliably set in the wireless config.

You have 2 options to create a multi hop backhaul:

Use the mesh11sd package - it can set all mesh parameters to enable 802.11s mac-routing.
Use BATMAN - it does its own thing, setting up an ipv6 dynamic routing system to create a stable backhaul.

As @brada4 says, if you want to have some cabled segments in the backhaul, you have to be careful. Both BATMAN and mesh11sd prevent the mesh/bridge loop storm that a cabled segment would otherwise produce.

I would suggest you read the OpenWrt mesh documentation here:

https://openwrt.org/docs/guide-user/network/wifi/mesh/start

For your use case, a simple way forward is to follow the 802.11s Rapid Deployment document. This allows the exact same image to be used on all your RPi meshnodes.

https://openwrt.org/docs/guide-user/network/wifi/mesh/rapiddeployment

The Rpi is a bit different to most devices running OpenWrt, so it might present a few gotchas. If you decide to go the Rapid Deployment route, just ask here and I can talk you through it.

alexrebell · October 4, 2024, 1:11pm

To be honest, I didn't quite understand what you need me to do. Could you please clarify?

alexrebell · October 4, 2024, 1:23pm

Other devices will not be connected via cables. In the future, I intend to operate as follows: the mobile phone should connect to the RPi_3 device and receive all network configurations from the router, as well as have access to the local network and the internet.

Yes, of course, I want to proceed with the rapid deployment approach.

I understand that, for my situation, I will need to use the mesh11sd package?

psherman · October 4, 2024, 1:25pm

I don’t have any specific info to add about the configuration here, but as far as the devices….

Is this mesh network being built as an educational exercise or for practical/functional purposes on a real network?

The former makes sense, the latter does not. The Pi, even with external USB-wifi sticks, is a particularly bad option in terms of wifi performance. Any “plastic box” router made in the past 10+ years will easily outperform anything you can do on the Pi.

alexrebell · October 4, 2024, 1:40pm

exclusively for educational purposes, to test and understand the work of the mesh network in practice.

alexrebell · October 9, 2024, 11:22am

Could you tell me please, why is it that after installing the mesh11sd on the Rpi device, on my router, which distributes IP addresses via DHCP, the address pool immediately fills up, only one device uses all IP addresses at once, why is this? which setting should be changed so that this does not happen, but at the same time the mesh network works

bluewavenet · October 9, 2024, 12:51pm

This can only happen if the device in question keeps changing its mac address....

What kind of device is it? Do you have some kind of mac randomising script running on it?

You will have to share your configs again, the current ones please.

alexrebell · October 9, 2024, 1:58pm

I'm sorry, maybe I didn't put it that way, I just decided to install the mesh11sd package on a new system where nothing has been configured yet and just see how it works, what it is and what settings there are for mesh networks. As you can see from my config on the Raspberry device - none of it is configured.

cat /etc/config/network:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd5c:70d1:9761::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

config interface 'lan'
        option device 'br-lan'
        option proto 'dhcp'

cat /etc/config/wireless:

config wifi-device 'radio0'
        option type 'mac80211'
        option path 'platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1'
        option channel '36'
        option band '5g'
        option htmode 'VHT80'
        option disabled '1'

config wifi-device 'radio1'
        option type 'mac80211'
        option path 'scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.1/1-1.1:1.0'
        option channel '1'
        option band '2g'
        option htmode 'HT20'
        option disabled '1'

there are no additional scripts and other things.after installing the mesh11s package, here's what happens on my router in the IP address distribution section:

as you can see, the MAC address is the same

bluewavenet · October 9, 2024, 4:39pm

With these network and wireless configs, there would be no mesh network.
Note that the /etc/config files only show the static config, so it is obviouys that you have indeed made some mesh11sd config changes for the mesh to come up.

To see the dynamic configurations, please show the outputs of:
uci show network
uci show wireless
and
uci show mesh11sd

It is important to see what you have changed in the mesh11sd config.

Have you read the documentation?
See here:
https://openwrt.org/docs/guide-user/network/wifi/mesh/mesh11sd

Particularly note the following from that documentation:

If you have not done any manual mesh configuration, mesh11sd will do nothing and wait in the background for you to do so.

Switching on Auto Configuration is a simple task and is recommended if you are starting from a basic OpenWrt reflash.
ie you have not done any configuration after flashing the image.

WARNING: If you have done ANY mesh configuration manually, you might cause a soft-brick condition.
Pre-configuring before enabling auto_config is for advanced users only. Make sure you know what you are doing!

If the mesh network interface is defined in the wireless configuration file, mesh11sd will attempt to use it, but be warned, this may have very unpredictable results and is not normally recommended.

alexrebell · October 10, 2024, 6:21am

root@OpenWrt:~# uci show network
network.loopback=interface
network.loopback.device='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.globals=globals
network.globals.ula_prefix='fd5c:70d1:9761::/48'
network.@device[0]=device
network.@device[0].name='br-lan'
network.@device[0].type='bridge'
network.@device[0].ports='eth0'
network.lan=interface
network.lan.device='eth0'
network.lan.proto='dhcp'

root@OpenWrt:~# uci show wireless
wireless.radio0=wifi-device
wireless.radio0.type='mac80211'
wireless.radio0.path='platform/soc/fe300000.mmcnr/mmc_host/mmc1/mmc1:0001/mmc1:0001:1'
wireless.radio0.channel='36'
wireless.radio0.band='5g'
wireless.radio0.htmode='VHT80'
wireless.radio0.disabled='1'
wireless.radio1=wifi-device
wireless.radio1.type='mac80211'
wireless.radio1.path='scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb1/1-1/1-1.1/1-1.1:1.0'
wireless.radio1.channel='1'
wireless.radio1.band='2g'
wireless.radio1.htmode='HT20'
wireless.radio1.disabled='1'

root@OpenWrt:~# uci show mesh11sd
mesh11sd.setup=mesh11sd
mesh11sd.mesh_params=mesh11sd
mesh11sd.mesh_params.mesh_fwding='1'
mesh11sd.mesh_params.mesh_rssi_threshold='-65'
mesh11sd.mesh_params.mesh_gate_announcements='1'
mesh11sd.mesh_params.mesh_hwmp_rootmode='4'
mesh11sd.mesh_params.mesh_hwmp_rann_interval='5000'
mesh11sd.mesh_params.mesh_hwmp_root_interval='5000'
mesh11sd.mesh_params.mesh_hwmp_active_path_timeout='5000'
mesh11sd.mesh_params.mesh_hwmp_active_path_to_root_timeout='6000'
mesh11sd.mesh_params.mesh_max_peer_links='16'
mesh11sd.mesh_params.mesh_plink_timeout='10'

Unfortunately, I haven't read it. seis will study what's there. but it seems to me that the content of such documentation does not suit my cases, some kind of individual approach is needed.
Thanks!

psherman · October 10, 2024, 6:28am

Shouldn’t this be br-lan? Or, remove eth0 from br-lan if you aren’t using the bridge.

alexrebell · October 10, 2024, 6:40am

You are absolutely right. There should be a bridge there. But I was just experimenting with the settings and set up a different interface myself. Now everything is returned as it is by default.

root@OpenWrt:~# uci show network
network.loopback=interface
network.loopback.device='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.globals=globals
network.globals.ula_prefix='fd5c:70d1:9761::/48'
network.@device[0]=device
network.@device[0].name='br-lan'
network.@device[0].type='bridge'
network.@device[0].ports='eth0'
network.lan=interface
network.lan.device='br-lan'
network.lan.proto='dhcp'

bluewavenet · October 10, 2024, 6:53am

Sorry, but the mesh cannot be working with the radios disabled.

You are clearly not showing what you actually have configured.
How do you expect a sensible response?

Your declared use case:

The documentation that you say does not suit your use case (even though you have not read it), tells you how to build a mesh network....

If you do not understand, just say so and you will find people on here are willing to explain.

alexrebell · October 10, 2024, 7:08am

That's right, as you say. What I did was I reinstalled the operating system on my raspberry devices, but without configuring anything, and installed this package, just to see what it is, and after installation I found that the DHCP pool was filling up and I reported this, I wanted to clarify what was wrong or vice versa, this is a normal phenomenon for him.

You noticed correctly - I don't really understand this, and I make the settings according to my feelings. I would be very grateful if they would tell me how to set up the mesh network correctly, as I indicated this in my first message.
Thank you very much!

bluewavenet · October 10, 2024, 7:27am

Follow this example:
https://openwrt.org/docs/guide-user/network/wifi/mesh/rapiddeployment

Make sure you add the packages you require to make the usb wireless work.
Generate an image and put it on all your rpis.

See what happens.

Look at the resulting configs and see if you can understand what is happening and ask any questions you may have.

To not try to pre-empt any configuration, follow the documentation.

Once you get it working and see the working configs you can move on to your own experimentation.

alexrebell · October 10, 2024, 7:37am

Did I understand correctly from the documentation what I need to do to create a new installation image with pre-installed packages for the device? And if you install a regular image and install the necessary packages there, is this not possible? like I'm doing now. For information, for my own development. thank you!

bluewavenet · October 10, 2024, 7:47am

Yes, you understand correctly.

This is of course possible, but is the hard way to do it and not recommended.
People who do not know what they are doing will get it wrong and people who do know will not want to do it the hard way because it is a waste of time.

Get it working first, then take it apart to see how it works and learn about it.

alexrebell · October 10, 2024, 10:46am

Could you tell me please, what should I do next?

I have assembled the image as indicated in the documentation, exactly as indicated.
I installed the assembled image on raspberry.
I went to the specified ip address, which I set in the configuration.
I see that no wireless interfaces have been created.
Now what should I do?
I understand what needs to be done:
install the driver package kmod-ath9k-htc to make my external Wifi adapter (radio1) work. Next, create an interface with the mesh network mode.
or is there nothing to do?
Thank you very much!
at the same time, my pool of IP addresses on the router is full again

image780×513 34.9 KB