Mesh (batman-adv) - Routing between interfaces

astero · October 1, 2019, 12:19pm

Hello guys,

I'm trying to build a WiFi Mesh consisting of multiple Ubiquiti Bullet M2 devices using batman-adv. Instead of bridging the mesh with my lan interface (as described in https://openwrt.org/docs/guide-user/network/wifi/mesh/batman) I want to use a dedicated subnet for the mesh network (does this make sense?).
Every Mesh Point (802.11s) is connected to the subnet 172.31.1.0/24, the lan port and regular 802.11n network (I'm using different SSIDs) are bridged and connected to subnets 172.31.1x.0/24, with a different x for every access point.

So far, the mesh seems to be working and I can SSH to every Bullet from my computer. However, I cannot ping other clients that are connected to a different mesh point. For example, I'm connected to one of the bullets. My PC obtains the IP 172.31.10.50 from dhcp.
ping 172.31.10.1 # 'local' mesh point -> works
ping 172.31.14.1 # different mesh point, connected through 802.11s -> works
ping 172.31.14.146 # client on different mesh point -> no response!

Here are my config files of one of the mesh points (the others are set up accordingly with different IPs):
/etc/config/network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix '<value>'

config interface 'lan'
	option type 'bridge'
	option ifname 'eth0'
	option proto 'static'
	option ipaddr '172.31.14.1'
	option netmask '255.255.255.0'
	option delegate '0'

config interface 'mesh'               
        option type 'bridge'          
        option ifname 'bat0' 
        option proto 'static'
        option ipaddr '172.31.1.14'
        option netmask '255.255.255.0'      
        option delegate '0'

config interface 'bat0'
	option proto 'batadv'
	option routing_algo 'BATMAN_V'
	option aggregated_ogms '1'
	option ap_isolation '0'
	option bonding '0'
	option fragmentation '1'
	option gw_mode 'off'
	option log_level '0'
	option orig_interval '1000'
	option bridge_loop_avoidance '1'
	option distributed_arp_table '1'
	option multicast_mode '1'
	option network_coding '0'
	option hop_penalty '30'
	option isolation_mark '0x00000000/0x00000000'

config interface 'nwi_mesh0'
	option mtu '2304'
	option proto 'batadv_hardif'
	option master 'bat0'

config route
	option target '172.31.10.0'
	option gateway '172.31.1.10'
	option netmask '255.255.255.0'
	option interface 'mesh'

config route
	option target '172.31.11.0'
	option gateway '172.31.1.11'
	option netmask '255.255.255.0'
	option interface 'mesh'

config route
	option target '172.31.12.0'
	option gateway '172.31.1.12'
	option netmask '255.255.255.0'
	option interface 'mesh'

config route
	option target '172.31.13.0'
	option gateway '172.31.1.13'
	option netmask '255.255.255.0'
	option interface 'mesh'

config route
	option target '172.31.15.0'
	option gateway '172.31.1.15'
	option netmask '255.255.255.0'
	option interface 'mesh'

/etc/config/wireless

config wifi-device 'radio0'
	option type 'mac80211'
	option channel '11'
	option hwmode '11g'
	option path 'pci0000:00/0000:00:00.0'
	option htmode 'HT20'
	option disabled '0'
	option country 'DE'

config wifi-iface 'default_radio0'
	option device 'radio0'
	option network 'lan'
	option mode 'ap'
	option ssid '<SSID1>'
	option encryption 'psk2'
	option key '<value>'

config wifi-iface 'mesh0'
        option device 'radio0'
        option ifname 'mesh0'
        option network 'nwi_mesh0'
        option mode 'mesh'
        option mesh_fwding '0'
        option 'mcast_rate' '24000'
        option mesh_id '<SSID2>'
        option encryption 'psk2+ccmp'
        option key '<value>'

/etc/config/firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'mesh'
	list network 'mesh'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'

config forwarding
	option src 'lan'
	option dest 'mesh'

config forwarding
	option src 'mesh'
	option dest 'lan'

config forwarding
	option src 'lan'
	option dest 'wan'

Does anybody know what's the problem?

astero · October 3, 2019, 6:46pm

Is there really nobody who can help and give me some advice?

jeff · October 8, 2019, 12:15pm

If you can ping all the mesh nodes, then it’s likely a “general” routing problem. tcpdump is a good tool to determine where the packets are getting lost.

astero · October 8, 2019, 12:33pm

Thank you for your reply.
Yes, I was also thinking about a routing problem.
Here is my routing table:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.31.1.0      *               255.255.255.0   U     0      0        0 br-mesh
172.31.10.0     172.31.1.10     255.255.255.0   UG    0      0        0 br-mesh
172.31.11.0     172.31.1.11     255.255.255.0   UG    0      0        0 br-mesh
172.31.12.0     172.31.1.12     255.255.255.0   UG    0      0        0 br-mesh
172.31.13.0     172.31.1.13     255.255.255.0   UG    0      0        0 br-mesh
172.31.14.0     *               255.255.255.0   U     0      0        0 br-lan
172.31.15.0     172.31.1.15     255.255.255.0   UG    0      0        0 br-mesh

In my opinion the table is looking good and I have rules for firewall forwarding?
I'm running out of ideas here...

jeff · October 8, 2019, 7:46pm

If ip route get <some attached client IP> is returning something sensible and doesn't provide further clues, then packet tracing with tcpdump would be the next step I'd take. tcpdump-mini has always been sufficient for my needs.

astero · October 10, 2019, 2:23pm

I was able to solve the problem. The packages did arrive at the client but the default route for the client was not set correctly. Stupid mistake and not mesh related at all

Now I have another question regarding mesh. I want to use a different device supporting 802.11ac. How do I have to change the configuration if I want to use both wifi interfaces (2.4 and 5 GHz) for batman-adv (https://www.open-mesh.org/projects/batman-adv/wiki/Multi-link-optimize)?

jeff · October 10, 2019, 3:11pm

That is a different question and one that is very unique. Opening a new thread for it with "Multi-Link" in the title would seem to have a better chance of attracting the attention of someone with experience with that.

(I've never considered dual-link over the same path as, at least where I am, one of the two bands is almost always superior to the other, and 2.4 GHz 802.11n is significantly slower than 802.11ac in my environment.)

AndrewZ · October 10, 2019, 3:46pm

Actually I will be interested as well because I was in the situation where 5G radio was used for outdoor mesh link and it was desirable to use 2.4G indoor for the same purpose. Fortunately I ended up using PLC indoor.

emi · February 26, 2020, 11:38pm

Hi

The way astero configures the network is quite interesting.
My question is however, how does the layer 3 route configurations "co-work" with the layer 2 routing performed by Batman-adv? My notion is that Batman is supposed to take care of all routing, even from a non mesh-host to another non mesh-host. So, the question is, in your setup, is it Batman or the layer 3 routing rules that does the routing job?

Also, suppose one of the mesh-node has internet connectivity and therefore acts as a GW. How do you configure such that all internet traffic routed to this GW? or will batman automatically take care of this?

Kindly hope you can share your experience and knowledge.

Regards

Em