General Discussion on Mesh Technology

Installing and Using OpenWrt
1

I have some questions about available Mesh technologies, and meshing concepts in general. I'd like to start by describing my understanding of some basic classes of mesh technology.

  1. Layer 2 meshing: BATMAN-adv for example
  2. Layer 3 Meshing: OLSR, BATMAN, AODV, ABR, DSR
  3. Dynamic routing (Layer 3): BGP, OSPF, IS-IS etc

Technically it seems to me like 2 and 3 are basically the same thing, but with different assumptions built in about how dynamic things are. The technologies in 3 are used where changes happen infrequently, like at an ISP or a server farm, you might have a temporary outage between you and some peer, but changes happen a couple times a day or week, whereas the assumptions about (2) are that you could at least have nodes like say robots potentially wandering around a space, and they come in and out of range of each other at any time. So the kinds of information needed and the kinds of algorithms are different.

Nevertheless it's my understanding that type (2) meshes do in fact operate by adjusting routing tables. So what does that look like? In particular imagine you are doing IPv6.

Suppose we want to set up a local area network and can't assume that all the machines are capable of being mesh peers. So we have multiple APs, using a layer 3 mesh routing protocol as backhaul (not something like BATMAN-ADV which provides a layer 2 abstraction). We have an IPv6 subnet 2001:db8:0:1::/64 available to us, and we can use a ULA network for the mesh routing.

An AP client says "I want to send something to the internet" so it formulates a TCP SYN packet say, and it looks at its routing table and says "2001:db8:0:1::1 is my gateway" and if it doesn't have a MAC address it then sends out a NDP packet asking for the MAC. Who responds? In particular, does the mesh act as a multicast routing system and flood Router Advertisement packets across all the clients? Does the AP act as an NDP proxy and respond as if it were all the other nodes on the network? How can two devices connected to two different APs which are in the same layer 3 subnet 2001:db8:0:1::/64 directly talk to each other?

:thinking:

2

OK, type 3 would not be a mesh for the clients; but the routers only.

The router in e.g. OLSR, when it doesn't have a route. Also, OLSR keeps routes on the mesh-side to 0.0.0.0/0 and/or ::0/0 or less specific routes. This method can be used to offer multiple gateways on mesh.

They don't. They would connect to the Mesh if it needed to be system-wide; not the LAN. And they would announce their /32 or /128 presence directly via OLSR protocol. With OLSR, it's possible to bridge into the OLSR network instead.

3

first, thanks for the help!

sorry, obviously they wouldn't actually talk directly, but the /64 subnet is intended to offer the abstraction that they could right? So they do NDP, the AP proxies this, then they send their packet, the AP receives it at layer 2, then looks at it's routing table and routes it into the mesh looking for the other participant on the /64. Or is this not possible?

think of trying to let a bunch of phones, who are not going to be mesh participants no matter how hard you try act as if they are on a LAN by using a layer 3 mesh as backhaul. is this not possible?

4

Hummmm... you could DHCP (static) them on the mesh, someone still has to announce their /32 with proper protocol. OLSR only announces them...and there has to be unified DHCP...

BGP would be [somewhat] similar...

Someone has to speak the routing protocol...and if you announce a subnet, that's the interface that has it...maybe the OLSR can off/on the /32 and /128 by script based on giving a DHCP lease...

:thinking:

But then again, you said thee are clients "who are not going to be mesh participants," so you conceded that only their router can be the participant...and their router must therefore announce a subnet..?

Have you installed and tested any protocols?

NDP proxied...hummm...I'd have to look that one up.

5

I guess I'm imagining layer 3 mesh acting as if it were a unified backbone. I guess the mesh would announce a route to each AP client...

ignore ipv4 for the moment. assume ipv6 with SLAAC.

6

You could mesh a Layer 2 tunnel and serve it as your client AP; but that requires a lot of backend configuration and redundancy.

Needs a subnet again...

:thinking:

7

Yes... :thinking:

So let's do 2001:db8:0:1/64 as a subnet that all the clients believe they are connected to.

Let's have the mesh access points have a separate mesh local only subnet, fd11:1111:0000::/48

is there a way for the APs to present a unified /64 to the clients, while using their own layer 3 mesh for backhaul

without using a layer 2 tunnel

8

Absoutely!

Only if that same Layer 3 mesh is also the Layer 2 that carries the subnet. But...

:warning:

Which is separate from dynamic addressing.

And...maybe some device could...

Bascially...

Who announces the IP [in the proper protocol] :grey_question:

9

Yes assume the APs each have a routing Daemon. I guess it's problematic if you have clients doing privacy addressing, you'll be announcing a lot of /128s on the back end, and with roaming these routes might churn a lot. Still assume at most 100 clients at any given moment, so we're announcing a few hundred /128s at any given time

1 Like
10

You just made me think...ULAs would have to be disabled.

And in this hypothetical network...you assume a mesh where one Network Admin/Engineer controls all endpoints. :wink:

11

Well I think another single ULA/64 could be for clients that'd be ok.

1 Like
12

Except all routers cannot announce it. :wink:

13

or at least everyone cooperates somehow. Imagine for example a conference, or a disaster response.

14

right, just like the 2001:db8:0:1/64 they would announce individual 128s on the mesh?

15

I think you wanted to present a catch-22 on the /128s.

I merely assumed the script would make the announcement die with any unified DHCP leases...whose leases must be very short.

Yes, I control such [a portion of] a network using an OpenWrt router - it uses a modified version of RIPv2 that must operate at each gateway. The IPs are known, registered and coordinated before any emergency. I'll DM you. :smiley:

We also have a method of dynamically announcing the gateways to the Public Internet - including BGP.

16

that sounds like a cool system. Definitely send the DM.

But for this thing im imagining, suppose the AP clients operate not with DHCP but with SLAAC. The APs all offer a /64 to their clients, who hear RAs, and generate addresses. They send router solicitations and get responses from the APs (proxy NDP).

If each AP proxies the rest of the /64 via NDP, the AP clients wouldn't have a clue it'd look just like a bunch of Ethernet connected APs right?

:thinking:

Even better, you could utilize an http/https proxy, which runs on each AP. Since a huge quantity of traffic is just http/https, this would avoid a certain amount of churn. The individual end clients just request URLs from the proxy, and then the proxy goes onto the mesh with its own mesh backend IP, thereby reducing the routing table when the client is not doing anything except grabbing URLs.

You'd only have to announce the client's /128 on the mesh backend if it sent packets direct to the internet.

Ok, I'm going to draw a diagram and post it... it might help.

1 Like
17

Ok, here's sort of what I'm thinking: https://photos.app.goo.gl/taSUdzMzsxShqgPUA

The devices in the red region are "normal" AP clients who are all in a unified single /64 with SLAAC (think Android / iOS phones, tablets, laptops etc). Each of the devices in the "mesh backend" acts as both an AP to the red zone, as well as a mesh participant in the meshed backend whose purpose is to transport packets to and from the internet (possibly from several entrance/exit points) over whatever kinds of links are available, whether they're WiFi, wireless point to point, ethernet, optical, or emergency radio services.

Each AP sends RAs to its clients... listing an "anycast" IP for example 2001:db8:0:1::abc/64 as the router which each AP has bound to its wifi AP interface.

Each AP could also provide an http/https proxy from the same anycast IP.

Now, AP clients have no problem talking to the internet, they send internet packets to 2001:db8:0:1::abc which resides on the AP, the AP then routes this packet through the mesh to and from the internet... (they would need to announce the /128 of each AP client on the mesh). This makes sense.

The big problem is anyone who wants to talk from red to red... suppose we have two devices with IP addresses

A = 2001:db8:0:1::a
and
B = 2001:db8:0:1::b

Suppose they are connected to different APs. First of all, how could they find out about each other? Second of all, how do they send packets to each other?

The only thing I can think of is for the APs to do NDP proxy, so whenever anyone asks for neighbors, such as device A, they hear the response coming back from the MAC of the AP. Then the APs on the back end announce /128s to the rest of the mesh, so that when machine A sends to machine B, A does NDP and finds out that B is at the MAC address of the AP. Then, the AP receives at its MAC, a packet destined for B, it routes it on the mesh until it gets to the right AP where it can be sent direct to B.

18

@lleachii I'm just pinging this in case you didn't see my reply which would have been late last night for you.

1 Like
19

LOL, I made a draft I never posted. Your assumed hypothetical keeps requiring a united /64...Yet, your hypothetical network has 6 APs, with only 2 connected to the Internet...so I'll ask this...

What if the wrong AP announced 0.0.0.0/0 or ::0/0 too?

  • So the question becomes how do you write a script so the correct router announces the /128
  • the next problem is - this instantly means no given router can announce the /64

If every AP participated in an anycasted backend Layer 2 VPN, and served that network to clients instead...I think your hypothetical could work. Then I thought of the downside...this means that devices are not using the fastest matrix/routes to the hosts. :thinking:

But since only the clients move and not the subnets/routers, I'm no longer convinced your hypothetical problem needs a mesh - but a broadcast domain.

I'd simply bridge all client APs to the same "red zone" Ethernet network/VLAN/Ad-Hoc, no mesh technology running whatsoever.

The Internet gateways could just use VRRP or something.

So long as the hypothetical assumes only the clients move, this should work.

20

The goal is that devices could roam between the APs without switching networks, or dropping connections, hence all the APs are part of one /64

Couldn't any of the APs that have internet connections announce the /64 to whoever it is they connect to? I actually imagine that they're all tunneling to a point of presence, like a VPS or something, because this would allow any old person with a home internet connection to participate as a mesh node with internet gateway. So imagine that scenario where several say wireguard clients are all connecting to a VPS and each of them announces the entire /64 as available through them.

But this winds up with a lot of noise and inefficiency on the mesh right?

Yeah, I mean normally if you had a static ethernet network between the APs you could run something like STP and wind up with a redundant broadcast domain... But actually I'm imagining that the APs are potentially mobile too.

Imagine the following scenario. You have 10 or 20 volunteers, and several city buildings, who each have a good internet connection, and run an outdoor access point on a mast outside.... Now every fire truck, ambulance, police vehicle, maintenance worker car, bus, etc has one of these mesh AP nodes... They drive around a small town... hopefully everywhere they go, they have a connection to some of the stationary mesh points. Now, any person with an android phone, near enough to a police vehicle can communicate say through Signal (end to end encrypted chat). Furthermore, it's not dependent on one licensed provider, like T-Mobile or ATT, anyone can set up an additional mesh point anywhere without approval from some central authority, on a moments notice.

I see this as super helpful in a scenario like the massive wildfires in northern california last year, among other things.