Best method for WAN multipath / link aggregation / channel bonding

Installing and Using OpenWrt Network and Wireless Configuration
1

Hello, I recently moved to an area with no good internet options and only a handful of marginal ones (i.e. ~4 long-range open wifi access points and a 4G hotspot with poor reception). My goal is to combine these unreliable, low throughput connections into something approaching adequacy. I'm currently using Mwan3 but I'm getting a lot of dropped connections when the wind rustles the nearby trees in the wrong way.

My research has basically suggested I do a custom build of LEDE/OpenWRT that includes the multipath support patches, then use something like a MLVPN or glorytun tunnel to a VPS. Any other ideas? I'd rather not reinvent the wheel if I can help it...

2

You should probably take a look at this project, which is based on LEDE/OpenWRT and features MultipathTCP:

https://www.openmptcprouter.com

2 Likes
3

I saw that earlier, but noticed their build targets are x86 and the Raspberry Pi, and their documentation leaves a bit to be desired. Any idea if its stable enough for everyday use on routers?

That said, that may well be the most mature platform for this that's currently available for my hardware.

4

One of the challenges of multi-link technologies is the routing of return packets. Consumer end-points can't generally influence the routing outside of the user's own network. You might want to consider a "cloud" server as a well-connected end-point and use NAT / NPT to have the rest of the world route your traffic via that end point. Then you have control over how traffic is routed from that remote router to your "home" network. Such cloud servers can be "rented" for as little as US$5 per month.

1 Like
5

It's limited in terms of hardware selection, but the RPi actually is a pretty decent router. I have a number of them deployed. Team it up with a Ubiquiti UniFi AP Lite for WiFi and a gigabit switch to keep your LAN traffic moving (single subnet only to avoid routing over the slower RPi Ethernet) and I think you'll be pleased.

With either an x86 VM or RPi, setting up a test network is pretty cheap and easy to evaluate the stability for your set of needs. You can always roll your own later...

6

Old thread, I know - I was just interested how you have that RPi connected with the Ubiquiti UniFi AP Lite and the gigabit switch. I'm looking at https://www.openmptcprouter.com/ now and I'm trying to see how you use that hardware with that setup. Any tips welcome.
I need to aggregate a flaky ADSL line with one or more 4G connections.

7

Dan -

I'm a big fan of OpenMPTCProuter (OMR) myself. I did a fairly extensive post on it over on the LTE Hacks forum:

http://ltehacks.com/viewtopic.php?f=21&t=1078

If you haven't seen that, I'd suggest starting there. The RPi can be used for OMR, though I prefer an industrial x86 box like those from Qotom.

The Ubiquiti UniFi series is a great choice for WiFi. You'll need to use a PoE injector, or better yet a PoE switch. I use both UniFi switches and those from PoE Texas. The UniFi switches are nice because you don't have to decide between 24V Passive and 48V Active PoE -- most models can do either. If you end up with any MikroTik products in the mix ( I use the MikroTik RBM33G for my cellular router builds now instead of the RPi), then you can run everything (Ubiquiti and MikroTik stuff) on 24V PoE.

Hope that helps!

Scott

8

Thanks Scott - there's a lot to unpack there.

I was actually curious as to how best to use an RPi (3b+/4) in particular to base OMR off. I am based in Mozambique in southern Africa right now, and I have limited resources and ability to purchase premium kit.
I was trying to scope out a minimal hardware spec, and I was intrigued that you suggested using it with a specialized AP unit (probably outside my budget and reach) and a gigabit switch - on a single subnet only to avoid routing over slower RPi Ethernet as you said.
I'm not clear on the physical connections you have across the setup.

So in my example scenario, where I wish to aggregate for reliability+bandwidth, with:
WAN interfaces:
1 - ADSL modem on flaky 4-8Mbps line
2 - 4G Mifi router (micro/mini USB interface only)
3 - possible extra 4G Mifi router (micro/mini USB interface only)

  • how can I deploy a RPi OMR setup with minimal additional hardware purchases that will optimize performance both on WAN and not limit LAN?
    (Bearing in mind your comment about single subnet and routing.)

Or: how do I physically connect everything while optimizing cost and performance?

I understand that I would need to run the RPi as an AP, and would therefore need some antenna upgrade at the least.

9

Don't even attempt this. Get a separate piece of kit for the AP functionality. You can use an old wifi router running OpenWrt or a commercial AP like the ubiquiti or a tp-link eap224 but the wifi onboard the Pi is exceedingly weak with a fixed tiny antenna and chipset limitations

2 Likes
10

@dlakelan Thanks for the tip.
Having taken a closer look at the referenced posts from @bnhf , I'm now imagining this setup:

  • RPi running OMR (I have RPi already and I am on a budget)
  • TP-Link Omada AC1350 to do the wifi AP (can be had for about $60)
  • 2 x WE826-T2 or Netgear LB1120 4G WAN routers - (about $100 each)
  • 3 x USB-Ethernet adapters to connect to WAN routers including existing ADSL router ($15 each)

In this setup the RPi Ethernet will connect to the AP, and 3 of the USB ports will be used through USB-Ethernet adapters to connect to the Ethernet on the WAN routers. (The only way with an RPi is to use USB-Ethernet adapters, right? Unless I missed something)

This all comes to around $300, which is worth it if I can really up my bandwidth (I'm in rural Mozambique).

PoE is convenient and cool but I don't have the budget for the extra kit right now.

11

Cool. I love helping people in places like that get connected to the rest of the world.

Instead of your suggested kit with 3 USB-Ethernet adapters, I suggest to buy a TP-Link sg108e switch which will cost less than the $45 for the three adapters, and work better.

Using VLANs you can hook up one 4G WAN to one port, one 4G WAN to another port, and the RPi to the 3rd port, then still have 5 ethernet ports to use for LAN clients if you want.

None of this will be slowed down by use of VLANs unless you're getting multi-hundred megabit rates on those 4G WANs, and if you are, you can add just ONE USB3 UE300 to the RPi and have full gigabit both directions (assuming it's an RPi4, if it's an RPi3 you're limited to a few hundred megabits both onboard and add-on USB anyway).

1 Like
12

Dan -

You don't need Ethernet dongles for each cellular WAN device, as OMR supports MacVLAN, which allows you to use a switch instead. USB also works for any devices supporting tethering like many "MiFi" type devices do. I've used both Novatel and Netgear devices with OMR (here in the U.S.), and both have worked fine.

OMR requires significant horsepower to do what it does. An RPi will do the job, as it's more powerful than a typical router -- however the higher end the RPi the better. An Rpi4 would be the best, as it has true Gigabit Ethernet, USB3 ports and supports 802.11ac. If you can live with WiFi that doesn't have much range, the onboard WiFi should be usable for AP purposes. An RPi3B+ would be the next best choice -- with the 3B being an OK choice, when combined with a Gigabit switch on the LAN side (using a single subnet), so that LAN to LAN Ethernet traffic won't be slowed to "Fast" (100Mbps) Ethernet speeds

For the VPS side of things, is there a suitable datacenter in your region on the supported list? If you haven't examined that side of things yet, I'd suggest doing it soon. There'll be a monthly cost associated with it, and if there's nothing in southern Africa then ping times will increase.

Best,

Scott

13

@dlakelan Thanks, I hadn't twigged that VLAN support was possible now with a generic switch.
Crazy cool feature. I haven't checked into OpenWRT dev for a while, and it's amazing to see the way it continues to evolve - it's great.
So, yes, I will definitely get a gigabit Ethernet switch in the mix...

As for the VPS, there are Azure and AWS datacenters in Johannesburg and Cape Town, so that should be fine (I'm a DevOps engineer as it happens).

Although... @bnhf I was intrigued and a bit confused by one or two things you said:

USB also works for any devices supporting tethering like many "MiFi" type devices do.

Do you mean in this case the common USB 4g-wifi "dongles" that are often offered by telco providers? eg the HUAWEI E8372 (which also has external antenna connection).
Connecting 2 of these via USB could be a solution for me.

In general, though, MiFi routers do NOT have the ability to form network connections via USB, as far as I know. They generally have mini USB ports for charging, and all network connections are over 802.11. Hence why I had to hunt to find slightly more costly models like the Netgear LB1120 which have Ethernet ports.
Please correct me if I have any of this wrong.

Regarding RPi as AP - I think I will go with probably the TP-Link AP that I mentioned, as I need to cover a good area.

I'm still a bit mystified by this comment though:

with the 3B being an OK choice, when combined with a Gigabit switch on the LAN side (using a single subnet)

Are you saying that if I use a 3B and Gigabit switch on the LAN side, then I CANNOT use VLANs and must use a single subnet, unless I want to be slowed to 100Mbps? Thereby meaning that I cannot use the switch now to connect my Ethernet-enabled WAN routers?
I'm sorry for the confusion - I just can't get clear on the exact scenario that you're advising on(ie what is connected to what?).

14

Yes, that's what I mean. OpenMPTCProuter includes ModemManager, so any devices compatible with that should play well with OMR.

I see where you're going with this -- which is to use a single switch for both the LAN and WAN side of the things, connected to the RPi's single Ethernet port. Generally what you're suggesting is possible, in fact I've done this myself. I don't believe your LAN to LAN traffic on the same subnet (which doesn't require routing) would be slowed down because of the RPi's slower Ethernet port. Your MacVLAN WAN traffic definitely would, though that shouldn't be an issue unless your Internet speeds exceed 100Mbps. BTW, you can also use more traditional tagged VLANs (software based on the RPi).

15

You can absolutely use tagged VLANs and they have nothing to do with the slowness... The issue is that the Pi3 has a slow ethernet port... If you can get a pi4 it has gigE. The 3+ has a gig E port but it's connected over a USB2 bus so it will still be slower than gig... 300-400Mbps I think

16

@dlakelan @bnhf
Thank you guys for your comments. I think I'm pretty well set up now.
It seems like I'll get away with a single switch (even with an RPi3 which I may end up with) - my WAN speeds are unlikely to be much over 10Mbps for the 4g connections, and the ADSL will max out at 8Mbps (thus a theoretical combined WAN max around ~30Mbps).

Really looking forward to getting this set up and expanding it over time.

17

Just a couple of OMR and RPi specific comments here:

The Raspberry Pi version of OpenMPTCProuter comes configured out of the box to use MacVLANs for two WAN interfaces, rather than tagged VLANs (software based on the RPi). This makes it fairly straightforward to get a pair of WANs bonded and operational on the RPi.

No problem to switch to tagged VLANs on the 3B -- if you're already using them for your WAN interfaces (which I was), but you need to disable the MacVLANs and configure by hand rather than using the wizard.

Also, be aware that tagged VLANs did not work properly on the 3B+ last time I tried to use them. Lots of buggy behavior and errors in the log. This is actually the reason I moved away from the Raspberry Pi in favor of the MikroTik RBM33G for my ROOter-based (fork of OpenWRT) cellular router builds.

I use a Qotom i3-based industrial PC for the two OpenMPTCProuter setups I have up-and-running. It has 4 Gigabit Ethernet ports, 8GB of RAM, a 16GB SSD and a couple of USB3 ports -- I'd consider this close to the best available price/performance platform for OMR as of this writing. The RPi is a good way to get going with OMR though, and is an amazing SBC for the money.

18

I have a particular setup I have been working with and I am not sure if MPTCProuter can handle it. I am preparing for the upcoming volleyball season and with COVID there is a good chance we will be restricted on how many fans, if any, can attend. So I wanted to get a good setup for livestreaming and was looking at bonding multiple wireless wan connections. We would use multiple cell phones in Hotspot mode. This would require 3 total wireless interfaces on the MPTCProuter. 1 for the camera/phone capturing the video. 2 other wireless interfaces to connect to 2 separate wireless hotspots which would then be bonded.

Do you know if this is possible? I have played with it and not had much luck.

Thanks for any direction you can give.

Peter

19

Tcp for video is problematic... It resends packets for frames that happened a while ago while not yet sending the current images. In general UDP is a better choice for video, audio etc in real time. Not sure how mptcp really works but I think it's more about transferring large files than streaming video

20

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.