Download

For Chaos Calmer 15.05.

Internet bonding package for link aggregation available on github. Please read the BONDING.md

Introduction

For over year, I have been running an internet link that is comprised of two ADSL annex M lines bonded together using  modified kernel bonding driver, openvpn and a bunch of scripts. The performance is pretty good, about 90% of 2x the speed of the slowest link.

I recall when I put this together that there was no solution except for a couple of black-box solutions from commercial providers. There still isn't a solution today, over a year later. As such, I've finally gotten around to packaging up my scripts and driver so that it just installs and is usable by a moderately skilled person.

Setup requires a reasonable level of skill, although it is all configured via standard OpenWrt UCI config and init.d scripts. You will, however, need to setup the server yourself and understand what you're doing, otherwise problem solving becomes tricky.

You will need a hardware platform for OpenWrt that is capable of the necessary AES throughput that can support wire speed encryption. For this project, I built my own router as most commercial consumer routers tend to top out at about 15Mbps of throughput. I used a platform based on the Intel C2558 SoC, using a Supermicro A1SRi-2558F motherboard. This is a 64 bit 4 core Atom processor with AES-NI hardware acceleration capable of sustaining 3 - 4Gbps of AES throughput. It's important to note that if you don't use a powerful enough platform, you simply won't get the performance you're expecting.

You will also need to get a VM in a datacentre to serve as the endpoint for your bonded connection. The most important thing (apart from the pricing of your host) is the latency between you and the datacentre, as this will impact your maximum achievable speed. The latency between my router and the datacentre is 5ms. You also need to have a sufficiently powerful instance that can support your wirespeed encryption. Depending on the OS you use, you may also need to make a kernel modification (Centos 6 requires no kernel mod, whereas Centos 7 does). Your VM provider will need to be able to allocate you two static IP addresses on your VM.

In addition, you will need two internet links. I bonded two ADSL links from BT, opting for one of their business class connections with a higher quality of service than the consumer connections (priority over all consumer traffic). For each connection, you will require a static IP address. It's important to get two links that are as symmetrical as possible in terms of latency and sync speed, as the maximum performance is approximately 90% of 2x the slowest link.

In a nutshell, this is an expensive solution. It's cheaper to get a faster line than to bond two lines together. I only did this because my road was bypassed by fibre. Total cost for the lines and the VM amounts to about £110 pounds sterling per month. The ADSL business class lines are expensive and you could definitely get a cheaper solution by opting for a consumer class connection if your provider can offer static IP addresses (normally only the province of a business class line), but the business class connections offer substantially less latency and so contribute significantly to an improved throughput.

If you have good enough hardware on both ends, there's nothing stopping you bonding together two Gbps fibre links (although you'll need 10Gbe cards of course)...so this may be a solution for speed freaks for whom money is no object.

Note

One of the keys to setting this up properly, apart from the bonding driver and appropriate source routing is not to have Openvpn assign IP addresses to the tun interfaces on either end of the connection. The IP addresses are assigned to the bond0 interface, and since the bonding driver assigns the tun interfaces as slaves, it automatically round robins packets onto each interface which are then picked up by Openvpn and sent to the other end of the tunnel. Here is a ifconfig of the bond0 interface and tun0/tun1 interfaces after dumping a 100MB file across the bonded link. Note how the interfaces are labelled as MASTER for bond0 and SLAVE for the tunX interfaces. Note also how the aggregate throughput is 100MB on the bond master and 50MB each on the tun devices.

root@openwrt:/sys/devices/virtual/net/bond0/bonding# ifconfig bond0
bond0     Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.27.105.2  Bcast:10.255.255.255  Mask:255.0.0.0
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:79576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82526 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:105025001 (100.1 MiB)  TX bytes:6354903 (6.0 MiB)

root@openwrt:/sys/devices/virtual/net/bond0/bonding# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          UP POINTOPOINT RUNNING NOARP SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:39800 errors:0 dropped:0 overruns:0 frame:0
          TX packets:41272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:52534101 (50.0 MiB)  TX bytes:3179354 (3.0 MiB)

root@openwrt:/sys/devices/virtual/net/bond0/bonding# ifconfig tun1
tun1      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          UP POINTOPOINT RUNNING NOARP SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:39778 errors:0 dropped:0 overruns:0 frame:0
          TX packets:41254 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:52491020 (50.0 MiB)  TX bytes:3175549 (3.0 MiB)

(Last edited by dl12345 on 8 Sep 2016, 00:50)