Hello,

I am trying to learn about the possibilities (and limitations) for link bonding / link aggregation with OpenWRT. It might even be better to call it parallel routing.  I have focused more on BSD variants for my routing needs to date, but the best hardware at hand is OpenWRT only, so I would really appreciate insight, links, thoughts, relevant documentation, etc. to get me going. I don't need total solutions, but data, experiences, anecdotes, etc. are all helpful.

First, I would like to create a virtual interface which aggregates 1-3 physical ethernet interfaces (all on the same LAN) and as much as 10-50 'routes' (imagine a small LAN of OpenWRT systems and each 'route' is a physical ethernet interface on an OpenWRT board) where each route sends traffic to the  same single destination. The goal would be to have a 'pool' of routes and balance traffic (on a packet level perhaps) among all these different routes, with the ability to add or remove routes without significant downtime, and to have bad routes automatically offline'd. (And hopefully automatically monitored and enabled should they come back online.)

Second, I would also like to create another virtual aggregated interface for wireless radios, except it would aggregate 2-3 MiniPCI wireless radios at most, and each radio would directly talk to another OpenWRT system on the other end with a similar configuration. (e.g. 1 interface = 1 radio = link to 1 remote OpenWRT system) Due to the nature of wireless, adaptive load balancing aggregation would be important, as is the ability to have a failed link automatically be taken out of the aggregation pool.

The end goal is to have a single fast server on one end, and a number of clients on the other end, with OpenWRT configurations as described above moving the traffic between them. The core functional objective is facilitate communication between the client and the fast server, and only UDP, TCP, ICMP traffic. No broadcast traffic, and client-to-client communications are not a concern.

The purpose of this complex infrastructure is to enhance both capacity and reliability. I leverage the switched LAN as a sort of 'aggregation point' so that the traffic load can be evenly distributed among all available OpenWRT boards with wireless links. I realize this creates potentially a very complex aggregation scenario with 10, 20, 30 even 50 routes being possible. However, this also means that the loss of a single OpenWRT board or wireless link will have a very small impact on overall traffic flows.

Because all routes (wired and wireless) operate in a parallel and flat manner (hence "parallel routing"), so we're not going to end up with situations where traffic gets routed in ways that are dramatically slower, etc. and the latency on these links should be very low since they are really just LAN links. Combined with the clear client/server communication goal means that I think an adaptive, packet-level load balancing / link aggregation approach could work well here.

Honestly, I'm not sure if it's reasonable or not to aggregate this many different routes. I also don't know the kind of CPU we're talking about needing here. The boards in question here are 800 MHz MIPS CPUs. I suspect if we're simply routing packets and planning for 180-200 mbit/sec peak throughput, we can probably do it. (We don't expect too very small packets.)

I have created a simplified diagram here. I have left off (and noted where applicable) more than just a single "path" of hardware, but the core idea here is that this should be usable with one, three, ten OpenWRT boards operating in parallel. Chances are that there will be more client-serving OpenWRT boards than bridging-only OpenWRT boards at the remote site.

Hi, blueandwhiteg3.

Bonding driver might help you.
http://www.mjmwired.net/kernel/Document … onding.txt
for part of virtual aggregated ethernet/wireless interface.

It is there.
https://dev.openwrt.org/browser/trunk/p … e/Makefile

zukky

yapoo, that is some helpful documentation and good confirmation of support in OpenWRT.

I'm a little indeterminate as to the best option for aggregating wireless links. None of the models seem quite ideal. One of the challenges is that we have essentially dynamically variable throughput. Even if we have a defined PHY bitrate and perfect link conditions, we will still have MAC-level re-transmits when collisions happen, lowering throughput. In more realistic conditions, occasional interference and varying link conditions will result in even larger bitrate variances.

Ideally, it seems best to move a continual stream of ICMP traffic and use the latency and packet loss statistics to assess the viability of each link and to balance in a manner that tries to keep these uniform among links.

Another perhaps acceptable approach would be if load was distributed based on current PHY bitrate. Do current Atheros drivers support outputting actual current PHY bitrate via ethtool? (In the past I've seen some wireless drivers reporting peak theoretical PHY bitrate, regardless of operating bitrate.) Particularly if we adjusted the driver's PHY bitrate selection algorithm to be slightly conservative, we could help ensure our PHY bitrate and realistic bitrate are as closely correlated as possible.

As for wired ethernet links, it looks like balance-rr (0) may be the best option for aggregating physical ports as the links have predictable throughput. I suppose the other models might be theoretically better, but I am pretty sure that 100 mbit/sec of very mixed traffic packets is going to average nearly perfectly, and be very predictable, while the other models seem more vulnerable to issues and could perhaps even lead to unexpected issues in some cases.

The really messy part appears to be the fact that none of these bonding methods handle the routing aspect of things - see the above example where I have two NICs leading to a switched LAN with many more than 2 possible routes. It seems like I would have to manually manage the routes on all the machines! This sounds like it could get pretty messy. Are there other solutions?