OpenWrt Forum Archive

Topic: Multi-WAN Load Balancing

The content of this topic has been archived between 29 Mar 2018 and 3 May 2018. Unfortunately there are posts – most likely complete pages – missing.

This script can handle more than two wans, and provides QoS for the wan links if qos is enabled on each wan link.

If you're using the script from https://forum.openwrt.org/viewtopic.php?id=23199, make sure to remove it prior to installing this one. (opkg remove luci-app-dualwan, opkg remove dualwan)

The Multi-WAN agent script itself:
ftp://ftp.netlab7.com/multiwan_1.0.18.ipk

And it's corresponding Luci Configuration Module:
ftp://ftp.netlab7.com/luci-app-multiwan_1.0.16.ipk

Update 1.0.18 - Changes per #7792 (Thanks buildster!):
a) make the process-killing of multiwan agent work
b) comment out a line of typo (fdown), which isn't needed anyway
     because ifup does ifdown first
c) run ifup in foreground because many heavy ifup processes
     crash/restart the system, probably running out of memory
d) introduce the specification of multiport (see iptables man page)
     and the ability to specify source-ports, as well as
     source-ports, or both
e) show logger messages if debug is on


Update: 1.0.17 - Copies all routes (except default) to multiwan routing tables for vpn/routing compatibility.

Any feedback is welcome, please let me know if anything doesn't work, or you run into any issues.

Thanks smile

(Last edited by SouthPawn on 14 Sep 2010, 19:55)

I would really appreciate it if you provide diffs this time. Thanks.

(Last edited by Dogge on 15 Mar 2010, 08:00)

Dogge wrote:

I would really appreciate it if you provide diffs this time. Thanks.

What would I be creating a diff against exactly?

Against the build-system with 'svn diff'.

You do realize it's a script right? And that there isn't really a build-system.

SouthPawn wrote:

You do realize it's a script right? And that there isn't really a build-system.

Well, providing patches is the standard procedure to get something included into trunk, also tracking changes is a pita with only binary packages provided (yes I know they contain just scripts and I could unpack and diff them).

Do you consider the multiwan stuff stable? In this case I'd commit it to the LuCI repo as source package so that the buildbots can pick it up and provide it in future snapshots.

~ JoW

jow wrote:
SouthPawn wrote:

You do realize it's a script right? And that there isn't really a build-system.

Well, providing patches is the standard procedure to get something included into trunk, also tracking changes is a pita with only binary packages provided (yes I know they contain just scripts and I could unpack and diff them).

Do you consider the multiwan stuff stable? In this case I'd commit it to the LuCI repo as source package so that the buildbots can pick it up and provide it in future snapshots.

~ JoW

Hey JoW,

Thanks for the info,

I am hoping to get some feedback on it first, positive or negative to make sure everything is working correctly, and running stable, currently I'm testing it only on a single WAN so I want to make sure that things like the load balancer and what not are functioning correctly. (The script can and will work on a single WAN.)

Any information you could provide on the aforementioned items would be greatly appreciated as I have no expertise in that department.

Thanks Again,
-Craig

If you provide a patch I can test with triple WAN (normal WAN, 3g as WAN and wireless as WAN)

(Last edited by Dogge on 15 Mar 2010, 19:33)

Hi SouthPawn,


Thanks for the nice script, will try it this weekend and report back. Nice to see triple (and more) wan feature!

Im curious to see how and why you are "using" QoS scripts... In my opinion the default OpenWRT QoS script/package works fine and needs no fidling, even with multiple wans.


Thnx again big_smile

Adze wrote:

Hi SouthPawn,


Thanks for the nice script, will try it this weekend and report back. Nice to see triple (and more) wan feature!

Im curious to see how and why you are "using" QoS scripts... In my opinion the default OpenWRT QoS script/package works fine and needs no fidling, even with multiple wans.


Thnx again big_smile

Correct, but we need to mark the traffic to say which WAN to use, qos-scripts uses these same connection marks. So if we let it stay the way it is, nothing would receive QoS except traffic that was specifically unmarked.

Since all traffic is marked for their prospective routes, we need to adapt the QoS to deal with new connection marks. 0x1 0x2 0x3 0x4 (as qos-scripts uses for it's markings) becomes 0x11 0x12 0x13 0x14 for the first wan and 0x21 0x22 0x23 0x24 for the second wan, and we need to adapt the tc filters and ip rules to corrispond as well.

Kind Regards,
-Craig

(Last edited by SouthPawn on 15 Mar 2010, 20:30)

Dogge wrote:

If you provide a patch I can test with triple WAN (normal WAN, 3g as WAN and wireless as WAN)

Please give me any guidance on this as possible, I'm not sure how to do what you're asking. smile

Thanks Dogge,

Dear SouthPawn,

SouthPawn wrote:

Any feedback is welcome, please let me know if anything doesn't work, or you run into any issues.

Thank you so much for this release.

I use it in France with 2 ADSL lines (free.fr and orange.fr) and it works like a charm...

(Last edited by mynetmemo on 15 Mar 2010, 21:16)

mynetmemo wrote:

Dear SouthPawn,

SouthPawn wrote:

Any feedback is welcome, please let me know if anything doesn't work, or you run into any issues.

Thank you so much for this release.

I use it in France with 2 ADSL lines (free.fr and orange.fr) and it works like a charm...

hi,
i would like to know, if this works with ssh and/or ftp connections as well?
i am running a simple 3xmultiwan setup on a asus wl500gp based on ip and i have problems using ftp or ssh sessions, they will abort spontaneously.
here my script:

#!/bin/sh

. /etc/functions.sh
include /lib/network
sleep 120
scan_interfaces

config_get WAN1 wan1 ifname
config_get WAN1_IP wan1 ipaddr
config_get WAN1_GATEWAY wan1 gateway
config_get WAN2 wan2 ifname
config_get WAN2_IP wan2 ipaddr
config_get WAN2_GATEWAY wan2 gateway
config_get WAN3 wan3 ifname
config_get WAN3_IP wan3 ipaddr
config_get WAN3_GATEWAY wan3 gateway

###check if all 3 pppoe connections are up and running
if [ $WAN1_IP='217.xxx.yyy.zzz' ] & [ $WAN2_IP='217.xxx.yyy.zzz' ] & [ $WAN3_IP='217.xxx.yyy.zzz' ];
then 
ip route add $WAN1_IP/32 dev $WAN1 src $WAN1_IP table wan1
ip route add default via $WAN1_GATEWAY table wan1
ip route add $WAN2_IP/32 dev $WAN2 src $WAN2_IP table wan2
ip route add default via $WAN2_GATEWAY table wan2
ip route add $WAN3_IP/32 dev $WAN3 src $WAN1_IP table wan3
ip route add default via $WAN3_GATEWAY table wan3
ip rule add from $WAN1_IP table wan1
ip rule add from $WAN2_IP table wan2
ip rule add from $WAN3_IP table wan3
ip route add default scope global nexthop via $WAN1_IP dev $WAN1 weight 1 nexthop via $WAN2_IP dev $WAN2 weight 1 nexthop via $WAN3_IP dev $WAN3 weight 1;
fi

thanks ulf

ulf_k wrote:

i would like to know, if this works with ssh and/or ftp connections as well?
i am running a simple 3xmultiwan setup on a asus wl500gp based on ip and i have problems using ftp or ssh sessions, they will abort spontaneously.

This should work without problem, as the multiwan script uses connection marking in netfilter for traffic management. The first rule being restore any existing connection marks, which means it'll restore related connection marks as well. (Such as passive connections made following the initial connection)

FTP fails when trying to do the load balancing solely with iproute2, because it won't detect that the passive connection going out is actually related to the prior port 21 connection to the same server, and it'll try to throw it out the nexthop.

Netfilter is able to track these connections with nat helpers.

(Last edited by SouthPawn on 26 Mar 2010, 01:24)

Hi Southpawn,


Your script works like a charm! Thank you for that.

I dont use the luci interface, but only the wan agent script itself. Having some trouble figuring all options out. Maybe you could make the config file a little more self-explanatory? Like some more examples about selecting specific wan interface for certain traffic.


Thank you !

SouthPawn wrote:

Please give me any guidance on this as possible, I'm not sure how to do what you're asking. smile

Checkout the build-system from subversion and integrate your package in the package feed. Then you can run 'svn diff' and you will have a patch ready.

Adze wrote:

Hi Southpawn,


Your script works like a charm! Thank you for that.

I dont use the luci interface, but only the wan agent script itself. Having some trouble figuring all options out. Maybe you could make the config file a little more self-explanatory? Like some more examples about selecting specific wan interface for certain traffic.


Thank you !

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'icmp_hosts' 'dns'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'balancer'

config 'interface' 'wan2'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'icmp_hosts' 'dns'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'wan3'

config 'interface' 'wan3'
        option 'weight' 'disable'
        option 'health_interval' 'disable'
        option 'icmp_hosts' 'gateway'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'disable'

config 'interface' 'wan4'
        option 'weight' '3'
        option 'health_interval' '20'
        option 'icmp_hosts' '208.67.222.222 208.67.220.220'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'wan'

config 'mwanfw'
        option 'src' '192.168.1.0/24'
        option 'dst' 'ftp.netlab7.com'
        option 'proto' 'tcp'
        option 'ports' '21'
        option 'wanrule' 'wan4'

config 'mwanfw'
        option 'proto' 'tcp'
        option 'ports' '21'
        option 'wanrule' 'wan2'

config 'mwanfw'
        option 'src' '192.168.0.3'
        option 'proto' 'icmp'
        option 'wanrule' 'wan'

config 'mwanfw'
        option 'dst' 'www.whatismyip.com'
        option 'wanrule' 'balancer'

Glad to hear it's working as expected! smile

Does this shed some more light on the config file?

Thanks,
-Craig

(Last edited by SouthPawn on 19 Mar 2010, 21:59)

Hi all,

I think I have got some issues getting the load balancer to do what I want :-)

First of all I want to thank SouthPawn for his nice work!

I have two ADSL lines with roughly 16MBit/s each. I configured two WAN interfaces to do PPPOE.
This is working so far. I am able to access the internet, but all the time only one WAN interface is used.

I tried downloading a Debian linux image via torrent. Multiple FTP / HTTP download.
Everytime only one WAN interface is used.

Please find attached some screenshots from my current config.

I guess I am missing something.
Thanks for your help in advance.

Routes
Concerning the "route" output I have a comment. I am sure that I sometimes saw two default routes. What would be the correct behavior?
http://img710.imageshack.us/img710/3076/routet.jpg

Switch Config
http://img195.imageshack.us/img195/1130/switchconfig.jpg

WAN Interfaces
For the WAN interfaces - which zone must be used? I am using the same zone for both WAN interfaces.
http://img196.imageshack.us/img196/2954/wanl.jpg
http://img92.imageshack.us/img92/8346/wan2.jpg

MultiWAN
http://img704.imageshack.us/img704/3848/multiwan.jpg

Interfaces Status
http://img517.imageshack.us/img517/5677/statusinterfaces.jpg


Best regards,
Patrik

Hey Patrik,

Thank you for the kind words, go ahead and remove wan2 from the Multi-WAN configuration, and add in it's place wan1.
This should take care of it, let me know if there are any issues after this.

Thanks Patrik,
-Craig

P.S. That outgoing rule for everything to go to the load balancer is unnecessary, as you selected the default route to be the load balancer anyhow. smile

(Last edited by SouthPawn on 23 Mar 2010, 17:07)

Hey Craig,

thanks for your fast reply.
I did the config change and my /etc/config/multiwan looks like this now:

root@OpenWrt:~# cat /etc/config/multiwan

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'disable'
        option 'icmp_hosts' 'disable'

config 'interface' 'wan1'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'icmp_hosts' 'disable'
        option 'failover_to' 'disable'

config 'mwanfw'
        option 'wanrule' 'balancer'

routing table looks like this

root@OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp1
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     *               255.255.255.0   U     0      0        0 br-lan
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp1
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp0
root@OpenWrt:~#

I am running 4 downloads from different http/ftp servers right now and still all traffic goes through WAN1 :-(

I dont know what I am doing wrong.

root@OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp1
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     *               255.255.255.0   U     0      0        0 br-lan
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp1
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp0
root@OpenWrt:~#

Looks like there is something wrong with your network setup. The gateways on both ppp interfaces have the same ip address... Dont think it is a multiwan issue. Try to get the routing table entries corrected by unsetting the "replace default route" setting, before enabling multiwan, to be sure it isn't a multiwan issue.

I will uncheck the "Let pppd replace the current default route to use the PPP interface after successful connect" for both wan connections now.
The gateways of both WAN connections are probably the same because of the fact that I have got two ADSL lines from the same provider.
I will come back with an update shortly.

Thanks and best regards,
Patrik

//edit
Still the same issue. Nothing changes after clearing the checkboxes.

routing table did not change...

root@OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp1
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     *               255.255.255.0   U     0      0        0 br-lan
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp1
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp0

any other ideas?

(Last edited by Patrik85 on 23 Mar 2010, 22:22)

I believe this issue is caused by both wan links being on the same subnet, sharing the same gateway.
I don't think there will be a workaround possible without introducing something to change that. (such as a another router behind one of the wans.)

(Last edited by SouthPawn on 23 Mar 2010, 23:23)

I changed the config as follows with the result that still all traffic goes through wan1 :-(

root@OpenWrt:~# cat /etc/config/multiwan

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'disable'
        option 'icmp_hosts' 'disable'

config 'interface' 'wan1'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'icmp_hosts' 'disable'
        option 'failover_to' 'disable'

config 'mwanfw'
        option 'wanrule' 'wan'

root@OpenWrt:~#

A solution to this would be to use the modem (if supported) or another router ahead of us to perform the natting, so that the router can differentiate the paths.

This doesn't necessarily mean you need to double nat, once the modem or router ahead is performing the natting, we can create static routes and disable the masquerading locally for the chosen wan link.

Or, you can just double nat the second uplink by simply sticking another router behind one of the modems.

This isn't a multiwan issue as its more of a general routing issue.

(Last edited by SouthPawn on 24 Mar 2010, 00:03)