1 (edited by SouthPawn 2010-09-14 19:55:17)

Topic: Multi-WAN Load Balancing

This script can handle more than two wans, and provides QoS for the wan links if qos is enabled on each wan link.

If you're using the script from https://forum.openwrt.org/viewtopic.php?id=23199, make sure to remove it prior to installing this one. (opkg remove luci-app-dualwan, opkg remove dualwan)

The Multi-WAN agent script itself:
ftp://ftp.netlab7.com/multiwan_1.0.18.ipk

And it's corresponding Luci Configuration Module:
ftp://ftp.netlab7.com/luci-app-multiwan_1.0.16.ipk

Update 1.0.18 - Changes per #7792 (Thanks buildster!):
a) make the process-killing of multiwan agent work
b) comment out a line of typo (fdown), which isn't needed anyway
     because ifup does ifdown first
c) run ifup in foreground because many heavy ifup processes
     crash/restart the system, probably running out of memory
d) introduce the specification of multiport (see iptables man page)
     and the ability to specify source-ports, as well as
     source-ports, or both
e) show logger messages if debug is on


Update: 1.0.17 - Copies all routes (except default) to multiwan routing tables for vpn/routing compatibility.

Any feedback is welcome, please let me know if anything doesn't work, or you run into any issues.

Thanks smile

2 (edited by Dogge 2010-03-15 08:00:57)

Re: Multi-WAN Load Balancing

I would really appreciate it if you provide diffs this time. Thanks.

Re: Multi-WAN Load Balancing

Dogge wrote:

I would really appreciate it if you provide diffs this time. Thanks.

What would I be creating a diff against exactly?

4

Re: Multi-WAN Load Balancing

Against the build-system with 'svn diff'.

Re: Multi-WAN Load Balancing

You do realize it's a script right? And that there isn't really a build-system.

Re: Multi-WAN Load Balancing

SouthPawn wrote:

You do realize it's a script right? And that there isn't really a build-system.

Well, providing patches is the standard procedure to get something included into trunk, also tracking changes is a pita with only binary packages provided (yes I know they contain just scripts and I could unpack and diff them).

Do you consider the multiwan stuff stable? In this case I'd commit it to the LuCI repo as source package so that the buildbots can pick it up and provide it in future snapshots.

~ JoW

Re: Multi-WAN Load Balancing

jow wrote:
SouthPawn wrote:

You do realize it's a script right? And that there isn't really a build-system.

Well, providing patches is the standard procedure to get something included into trunk, also tracking changes is a pita with only binary packages provided (yes I know they contain just scripts and I could unpack and diff them).

Do you consider the multiwan stuff stable? In this case I'd commit it to the LuCI repo as source package so that the buildbots can pick it up and provide it in future snapshots.

~ JoW

Hey JoW,

Thanks for the info,

I am hoping to get some feedback on it first, positive or negative to make sure everything is working correctly, and running stable, currently I'm testing it only on a single WAN so I want to make sure that things like the load balancer and what not are functioning correctly. (The script can and will work on a single WAN.)

Any information you could provide on the aforementioned items would be greatly appreciated as I have no expertise in that department.

Thanks Again,
-Craig

8 (edited by Dogge 2010-03-15 19:33:06)

Re: Multi-WAN Load Balancing

If you provide a patch I can test with triple WAN (normal WAN, 3g as WAN and wireless as WAN)

Re: Multi-WAN Load Balancing

Hi SouthPawn,


Thanks for the nice script, will try it this weekend and report back. Nice to see triple (and more) wan feature!

Im curious to see how and why you are "using" QoS scripts... In my opinion the default OpenWRT QoS script/package works fine and needs no fidling, even with multiple wans.


Thnx again big_smile

10 (edited by SouthPawn 2010-03-15 20:30:29)

Re: Multi-WAN Load Balancing

Adze wrote:

Hi SouthPawn,


Thanks for the nice script, will try it this weekend and report back. Nice to see triple (and more) wan feature!

Im curious to see how and why you are "using" QoS scripts... In my opinion the default OpenWRT QoS script/package works fine and needs no fidling, even with multiple wans.


Thnx again big_smile

Correct, but we need to mark the traffic to say which WAN to use, qos-scripts uses these same connection marks. So if we let it stay the way it is, nothing would receive QoS except traffic that was specifically unmarked.

Since all traffic is marked for their prospective routes, we need to adapt the QoS to deal with new connection marks. 0x1 0x2 0x3 0x4 (as qos-scripts uses for it's markings) becomes 0x11 0x12 0x13 0x14 for the first wan and 0x21 0x22 0x23 0x24 for the second wan, and we need to adapt the tc filters and ip rules to corrispond as well.

Kind Regards,
-Craig

Re: Multi-WAN Load Balancing

Dogge wrote:

If you provide a patch I can test with triple WAN (normal WAN, 3g as WAN and wireless as WAN)

Please give me any guidance on this as possible, I'm not sure how to do what you're asking. smile

Thanks Dogge,

12 (edited by mynetmemo 2010-03-15 21:16:32)

Re: Multi-WAN Load Balancing

Dear SouthPawn,

SouthPawn wrote:

Any feedback is welcome, please let me know if anything doesn't work, or you run into any issues.

Thank you so much for this release.

I use it in France with 2 ADSL lines (free.fr and orange.fr) and it works like a charm...

Re: Multi-WAN Load Balancing

mynetmemo wrote:

Dear SouthPawn,

SouthPawn wrote:

Any feedback is welcome, please let me know if anything doesn't work, or you run into any issues.

Thank you so much for this release.

I use it in France with 2 ADSL lines (free.fr and orange.fr) and it works like a charm...

hi,
i would like to know, if this works with ssh and/or ftp connections as well?
i am running a simple 3xmultiwan setup on a asus wl500gp based on ip and i have problems using ftp or ssh sessions, they will abort spontaneously.
here my script:

#!/bin/sh

. /etc/functions.sh
include /lib/network
sleep 120
scan_interfaces

config_get WAN1 wan1 ifname
config_get WAN1_IP wan1 ipaddr
config_get WAN1_GATEWAY wan1 gateway
config_get WAN2 wan2 ifname
config_get WAN2_IP wan2 ipaddr
config_get WAN2_GATEWAY wan2 gateway
config_get WAN3 wan3 ifname
config_get WAN3_IP wan3 ipaddr
config_get WAN3_GATEWAY wan3 gateway

###check if all 3 pppoe connections are up and running
if [ $WAN1_IP='217.xxx.yyy.zzz' ] & [ $WAN2_IP='217.xxx.yyy.zzz' ] & [ $WAN3_IP='217.xxx.yyy.zzz' ];
then 
ip route add $WAN1_IP/32 dev $WAN1 src $WAN1_IP table wan1
ip route add default via $WAN1_GATEWAY table wan1
ip route add $WAN2_IP/32 dev $WAN2 src $WAN2_IP table wan2
ip route add default via $WAN2_GATEWAY table wan2
ip route add $WAN3_IP/32 dev $WAN3 src $WAN1_IP table wan3
ip route add default via $WAN3_GATEWAY table wan3
ip rule add from $WAN1_IP table wan1
ip rule add from $WAN2_IP table wan2
ip rule add from $WAN3_IP table wan3
ip route add default scope global nexthop via $WAN1_IP dev $WAN1 weight 1 nexthop via $WAN2_IP dev $WAN2 weight 1 nexthop via $WAN3_IP dev $WAN3 weight 1;
fi

thanks ulf

14 (edited by SouthPawn 2010-03-26 01:24:58)

Re: Multi-WAN Load Balancing

ulf_k wrote:

i would like to know, if this works with ssh and/or ftp connections as well?
i am running a simple 3xmultiwan setup on a asus wl500gp based on ip and i have problems using ftp or ssh sessions, they will abort spontaneously.

This should work without problem, as the multiwan script uses connection marking in netfilter for traffic management. The first rule being restore any existing connection marks, which means it'll restore related connection marks as well. (Such as passive connections made following the initial connection)

FTP fails when trying to do the load balancing solely with iproute2, because it won't detect that the passive connection going out is actually related to the prior port 21 connection to the same server, and it'll try to throw it out the nexthop.

Netfilter is able to track these connections with nat helpers.

Re: Multi-WAN Load Balancing

Hi Southpawn,


Your script works like a charm! Thank you for that.

I dont use the luci interface, but only the wan agent script itself. Having some trouble figuring all options out. Maybe you could make the config file a little more self-explanatory? Like some more examples about selecting specific wan interface for certain traffic.


Thank you !

16

Re: Multi-WAN Load Balancing

SouthPawn wrote:

Please give me any guidance on this as possible, I'm not sure how to do what you're asking. smile

Checkout the build-system from subversion and integrate your package in the package feed. Then you can run 'svn diff' and you will have a patch ready.

17 (edited by SouthPawn 2010-03-19 21:59:17)

Re: Multi-WAN Load Balancing

Adze wrote:

Hi Southpawn,


Your script works like a charm! Thank you for that.

I dont use the luci interface, but only the wan agent script itself. Having some trouble figuring all options out. Maybe you could make the config file a little more self-explanatory? Like some more examples about selecting specific wan interface for certain traffic.


Thank you !

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'icmp_hosts' 'dns'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'balancer'

config 'interface' 'wan2'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'icmp_hosts' 'dns'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'wan3'

config 'interface' 'wan3'
        option 'weight' 'disable'
        option 'health_interval' 'disable'
        option 'icmp_hosts' 'gateway'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'disable'

config 'interface' 'wan4'
        option 'weight' '3'
        option 'health_interval' '20'
        option 'icmp_hosts' '208.67.222.222 208.67.220.220'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'wan'

config 'mwanfw'
        option 'src' '192.168.1.0/24'
        option 'dst' 'ftp.netlab7.com'
        option 'proto' 'tcp'
        option 'ports' '21'
        option 'wanrule' 'wan4'

config 'mwanfw'
        option 'proto' 'tcp'
        option 'ports' '21'
        option 'wanrule' 'wan2'

config 'mwanfw'
        option 'src' '192.168.0.3'
        option 'proto' 'icmp'
        option 'wanrule' 'wan'

config 'mwanfw'
        option 'dst' 'www.whatismyip.com'
        option 'wanrule' 'balancer'

Glad to hear it's working as expected! smile

Does this shed some more light on the config file?

Thanks,
-Craig

Re: Multi-WAN Load Balancing

Hi all,

I think I have got some issues getting the load balancer to do what I want :-)

First of all I want to thank SouthPawn for his nice work!

I have two ADSL lines with roughly 16MBit/s each. I configured two WAN interfaces to do PPPOE.
This is working so far. I am able to access the internet, but all the time only one WAN interface is used.

I tried downloading a Debian linux image via torrent. Multiple FTP / HTTP download.
Everytime only one WAN interface is used.

Please find attached some screenshots from my current config.

I guess I am missing something.
Thanks for your help in advance.

Routes
Concerning the "route" output I have a comment. I am sure that I sometimes saw two default routes. What would be the correct behavior?
http://img710.imageshack.us/img710/3076/routet.jpg

Switch Config
http://img195.imageshack.us/img195/1130/switchconfig.jpg

WAN Interfaces
For the WAN interfaces - which zone must be used? I am using the same zone for both WAN interfaces.
http://img196.imageshack.us/img196/2954/wanl.jpg
http://img92.imageshack.us/img92/8346/wan2.jpg

MultiWAN
http://img704.imageshack.us/img704/3848/multiwan.jpg

Interfaces Status
http://img517.imageshack.us/img517/5677/statusinterfaces.jpg


Best regards,
Patrik

19 (edited by SouthPawn 2010-03-23 17:07:52)

Re: Multi-WAN Load Balancing

Hey Patrik,

Thank you for the kind words, go ahead and remove wan2 from the Multi-WAN configuration, and add in it's place wan1.
This should take care of it, let me know if there are any issues after this.

Thanks Patrik,
-Craig

P.S. That outgoing rule for everything to go to the load balancer is unnecessary, as you selected the default route to be the load balancer anyhow. smile

Re: Multi-WAN Load Balancing

Hey Craig,

thanks for your fast reply.
I did the config change and my /etc/config/multiwan looks like this now:

root@OpenWrt:~# cat /etc/config/multiwan

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'disable'
        option 'icmp_hosts' 'disable'

config 'interface' 'wan1'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'icmp_hosts' 'disable'
        option 'failover_to' 'disable'

config 'mwanfw'
        option 'wanrule' 'balancer'

routing table looks like this

root@OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp1
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     *               255.255.255.0   U     0      0        0 br-lan
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp1
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp0
root@OpenWrt:~#

I am running 4 downloads from different http/ftp servers right now and still all traffic goes through WAN1 :-(

I dont know what I am doing wrong.

Re: Multi-WAN Load Balancing

root@OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp1
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     *               255.255.255.0   U     0      0        0 br-lan
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp1
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp0
root@OpenWrt:~#

Looks like there is something wrong with your network setup. The gateways on both ppp interfaces have the same ip address... Dont think it is a multiwan issue. Try to get the routing table entries corrected by unsetting the "replace default route" setting, before enabling multiwan, to be sure it isn't a multiwan issue.

22 (edited by Patrik85 2010-03-23 22:22:08)

Re: Multi-WAN Load Balancing

I will uncheck the "Let pppd replace the current default route to use the PPP interface after successful connect" for both wan connections now.
The gateways of both WAN connections are probably the same because of the fact that I have got two ADSL lines from the same provider.
I will come back with an update shortly.

Thanks and best regards,
Patrik

//edit
Still the same issue. Nothing changes after clearing the checkboxes.

routing table did not change...

root@OpenWrt:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp1
217.0.116.36    *               255.255.255.255 UH    0      0        0 ppp0
192.168.2.0     *               255.255.255.0   U     0      0        0 br-lan
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp1
default         217.0.116.36    0.0.0.0         UG    0      0        0 ppp0

any other ideas?

23 (edited by SouthPawn 2010-03-23 23:23:45)

Re: Multi-WAN Load Balancing

I believe this issue is caused by both wan links being on the same subnet, sharing the same gateway.
I don't think there will be a workaround possible without introducing something to change that. (such as a another router behind one of the wans.)

Re: Multi-WAN Load Balancing

I changed the config as follows with the result that still all traffic goes through wan1 :-(

root@OpenWrt:~# cat /etc/config/multiwan

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'failover_to' 'disable'
        option 'icmp_hosts' 'disable'

config 'interface' 'wan1'
        option 'weight' '5'
        option 'health_interval' '10'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'icmp_hosts' 'disable'
        option 'failover_to' 'disable'

config 'mwanfw'
        option 'wanrule' 'wan'

root@OpenWrt:~#

25 (edited by SouthPawn 2010-03-24 00:03:01)

Re: Multi-WAN Load Balancing

A solution to this would be to use the modem (if supported) or another router ahead of us to perform the natting, so that the router can differentiate the paths.

This doesn't necessarily mean you need to double nat, once the modem or router ahead is performing the natting, we can create static routes and disable the masquerading locally for the chosen wan link.

Or, you can just double nat the second uplink by simply sticking another router behind one of the modems.

This isn't a multiwan issue as its more of a general routing issue.