Re: Multi-WAN Load Balancing

hi,

hold off on that last order. i think issue 1. is caused by issue 2.
my dsl blinked and i got this:

=====
root@culiat-wg:~# ip route show table 123
121.96.128.1 dev ppp0  proto kernel  scope link  src 121.96.157.113
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.15
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
default
        nexthop via 114.108.202.1  dev eth0.1 weight 1
        nexthop via 121.96.128.1  dev ppp0 weight 1
=====

not static proto but i still have multipath. and the interfaces logs show wan2 still routing packets.
i won't delete/edit post #50 so people can check their connections before suspecting multiwan.

Re: Multi-WAN Load Balancing

btw, the significance of the above is everything happened during the same internet session.
here's wan2 routing alongside wan after the blink:
--
Interface wan
RX: 296347 Pkts. (239.31 MB)
TX: 246788 Pkts. (57.27 MB)
           
Interface wan2
RX: 22062 Pkts. (13.51 MB)
TX: 27993 Pkts. (13.81 MB)

53 (edited by SouthPawn 2010-04-04 07:00:16)

Re: Multi-WAN Load Balancing

Hey Andy,

If health monitor is enabled on that interface, even if the Multi-WAN script is loaded before the ppp connection is established, it will detect the changes and update the routing tables and netfilter rules when it becomes available and ready for use.
I did post a change, specifying the static for routes of the other tables, as this is more appropriate, but this will not have any effect on your current performance.

Thanks,
- Craig

Re: Multi-WAN Load Balancing

hi craig,

thanks for the update. i guess i'll just download it and test it next weekend. vacation's over so i'm back to work again.
stupid question but where do i turn on the health monitor? big_smile
although, i do see the routing adjust when my ip address changes. right now i'm itching to do a `ip route show table 123 | grep default | wc -l` if that's not equal to 2 then i'd restart multiwan.
reason why i'm still bickering over the late interface is i'm still having the issue although i may be the only one seeing the difference, i'm sure it's very common since heterogeneous connections are common for fault tolerance and i'm sure the pppoe authentication delay vs the dhcp ack is causing the disparity.
here's what i have so far (this is the i version with the hand fixed luci. not the k yet. ):
=====
root@culiat-wg:~# ip route show table 123
202.78.96.80 dev ppp0  proto kernel  scope link  src 110.55.231.21
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.179
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
default via 114.108.202.1 dev eth0.1
root@culiat-wg:~# /etc/init.d/multiwan restart
root@culiat-wg:~# ip route show table 123
202.78.96.80 dev ppp0  proto kernel  scope link  src 110.55.231.21
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.179
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
root@culiat-wg:~# ip route show table 123
202.78.96.80 dev ppp0  proto kernel  scope link  src 110.55.231.21
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.179
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
default
        nexthop via 114.108.202.1  dev eth0.1 weight 1
        nexthop via 202.78.96.80  dev ppp0 weight 1
=====
that's after more than 10 minutes that the router has been powered on and i checked that the pppoe connection was already up.
notice multipath started working after i restarted the script. i'm sure there is a cleaner solution to this but as i said probably checking the interfaces, 10 seconds apart after starting multiwan could be the solution OR the health monitor but pls tell me where to turn it on. i'm using backfire 10.03-rc2.
i'll test the k version when i get the chance and give more feedback.
overall, it's a superb implementation of multipath networking. i think yours is the only one i came upon that really worked. fyi, i've been doing multipath routing since lokiwall and you'll get really tired tracing the scripts to make it work. yours worked out of the box. my initial setup using your script was one isp on each router that's why i didn't have the lagging pppoe issue before. i got bold and connected both isp on one router with the other router just used to spread the workload that's why i'm seeing the delay now. smile

regards,
andy

Re: Multi-WAN Load Balancing

andyballon wrote:

hi craig,

thanks for the update. i guess i'll just download it and test it next weekend. vacation's over so i'm back to work again.
stupid question but where do i turn on the health monitor? big_smile
although, i do see the routing adjust when my ip address changes. right now i'm itching to do a `ip route show table 123 | grep default | wc -l` if that's not equal to 2 then i'd restart multiwan.
reason why i'm still bickering over the late interface is i'm still having the issue although i may be the only one seeing the difference, i'm sure it's very common since heterogeneous connections are common for fault tolerance and i'm sure the pppoe authentication delay vs the dhcp ack is causing the disparity.
here's what i have so far (this is the i version with the hand fixed luci. not the k yet. ):
=====
root@culiat-wg:~# ip route show table 123
202.78.96.80 dev ppp0  proto kernel  scope link  src 110.55.231.21
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.179
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
default via 114.108.202.1 dev eth0.1
root@culiat-wg:~# /etc/init.d/multiwan restart
root@culiat-wg:~# ip route show table 123
202.78.96.80 dev ppp0  proto kernel  scope link  src 110.55.231.21
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.179
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
root@culiat-wg:~# ip route show table 123
202.78.96.80 dev ppp0  proto kernel  scope link  src 110.55.231.21
114.108.202.0/24 dev eth0.1  proto kernel  scope link  src 114.108.202.179
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
default
        nexthop via 114.108.202.1  dev eth0.1 weight 1
        nexthop via 202.78.96.80  dev ppp0 weight 1
=====
that's after more than 10 minutes that the router has been powered on and i checked that the pppoe connection was already up.
notice multipath started working after i restarted the script. i'm sure there is a cleaner solution to this but as i said probably checking the interfaces, 10 seconds apart after starting multiwan could be the solution OR the health monitor but pls tell me where to turn it on. i'm using backfire 10.03-rc2.
i'll test the k version when i get the chance and give more feedback.
overall, it's a superb implementation of multipath networking. i think yours is the only one i came upon that really worked. fyi, i've been doing multipath routing since lokiwall and you'll get really tired tracing the scripts to make it work. yours worked out of the box. my initial setup using your script was one isp on each router that's why i didn't have the lagging pppoe issue before. i got bold and connected both isp on one router with the other router just used to spread the workload that's why i'm seeing the delay now. smile

regards,
andy

Hey Andy,

Looking at the problem your experiencing it's either,

A. The firewall is blocking the ICMP requests/responses, or B. The DNS servers are not responding to an icmp ping.

The reason it keeps getting dropped of the load balancer routing table (123), is the Multi-WAN script beleives it's a failed link. (Check syslog to verify this.)

Try selecting Gateway or Disable on the Health Monitor ICMP Host setting.

Thanks Andy,
-Craig

56

Re: Multi-WAN Load Balancing

Craig,

I converted the .lua translation file to the appropriate bianry format used in recent revisions, you can find it here:
http://luci.subsignal.org/~jow/mw-i18n/

The "multiwan.en.lmo" file must be placed into /usr/lib/lua/luci/i18n/ .

~ JoW

Re: Multi-WAN Load Balancing

hi craig,

thanks for the update. that may be my problem. i have health monitor turned off. yes i checked before and an isp's dns server was not responding to pings that's why i disabled it. i was planning to put up my own check but i'm not sure if dead gateway detection is properly implemented in the openwrt's kernel. anyway, i haven't got time now to check that but i think last word i got from the internet is it's properly implemented in the 2.6 kernel. that's why i'm updated to 2.6 kernel. anyway, your scripts are giving me more download than the higher rated isp so that means it's working. (btw, when i say that i'm not saying i'm just downloading 1 big file. what i mean is i'm downloading several files that when you add them up it's going more than the higher rated internet connection.)
btw, i mentioned lokiwall as a relic above but upon checking they're active again. smile that's a pretty cool application. and they got a gui now! makes me look double silly. big_smile
so, i'll turn on health check later when i get home and see how everything goes.
btw#2, if you have not implemented this in the k version maybe look into it?
"    ip rule add prio 5 table main
    ip route del default table main
"
i was doing that after running multiwan but forgot to mention it before. that puts the main table ahead of the default table and the main table is the one that does mutlipath routing. i'm not sure if that's still needed but it's in nano.txt.

thanks,
andy

58 (edited by SouthPawn 2010-04-05 16:54:02)

Re: Multi-WAN Load Balancing

andyballon wrote:

hi craig,
btw#2, if you have not implemented this in the k version maybe look into it?
"    ip rule add prio 5 table main
    ip route del default table main
"
i was doing that after running multiwan but forgot to mention it before. that puts the main table ahead of the default table and the main table is the one that does mutlipath routing. i'm not sure if that's still needed but it's in nano.txt.

thanks,
andy

Each of the routing tables set up by the multiwan have a copy of the main routing table, a default gateway will be required regardless of whether we use it or not. I could set the default gateway to the loopback address 127.0.0.1, but I find it more human readable to keep both default gateways there, as well it's a point a reference to see if the routing table has been changed by some external event.

Thanks Andy,
-Craig

59 (edited by rockpilp 2010-04-07 21:14:15)

Re: Multi-WAN Load Balancing

Hi Craig,

As an early poster on the Dual-Wan thread, I wanted to congratulate you on the great improvements you've made in the past few weeks/months.

I do agree with some of the other commenters that the LuCi module could use some help text.

Here are some suggestions:

http://s3.amazonaws.com/ember/tQh7HTPbablmUiSzPd1CIP3c2D4nSjXK_l.png

Re: Multi-WAN Load Balancing

Hi Craig,


Yesterday one of my isp's had a mayor malfunction. Didn't notice a thing until i read it in the news the day after (affected 400.000 customers). After this i checked the logging in the router and indeed saw some notifications that one of my isp had failed and had recovered. Never noticed a thing big_smile

Just wanted to let you know this succes stroy wink


Adze.

61 (edited by SouthPawn 2010-04-08 17:44:06)

Re: Multi-WAN Load Balancing

rockpilp wrote:

Hi Craig,

As an early poster on the Dual-Wan thread, I wanted to congratulate you on the great improvements you've made in the past few weeks/months.

I do agree with some of the other commenters that the LuCi module could use some help text.

Thanks rockpilp, I do agree that there may be some confusion on what the DNS configuration file does.
Essentially its the file we're going to write to, and perhaps that should be stated.

One of the functions that the script does is gather the dns server settings from the different interfaces and creates netfilter rules to always send dns client queries to their corresponding WAN interface, and then inserts all those entries in to the resolv.conf.auto which should be linked within the /tmp folder as to not write to the physical file system.

Ideally this setting would not need to exist and all interfaces with dns settings would be written to the resolv.conf.auto, however at the moment the interface that is initialized last gets it's dns written and any interfaces before it are simply overwritten instead of being appended to.

Adze wrote:

Hi Craig,


Yesterday one of my isp's had a mayor malfunction. Didn't notice a thing until i read it in the news the day after (affected 400.000 customers). After this i checked the logging in the router and indeed saw some notifications that one of my isp had failed and had recovered. Never noticed a thing big_smile

Just wanted to let you know this succes stroy wink


Adze.

Awesomesauce!

Thanks for all the feed back! smile
-Craig

Re: Multi-WAN Load Balancing

Can someone maybe post a few files that are needed for this to run correctly? I believe just editing the luci module isn't enough and some configuration files need modifications too, am I correct?
I've been trying to get this thing to run for weeks but it just won't do anything.

Re: Multi-WAN Load Balancing

skerit wrote:

Can someone maybe post a few files that are needed for this to run correctly? I believe just editing the luci module isn't enough and some configuration files need modifications too, am I correct?
I've been trying to get this thing to run for weeks but it just won't do anything.

Hey Skerit,

Please provide a few more specifics about what the problem is, are you seeing anything in syslog? what platform, version? does /etc/config/multiwan exist?

Thanks Skerit,
-Craig

Re: Multi-WAN Load Balancing

skerit, you need this packages for multiwan: ip, iptables, iptables-utils, iptables-mod-conntrack, iptables-mod-conntrack-extra
then you need to break off a port for the additional wan. then you can configure everything from the web interface - Network > Interfaces Wan, Wan2.
Network > Multiwan
I have been using this for two weeks now and it's working properly for me. I've tried it on pppoe, static ip, dhcp wan interfaces and several kamikaze and backfire versions.
Please provide more details on your setup and i'll try to pitch in on getting your router to work. although i might have more time during the weekend but southpawn is here.

@southpawn - fyi this is working great for me. just trying to get equal cost multipath to work more efficiently. <greedy> smile

65 (edited by kubu 2010-04-21 15:20:58)

Re: Multi-WAN Load Balancing

Thanks for providing multiwan!

I installed multiwan_1.0k.ipk and luci-app-multiwan_1.0k.ipk from ftp://ftp.netlab7.com/ on an Asus wl-500gp with Backfire 10.03 brcm 47xx generic image. I have two wan interfaces (wan15cable and wan14dsl), each having a public IP address configured by dhcp. Multiwan finds the interfaces and configures them. Failover seems to work -- apart from DNS complaining:  nameserver 195.186.1.162 refused to do a recursive query.

However load balancing does not. Traffic is always forwarded through the second interface as configured in /etc/config/multiwan. If I change the order of the interfaces in that file - traffic is forwarded trough the other interface. I observe traffic by the max download speed of a couple of torrents and the traffic counter on the interfaces as shown by ifconfig.

Any ideas?

Thanks
Kubu

# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0      0.0.0.0         255.255.255.0   U     0      0        0 eth0.13
92.105.81.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0.14
91.138.20.0     0.0.0.0         255.255.252.0   U     0      0        0 eth0.15
0.0.0.0         91.138.20.1     0.0.0.0         UG    0      0        0 eth0.15
0.0.0.0         92.105.81.1     0.0.0.0         UG    0      0        0 eth0.14
# ip route show table 123
192.168.1.0/24 dev eth0.13  proto kernel  scope link  src 192.168.1.1
92.105.81.0/24 dev eth0.14  proto kernel  scope link  src 92.105.81.205
91.138.20.0/22 dev eth0.15  proto kernel  scope link  src 91.138.21.138
default  proto static
        nexthop via 92.105.81.1  dev eth0.14 weight 8
        nexthop via 91.138.20.1  dev eth0.15 weight 4
# ip route show table 10
192.168.1.0/24 dev eth0.13  proto kernel  scope link  src 192.168.1.1
92.105.81.0/24 dev eth0.14  proto kernel  scope link  src 92.105.81.205
91.138.20.0/22 dev eth0.15  proto kernel  scope link  src 91.138.21.138
default via 92.105.81.1 dev eth0.14  proto static  src 92.105.81.205
# ip route show table 20
192.168.1.0/24 dev eth0.13  proto kernel  scope link  src 192.168.1.1
92.105.81.0/24 dev eth0.14  proto kernel  scope link  src 92.105.81.205
91.138.20.0/22 dev eth0.15  proto kernel  scope link  src 91.138.21.138
default via 91.138.20.1 dev eth0.15  proto static  src 91.138.21.138

The configuration is:

# cat /etc/config/multiwan

config 'multiwan' 'config'
        option 'default_route' 'balancer'
        option 'resolv_conf' '/tmp/resolv.conf.auto'

config 'interface' 'wan14dsl'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'health_interval' '60'
        option 'icmp_hosts' 'gateway'
        option 'weight' '8'
        option 'failover_to' 'balancer'

config 'interface' 'wan15cable'
        option 'timeout' '3'
        option 'health_fail_retries' '3'
        option 'health_recovery_retries' '5'
        option 'health_interval' '60'
        option 'icmp_hosts' 'gateway'
        option 'weight' '4'
        option 'failover_to' 'balancer'

I reduced the health_interval. Failures of connections are very rare events. I do not fully understand the option failover_to.

To avoid problems with DNS I configured Google's public DNS server for the two interfaces:

# cat /etc/config/network

config 'interface' 'wan15cable'
        option 'ifname' 'eth0.15'
        option 'defaultroute' '0'
        option 'peerdns' '0'
        option 'dns' '8.8.8.8'
        option 'proto' 'dhcp'

config 'interface' 'wan14dsl'
        option 'ifname' 'eth0.14'
        option 'defaultroute' '0'
        option 'peerdns' '0'
        option 'dns' '8.8.4.4'
        option 'proto' 'dhcp'

Re: Multi-WAN Load Balancing

I think it would be best if you could, post the /etc/config/multiwan file to take a look at it, everything appears to be fine atleast on the routing table, but I'd also check that the other routing tables are in tact, 10 and 20.

67 (edited by malakudi 2010-04-21 15:00:08)

Re: Multi-WAN Load Balancing

I had the same problem as above. It seems I was missing packages iptables-mod-ipopt and kmod-ipt-ipopt (which provide -m mark and -j MARK options, among others)

Re: Multi-WAN Load Balancing

SOLVED! iptables-mod-ipopt and kmod-ipt-ipopt where missing. May the dependencies for the packet be fixed?

Thanks a lot!

Can someone explain me how failover_to affects failover?

Re: Multi-WAN Load Balancing

I'll update that today, but failover_to essentially specifies where the traffic should be forwarded for the wan in the event that the wan fails.

So if your default route or outbound rules are set to use the primary wan, and the primary wan has the secondary wan as it's failover_to setting it will forward traffic destined to the primary wan to the secondary.

Things that are set to use Load Balancer will failover automatically as wans are removed from the routing table during failover.

Re: Multi-WAN Load Balancing

I have another problem (which I am trying to fix but haven't found how to patch the multiwan script yet). One of my wan setup is a ppp interface. It seems the route "default dev ppp0  scope link" is not handled well from the multiwan script. The script adds this route to all routing tables (table 10, 20 and 123). If I remove this route by hand, then multiwan works fine. It adds a default route as "default via GATEWAYIP dev ppp0" for table 20 and a correct multilink route in table 123.

Re: Multi-WAN Load Balancing

malakudi wrote:

I have another problem (which I am trying to fix but haven't found how to patch the multiwan script yet). One of my wan setup is a ppp interface. It seems the route "default dev ppp0  scope link" is not handled well from the multiwan script. The script adds this route to all routing tables (table 10, 20 and 123). If I remove this route by hand, then multiwan works fine. It adds a default route as "default via GATEWAYIP dev ppp0" for table 20 and a correct multilink route in table 123.

OK, I fixed this by changing:
ip route | grep link | while read ROUTE
to
ip route | grep link | grep -Ev ^default | while read ROUTE

72 (edited by kubu 2010-04-21 20:38:03)

Re: Multi-WAN Load Balancing

SouthPawn wrote:

I'll update that today, but failover_to essentially specifies where the traffic should be forwarded for the wan in the event that the wan fails.

So if your default route or outbound rules are set to use the primary wan, and the primary wan has the secondary wan as it's failover_to setting it will forward traffic destined to the primary wan to the secondary.

Things that are set to use Load Balancer will failover automatically as wans are removed from the routing table during failover.

Thanks for your explanation.

Failover works well. However, I still seem to have problems properly resolving some hostnames after a failover (manual ifdown of one interface). dnsmasq complained about the DNS server not resolving queries... :-/ Brining both interfaces backup (manually by ifup) does change.

As far as I know, outbound queries to DNS servers shal be bound to the respective interface, corret?

Re: Multi-WAN Load Balancing

I can confirm that iptables-mod-ipopt and kmod-ipt-ipopt are needed for load balancing. That's what I was missing if I did not install qos-scripts.
The above workaround for ppp0 didn't work for me.:(
I have the dsl line still lagging behind the cable line.
For the DNS I configured mine with identical public DNS servers and I never had problems with it.
You got a great product here Craig!
It's a winner!
big_smile

74 (edited by SouthPawn 2010-04-23 19:00:28)

Re: Multi-WAN Load Balancing

Thanks for the kind words!, I added this to the dependency, as well as the recommendation for ppp connections from kubu.
I did also update the DNS queries, so it should go to the correct interface anything thrown at the listed dns servers.

Thanks again,
-Craig

Re: Multi-WAN Load Balancing

I hope the thread owner doesn't mind. I'm doing this for a friend.
Here's a 30 minute guide to setting up Multi-wan on a WRT54GL.

1. Install Backfire 10.03
- for first time install see Openwrt documentation
- for upgrades from Kamikaze use the .trx and install via the web interface
- for bricked <smirk> routers see tftp install of .bin file
http://wiki.openwrt.org/doc/howto/tftp
note: don't forget to login via telnet and change root passwd for ssh to work

2. Create extra Vlan for Wan2
I do this via /etc/config/newtork.
note: I removed port "0" from eth0_0 and gave it to eth0_2.
You can configure wan and wan2 proto as dhcp to strt and then use the web interface to configure the pppoe or static ip later.
Use the same DNS servers I'm using if you're having DNS problems. Some ISPs only allow DNS connections from their IP blocks.
 ---------------------------------------------------
root@culiat-wg:~# cat /etc/config/network

config 'switch' 'eth0'
        option 'enable' '1'

config 'switch_vlan' 'eth0_0'
        option 'device' 'eth0'
        option 'vlan' '0'
        option 'ports' '1 2 3 5'

config 'switch_vlan' 'eth0_1'
        option 'device' 'eth0'
        option 'vlan' '1'
        option 'ports' '4 5'

config 'switch_vlan' 'eth0_2'
        option 'device' 'eth0'
        option 'vlan' '2'
        option 'ports' '0 5'

config 'interface' 'loopback'
        option 'ifname' 'lo'
        option 'proto' 'static'
        option 'ipaddr' '127.0.0.1'
        option 'netmask' '255.0.0.0'

config 'interface' 'lan'
        option 'type' 'bridge'
        option 'ifname' 'eth0.0'
        option 'proto' 'static'
        option 'stp' '1'
        option 'ipaddr' '192.168.1.1'
        option 'netmask' '255.255.255.0'

config 'interface' 'wan'
        option 'ifname' 'eth0.1'
        option 'proto' 'dhcp'
        option 'dns' '216.146.35.113 216.146.36.113 8.8.8.8 8.8.4.4'
        option 'defaultroute' '0'
        option 'peerdns' '0'

config 'interface' 'wan2'
        option 'ifname' 'eth0.2'
        option 'dns' '216.146.35.113 216.146.36.113 8.8.8.8 8.8.4.4'
        option 'proto' 'dhcp'
        option 'defaultroute' '0'
        option 'peerdns' '0'

3. Install prerequisite software
ip, iptables, iptables-utils, iptables-mod-conntrack, iptables-mod-conntrack-extra, iptables-mod-ipopt and kmod-ipt-ipopt
this is how I do it:
a. go to backfire packages: http://downloads.openwrt.org/backfire/10.03/brcm47xx/packages/
b. right click a link to a package and "Copy Link Location"
c. go to /tmp directory and run 
"opkg install http://downloads.openwrt.org/backfire/10.03/brcm47xx/packages/ip_2.6.29-1-2_brcm47xx.ipk"
note: if you're new to all of this it's better to install all the applications first including multiwan before configuring vlans so your internet connection does not go kookie on you. If you did because you're folowing this guide, then just shutdown a interface like so: "ifconfig eth0.2 down"

4. Install multiwan and it's web control package (luci-app)
Reboot to refresh the web ui. I always need to do this otherwise the link does not show up in networking.

5. Configure Wans && configure multiwan
Wans:
Network > Interfaces > Wan/WAN2
note: when asked which firewall zone to add wan2 choose wan so it has the same firewall rules for wan connections. Otherwise you'll have to manually recreate the fw rules for wan2.

Multiwan:
Network > Multiwan
checkout the bottom page to see samples of the settings.
here's how i got mine setup:
a. I only have two internet connections so I always remove the last two wan interfaces. I also comment out MWAN3 and MWAN4 in /etc/iproute2/rt_tables (although it may not be necessary).
b. Load Balancer Distribution = 1 for even connection distribution
note: You'll get per connection distribution not per packet so don't expect one download to come from both gateways. Lots of talk on this in the internet.
Failover = LoadBalancer for both links
c.Traffic Rules
note: checkout the examples
Source, Destination, protocol, Ports, WAN Uplink
all, all,all,all, Load Balancer
all, all, UDP, all, wan <-- this is so all vpn and voip connection goes through 1 gateway only
that's it! 

6. Test.
- Status > Interfaces should show traffic going through both interfaces.
- route distribution
root@culiat-wg:~# ip route show table 123
192.168.2.0/24 dev eth0.2  proto kernel  scope link  src 192.168.2.214
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1
114.108.201.0/24 dev eth0.1  proto kernel  scope link  src 114.108.201.49
default  proto static
        nexthop via 114.108.201.1  dev eth0.1 weight 1
        nexthop via 192.168.2.1  dev eth0.2 weight 1
- "route" should give you two default gateways
- try a torrent with lots of seeders. If you have a internet line that can do 90kbps max download, and another that can do 180kbps max if multiwan is working properly you should get a download rate greater than the higher rated link.
- pulling the plug from a wan port should still give you internet connection

7. Troubleshooting
There's a problem if 
- when you refresh the Interface status page and the Transfer rate of one interface does not change
- when you you go the the Interface Status page you only see one wan interface
- when you do "route" you only get one default gateway
- when you do "ip route show table 123" you don't get nexthops
- etc.
Fix:
- post your problem in the thread. :D

8. Extras
If you have two or more connections to the same ISP you should try:
 - ECMP using quagga http://quagga.net/faq/kodgehopper-ecmp.html <-- hard core network stuff (makes you feel like a genius. :D)
 - Channel bonding <-- this gives you per packet load distribution (effectively doubles your transfer rates)
 
That's it! Multiwan gives you protection from internet disconnection and doubles your transfer rate during heavy usage. 
Register for a free DDNS account and when you're in the office ask your maid to turn on the power on your routers and show off your shining new multiwan connection to your office mates. Open up several consoles and start downloading from several high bandwidth sites (kernel.org, nvidai, ati, etc.). show them how you're getting a combined download rate faster than the rated capacity of your individual internet subscriptions. Makes you top dog until the wife discovers you're paying for two internet lines. :D