OpenWrt Forum Archive

Topic: mwan3; multi-wan policy routing (general topic)

The content of this topic has been archived between 22 May 2013 and 6 May 2018. Unfortunately there are posts – most likely complete pages – missing.

Hi guys,

In succession of mwan2, i have made a new multi-wan script called mwan3. With the lessons i've learned making mwan2, i realized that the approach i took was not as efficient and configurable as it should be. Therefor i have created mwan3, which should resolve these issues. The reason i named it mwan3 and not a new version of mwan2 is because the config file changed quite a bit. If you were to upgrade this would break mwan2. I'd really like it if you could give it a go and let me know what you think of it.

edit: mwan svn repository is down. Please use git: https://github.com/Adze1502/mwan
edit2: With version 1.1, you can now ping multiple hosts per wan interface to check if it is still up. With the option reliability you set how many hosts should at least reply each round of testing, before triggering the interface as down. See the config example on how to configure it.
edit3: With the great help from Arfett, mwan3 now has a luci web interface. With this addition it is really easy to configure a multi-wan OpenWrt router. Thanks Arfett !!
edit4: GitHub repo is up: https://github.com/Adze1502/mwan . svn repo is down. Please update your "feeds.conf" file with the line "src-git mwan git://github.com/Adze1502/mwan.git" and remove the old svn repo.
edit5: New 1.4 version release which which should solve all load-balancing issues in previous release, like problem with wan connections not comming back online correctly.

The mwan3 package and the mwan3 luci app:
http://213.136.13.52/mwan3_latest_all.ipk
http://213.136.13.52/luci-app-mwan3_latest_all.ipk

And the source:
git://github.com/Adze1502/mwan.git

What is mwan3:
Mwan3 is a couple of lines of code that simplifies the usage of more (up to 250) WAN interfaces in OpenWRT. It is hotplug driven and it allows for any combination of primary, secondary or more failover interfaces, load balanced or not, for any combination of traffic. Mwan3 makes policy routing with multiple wan's easy. Mwan3 can monitor the state of interfaces by sending pings to a configured tracking host and failover if necessary.

Why should i use mwan3?:
- If you have multiple internet connections, you want to control which traffic goes through which wan's.
- Mwan3 can handle multiple levels of primary and backup interfaces, load-balanced or not. Different sources can have different primary or backup wan's.
- Mwan3 uses flowmask to be compatible with other packages (such as OpenVPN, PPTP VPN, QoS-script, Tunnels, etc) as you can configure traffic to use the default routing table.
- Mwan3 can also load-balance traffic originating from the router itself.

Requirements:
Mwan3 is successfully tested on OpenWRT trunk r40512. You need the following packages (which should be installed automatically if missing): ip, iptables, iptables-mod-conntrack, iptables-mod-conntrack-extra, iptables-mod-ipopt. Mwan3 is limited to max 250 wan interfaces.

How does it work:
Mwan3 is triggered by hotplug-events. When an interface comes up it creates a new routing table and new iptables rules. A new routing table is created for each interface. It then sets up iptables rules and uses iptables MARK to mark certain traffic. Based on ip rules the kernel determines which routing table to use. When an interface goes down, mwan3 deletes all the rules and routes to that interface in all created routing tables.

Mwan3 is not a daemon that runs in the background. Ones all the routes and rules are in place, it exits. The kernel takes care of all the routing decisions. If you want to apply a change you have made to mwan3 configuration, you have to trigger a hotplug event (replace eth1 and wan with your values):

ACTION=ifup DEVICE=eth1 INTERFACE=wan /sbin/hotplug-call iface

or you could bring up the interface using a mwan3 command:

mwan3 ifup wan

How to install and configure:
Please check the wiki http://wiki.openwrt.org/doc/howto/mwan3 for the more info.

I'll assume here you have a clean install of OpenWRT. Before installing mwan3, you need to make sure that all of your wan interfaces are correctly configured and work. Place a different metric on each WAN interface! This metric has only effect on the default routing table, not on the mwan3 routing tables. If it is configured correctly you should have a default gateway with a different metric set for each WAN interface. Something will look like this:

root@OpenWrt:~# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         213.154.232.9   0.0.0.0         UG    10     0        0 eth1
0.0.0.0         192.168.33.1    0.0.0.0         UG    20     0        0 eth0.2
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan
192.168.33.0    0.0.0.0         255.255.255.0   U     20     0        0 eth0.2
213.154.232.8   0.0.0.0         255.255.255.248 U     10     0        0 eth1

Check if above configuration works by trying to ping www.google.com form each interface:

root@OpenWrt:~# ping -c 1 -I eth1 www.google.com
PING www.google.com (74.125.136.106): 56 data bytes
64 bytes from 74.125.136.106: seq=0 ttl=50 time=23.012 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 23.012/23.012/23.012 ms
root@OpenWrt:~# ping -c 1 -I eth0.2 www.google.com
PING www.google.com (74.125.136.104): 56 data bytes
64 bytes from 74.125.136.104: seq=0 ttl=47 time=17.562 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 17.562/17.562/17.562 ms

If above ping test are successful, you can then continue installing mwan3. If you cannot ping www.google.com from all your WAN interfaces, there is a problem with your config and installing mwan3 won't fix that!

After installing mwan3, configure it by editting /etc/config/mwan3. For each WAN interface create an interface with a name that matches the one used in /etc/config/network. The status of each interface can be monitored with the track_ip option. Create at least two members. A member is a virtual representation of a wan interface. Each member can have different metric and or weight values. Members are used to make up a routing policy. Create a routing policy of at least one member. Members within one policy with a lower metric have precedence over higher metric members. Members with the same metric will load-balance. Load balancing members (with same metric) will distribute load based on those weights values.

After that, configure the rules. With the rules you can direct types of traffic based on ip, protocol or ports to certain policies, or use the default routing table by choosing "default". The order of the rules is very important, as they are loaded in that order. If a rule is matched, but all listed WAN interfaces/members are down, traffic is blackholed.


Example configuration

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan_wan2'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_wan'
    list use_member 'wan_m2_w3'
    list use_member 'wan2_m1_w2'

config rule 'example_1'
    option dest_ip '213.136.223.128/25'
    option dest_port '80'
    option proto 'tcp'
    option use_policy 'wan_wan2'

config rule 'example_2'
    option src_ip '1.2.3.4'
    option dest_ip '5.6.7.8'
    option src_port '12345:54321'
    option dest_port '12345:54321'
    option proto 'udp'
    option use_policy 'wan2_wan'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

The wiki page http://wiki.openwrt.org/doc/howto/mwan3 has more more info on this.

Now you're almost good to go. Last thing you should do is make sure the your firewall configuration allows traffic from lan to wan's and make sure the default OUTPUT policy is set to ACCEPT!

root@OpenWrt:~# cat /etc/config/firewall 

config defaults
    option syn_flood '1'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option drop_invalid '1'

config zone
    option name 'lan'
    list network 'lan'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option conntrack '1'
    option forward 'REJECT'

config zone
    option name 'wan'
    list network 'wan'
    list network 'wan2'
    option input 'REJECT'
    option output 'ACCEPT'
    option forward 'REJECT'
    option masq '1'
    option conntrack '1'

config forwarding
    option src 'lan'
    option dest 'wan'

Reboot your router and try if everything works. If not please feel free to ask your questions in this thread.

Status of load-balancing:
The command "mwan3 status" shows you the current state. It lists the state of each configured wan and the current output strategy for each policy. It also shows you a list of network which are "local" and not balanced. The last table lists the mwan3 rules currently active and it's policy.

root@OpenWrt:~# mwan3 status
Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)

Policy balanced:
 wan2 (40%)
 wan (60%)

Policy wan1_only:
 wan (100%)

Policy wan2_only:
 wan2 (100%)

Policy wan2_wan:
 wan2 (100%)

Policy wan_wan2:
 wan (100%)

Local connected networks:
destination        policy             hits     
------------------------------------------------
127.0.0.0/8        default            92       
224.0.0.0/3        default            4        
192.168.1.0/24     default            0        
192.168.33.0/24    default            4        
213.154.232.8/29   default            4        

Active rules:
source             destination        proto  src-port      dest-port     policy          hits     
---------------------------------------------------------------------------------------------------
0.0.0.0/0          213.136.223.128/25 tcp    0:65535       80            wan_wan2        0        
1.2.3.4            5.6.7.8            udp    12345:54321   12345:54321   wan2_wan        0        
0.0.0.0/0          0.0.0.0/0          all                                balanced        20533

Troubleshooting (if necessary):

root@OpenWrt:~# logread | grep mwan3
Tue Apr 15 14:47:33 2014 user.notice mwan3: ifup interface wan (eth1)
Tue Apr 15 14:47:47 2014 user.notice mwan3: ifup interface wan2 (eth0.2)

root@OpenWrt:~# iptables -L -t mangle -v -n -w
Chain PREROUTING (policy ACCEPT 84533 packets, 50M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 101K   54M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 21796 packets, 1953K bytes)
 pkts bytes target     prot opt in     out     source               destination         
31676 3119K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_connected (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  102  7198 MARK       all  --  *      *       0.0.0.0/0            127.0.0.0/8          mark match 0x0/0xff00 MARK or 0xff00
    4   292 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3          mark match 0x0/0xff00 MARK or 0xff00
    0     0 MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24       mark match 0x0/0xff00 MARK or 0xff00
    6   725 MARK       all  --  *      *       0.0.0.0/0            192.168.33.0/24      mark match 0x0/0xff00 MARK or 0xff00
    9   604 MARK       all  --  *      *       0.0.0.0/0            213.154.232.8/29     mark match 0x0/0xff00 MARK or 0xff00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 133K   57M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xff00
31717 2695K mwan3_ifaces  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
29628 2449K mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
28819 2388K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
 133K   57M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff00

Chain mwan3_iface_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  211  7624 MARK       all  --  *      *       213.154.232.8/29     0.0.0.0/0            mark match 0x0/0xff00 /* wan */ MARK or 0xff00
  148  9892 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan */ MARK xset 0x100/0xff00

Chain mwan3_iface_wan2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1066  157K MARK       all  --  *      *       192.168.33.0/24      0.0.0.0/0            mark match 0x0/0xff00 /* wan2 */ MARK or 0xff00
  147  6006 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan2 */ MARK xset 0x200/0xff00

Chain mwan3_ifaces (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  359 17516 mwan3_iface_wan  all  --  eth1   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
 1213  163K mwan3_iface_wan2  all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00

Chain mwan3_policy_balanced (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 8663  719K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 statistic mode random probability 0.39999999991 /* wan2 2 5 */ MARK xset 0x200/0xff00
12919 1073K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_policy_wan1_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_policy_wan2_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan2 2 2 */ MARK xset 0x200/0xff00

Chain mwan3_policy_wan2_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan2 2 2 */ MARK xset 0x200/0xff00

Chain mwan3_policy_wan_wan2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 mwan3_policy_wan_wan2  tcp  --  *      *       0.0.0.0/0            213.136.223.128/25   multiport sports 0:65535 multiport dports 80 mark match 0x0/0xff00 /* example_1 */
    0     0 mwan3_policy_wan2_wan  udp  --  *      *       1.2.3.4              5.6.7.8              multiport sports 12345:54321 multiport dports 12345:54321 mark match 0x0/0xff00 /* example_2 */
21582 1792K mwan3_policy_balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* default_rule */

root@OpenWrt:~# ip rule
0:    from all lookup local 
1001:    from all iif eth1 lookup main 
1002:    from all iif eth0.2 lookup main 
2001:    from all fwmark 0x100/0xff00 lookup 1 
2002:    from all fwmark 0x200/0xff00 lookup 2 
2254:    from all fwmark 0xfe00/0xff00 unreachable
32766:    from all lookup main 
32767:    from all lookup default 

root@OpenWrt:~# ip route
default via 213.154.232.9 dev eth1  proto static  metric 10 
default via 192.168.33.1 dev eth0.2  proto static  metric 20 
192.168.1.0/24 dev br-lan  proto kernel  scope link  src 192.168.1.1 
192.168.33.0/24 dev eth0.2  proto static  scope link  metric 20 
213.154.232.8/29 dev eth1  proto static  scope link  metric 10 

root@OpenWrt:~# ip route list table 1
default via 213.154.232.9 dev eth1 

root@OpenWrt:~# ip route list table 2
default via 192.168.33.1 dev eth0.2

(Last edited by Adze on 30 Mar 2015, 20:11)

Hi Adze,

I just tested you multiwan package and it works well in my case with 2x dsl lines.
Is there a public repository (github, svn...) where I could access your source code ?

--> got it smile

I would be interested to adapt the luci-multiwan app to your script.

Bye Bluse

(Last edited by Bluse-Blue on 8 Sep 2012, 18:18)

The link to the source code is in the start post.

mwan svn repository is up: svn://213.136.13.52/var/svn/mwan, it is now much easier to add mwan to the openwrt trunk. Just add the line "src-svn mwan svn://213.136.13.52/var/svn/mwan" to the file "feeds.conf.default".

mwan3 works fine in failover or load balancing configurations for me. I have PMed you about your package.

I'm trying to failover between l2tp->wan->wan2

mwan3 never seems to detect the l2tp link as being up

/etc/config/network:

config interface 'l2tp0'
        option _orig_ifname 'ppp0'
        option _orig_bridge 'false'
        option ifname 'ppp0'
        option proto 'none'

/etc/config/mwan3

config interface 'wan'
    option enabled '1'
    option track_ip '8.8.8.8'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'

config interface 'wan2'
    option enabled '1'
    option track_ip '8.8.4.4'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'

config interface 'l2tp0'
    option enabled '1'
    option track_ip '8.8.8.8'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'

config interface 'l2tp1'
    option enabled '1'
    option track_ip '8.8.8.8'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '5'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan_m3_w3'
    option interface 'wan'
    option metric '3'
    option weight '3'

config member 'wan_m4_w3'
    option interface 'wan'
    option metric '4'
    option weight '3'

config member 'wan2_m1_w3'
    option interface 'wan2'
    option metric '1'
    option weight '3'

config member 'wan2_m2_w3'
    option interface 'wan2'
    option metric '2'
    option weight '3'

config member 'wan2_m3_w3'
    option interface 'wan2'
    option metric '3'
    option weight '3'

config member 'wan2_m4_w3'
    option interface 'wan2'
    option metric '4'
    option weight '3'

config member 'l2tp0_m1_w3'
    option interface 'l2tp0'
    option metric '1'
    option weight '3'

config member 'l2tp0_m2_w3'
    option interface 'l2tp0'
    option metric '2'
    option weight '3'

config member 'l2tp1_m1_w3'
    option interface 'l2tp1'
    option metric '1'
    option weight '3'

config member 'l2tp1_m2_w3'
    option interface 'l2tp1'
    option metric '2'
    option weight '3'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w3'

config policy 'fail_wan_wan2'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m2_w3'

config policy 'fail_wan2_wan'
    list use_member 'wan_m2_w3'
    list use_member 'wan2_m1_w3'

config policy 'fail_l2tp_wan'
    list use_member 'l2tp0_m1_w3'
    list use_member 'l2tp1_m2_w3'
    list use_member 'wan_m3_w3'

config policy 'fail_l2tp_wan2'
    list use_member 'l2tp0_m1_w3'
    list use_member 'l2tp1_m2_w3'
    list use_member 'wan2_m3_w3'

config policy 'fail_l2tp_wan_wan2'
    list use_member 'l2tp0_m1_w3'
    list use_member 'l2tp1_m2_w3'
    list use_member 'wan_m3_w3'
    list use_member 'wan2_m4_w3'

config policy 'fail_l2tp_wan2_wan'
    list use_member 'l2tp0_m1_w3'
    list use_member 'l2tp1_m2_w3'
    list use_member 'wan2_m3_w3'
    list use_member 'wan_m4_w3'

config policy 'load_equal_wan_wan2'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w3'

config rule 'router_generated'
    option src_ip '127.0.0.0/8'
    option proto 'all'
    option use_policy 'fail_wan_wan2'

config rule 'lan'
    option src_ip '192.168.1.0/24'
    option proto 'all'
    option use_policy 'fail_l2tp_wan'

config rule 'lan2'
    option src_ip '192.168.2.0/24'
    option proto 'all'
    option use_policy 'fail_l2tp_wan'

config rule 'lan3'
    option src_ip '192.168.3.0/24'
    option proto 'all'
    option use_policy 'fail_l2tp_wan'

config rule 'lan4'
    option src_ip '192.168.4.0/24'
    option proto 'all'
    option use_policy 'fail_l2tp_wan_wan2'

root@OpenWrt:~# route -n

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.80.1    0.0.0.0         UG    10     0        0 eth1
0.0.0.0         192.168.1.1     0.0.0.0         UG    20     0        0 eth0.1
0.0.0.0         1.1.1.1         0.0.0.0         UG    100    0        0 ppp0
1.1.1.1         0.0.0.0         255.255.255.255 UH    0      0        0 ppp0
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0.1
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan2
192.168.3.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan3
192.168.4.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan4
192.168.80.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1

root@OpenWrt:~# ifconfig ppp0

ppp0      Link encap:Point-to-Point Protocol  
          inet addr:1.1.1.11  P-t-P:1.1.1.1  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1460  Metric:1
          RX packets:138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3 
          RX bytes:11370 (11.1 KiB)  TX bytes:11454 (11.1 KiB)

root@OpenWrt:/# ip rule

0:      from all lookup local 
200:    from all fwmark 0xc8 lookup TUN 
400:    from 192.168.2.0/24 lookup TUN 
400:    from 192.168.3.0/24 lookup TUN 
400:    from 192.168.4.0/24 lookup TUN 
1001:   from 192.168.80.112 fwmark 0x0/0x8000 lookup 1001 
1002:   from 192.168.1.118 fwmark 0x0/0x8000 lookup 1002 
1003:   from 1.1.1.11 fwmark 0x0/0x8000 lookup 1003 
1008:   from all fwmark 0x100/0xff00 lookup 1001 
1009:   from all fwmark 0x200/0xff00 lookup 1002 
1010:   from all fwmark 0x300/0xff00 lookup 1003 
1017:   from all fwmark 0x1100/0xff00 lookup 1017 
1018:   from all fwmark 0x1200/0xff00 lookup 1018 
1019:   from all fwmark 0x1300/0xff00 lookup 1019 
1021:   from all fwmark 0x1500/0xff00 lookup 1021 
1022:   from all fwmark 0x1600/0xff00 lookup 1022 
1023:   from all fwmark 0x1700/0xff00 lookup 1023 
1024:   from all fwmark 0x1800/0xff00 lookup 1024 
32766:  from all lookup main 
32767:  from all lookup default

root@OpenWrt:~# ping -I ppp0 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=46 time=91.126 ms

root@OpenWrt:~# tcpdump -i ppp0 icmp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
18:04:35.235293 IP 1.1.1.11 > google-public-dns-a.google.com: ICMP echo request, id 9877, seq 0, length 64
18:04:35.328886 IP google-public-dns-a.google.com > 1.1.1.11: ICMP echo reply, id 9877, seq 0, length 64

root@OpenWrt:/# logread | grep mwan3

Sep 17 17:54:21 OpenWrt user.notice root: mwan3: Lost 5 ping(s) on interface l2tp0 (ppp0)
Sep 17 17:54:21 OpenWrt user.notice root: mwan3: ifup interface l2tp0 (ppp0)
Sep 17 17:54:56 OpenWrt user.notice root: mwan3: ifdown interface l2tp0 ()
Sep 17 17:54:57 OpenWrt user.notice root: mwan3: Interface l2tp0 (ppp0) is offline
Sep 17 17:54:59 OpenWrt user.notice root: mwan3: ifdown interface l2tp0 (ppp0)
Sep 17 17:55:02 OpenWrt user.notice root: mwan3: ifup interface l2tp0 (ppp0)
Sep 17 17:55:06 OpenWrt user.notice root: mwan3: ifdown interface l2tp0 ()
Sep 17 17:55:11 OpenWrt user.notice root: mwan3: ifup interface l2tp0 (ppp0)
Sep 17 17:55:39 OpenWrt user.notice root: mwan3: Lost 3 ping(s) on interface l2tp0 (ppp0)
Sep 17 17:59:46 OpenWrt user.notice root: mwan3: Lost 66 ping(s) on interface wan2 (eth0.1)
Sep 17 18:00:06 OpenWrt user.notice root: mwan3: Interface wan2 (eth0.1) is online
Sep 17 18:00:07 OpenWrt user.notice root: mwan3: ifup interface wan2 (eth0.1)

root@OpenWrt:~# iptables -L -t mangle -v -n

Chain PREROUTING (policy ACCEPT 1577 packets, 280K bytes)
 pkts bytes target     prot opt in     out     source               destination         
23693 4594K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 537 packets, 43056 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 1027 packets, 236K bytes)
 pkts bytes target     prot opt in     out     source               destination         
14857 3737K zone_wan2_MSSFIX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
14857 3737K zone_wan_MSSFIX  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 557 packets, 216K bytes)
 pkts bytes target     prot opt in     out     source               destination         
10369 2491K mwan3_pre  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 1584 packets, 452K bytes)
 pkts bytes target     prot opt in     out     source               destination         
24619 6191K mwan3_post  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain mwan3_default (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   95 25788 MARK       all  --  *      *       0.0.0.0/0            224.0.0.0/3         mark match !0x8000/0x8000 MARK or 0x8000 
    8   672 MARK       all  --  *      *       0.0.0.0/0            1.1.1.1             mark match !0x8000/0x8000 MARK or 0x8000 
 2431  311K MARK       all  --  *      *       0.0.0.0/0            192.168.1.0/24      mark match !0x8000/0x8000 MARK or 0x8000 
    0     0 MARK       all  --  *      *       0.0.0.0/0            192.168.2.0/24      mark match !0x8000/0x8000 MARK or 0x8000 
    0     0 MARK       all  --  *      *       0.0.0.0/0            192.168.3.0/24      mark match !0x8000/0x8000 MARK or 0x8000 
 7406 1520K MARK       all  --  *      *       0.0.0.0/0            192.168.4.0/24      mark match !0x8000/0x8000 MARK or 0x8000 
 1272  195K MARK       all  --  *      *       0.0.0.0/0            192.168.80.0/24     mark match !0x8000/0x8000 MARK or 0x8000 

Chain mwan3_post (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  327 27468 MARK       all  --  *      eth0.1  0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x200/0xff00 
  695  175K MARK       all  --  *      ppp0    0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x300/0xff00 
 6930 1373K MARK       all  --  *      eth1    0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 MARK xset 0x100/0xff00 
14715 4190K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x8000/0x8000 MARK and 0xffff7fff 
24619 6191K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK save mask 0xff00 

Chain mwan3_pre (2 references)
 pkts bytes target     prot opt in     out     source               destination         
34062 7085K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore mask 0xff00 
 2953  960K MARK       all  --  eth0.1 *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8200/0xff00 
  364 30576 MARK       all  --  ppp0   *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8300/0xff00 
 5898 1484K MARK       all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x8100/0xff00 
24841 4610K mwan3_default  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match !0x8000/0x8000 
 2820  222K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff00 

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  352 38316 MARK       all  --  *      *       127.0.0.0/8          0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1200/0xff00 
  325 27300 MARK       all  --  *      *       192.168.1.0/24       0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1400/0xff00 
    0     0 MARK       all  --  *      *       192.168.2.0/24       0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1400/0xff00 
    0     0 MARK       all  --  *      *       192.168.3.0/24       0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1400/0xff00 
  968 58192 MARK       all  --  *      *       192.168.4.0/24       0.0.0.0/0           mark match 0x0/0xff00 MARK xset 0x1400/0xff00 

Chain qos_Default (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK restore mask 0xff 
    0     0 qos_Default_ct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff 
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x1/0xff length 400:65535 MARK and 0xffffff00 
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x2/0xff length 800:65535 MARK and 0xffffff00 
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff length 0:500 MARK xset 0x2/0xff 
    0     0 MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0           MARK xset 0x1/0xff 
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff tcp spts:1024:65535 dpts:1024:65535 MARK xset 0x4/0xff 
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff udp spts:1024:65535 dpts:1024:65535 MARK xset 0x4/0xff 
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           length 0:128 mark match !0x4/0xff tcp flags:0x3F/0x02 MARK xset 0x1/0xff 
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           length 0:128 mark match !0x4/0xff tcp flags:0x3F/0x10 MARK xset 0x1/0xff 

Chain qos_Default_ct (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff tcp multiport ports 22,53 MARK xset 0x1/0xff 
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff udp multiport ports 22,53 MARK xset 0x1/0xff 
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff tcp multiport ports 20,21,25,80,110,443,993,995 MARK xset 0x3/0xff 
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff tcp multiport ports 5190 MARK xset 0x2/0xff 
    0     0 MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0           mark match 0x0/0xff udp multiport ports 5190 MARK xset 0x2/0xff 
    0     0 CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK save mask 0xff 

Chain zone_wan2_MSSFIX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  207 12420 TCPMSS     tcp  --  *      eth0.1  0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU 

Chain zone_wan_MSSFIX (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   68  4080 TCPMSS     tcp  --  *      eth1    0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU 
    0     0 TCPMSS     tcp  --  *      ppp0    0.0.0.0/0            0.0.0.0/0           tcp flags:0x06/0x02 TCPMSS clamp to PMTU

With my rule for lan4 which should direct traffic over the l2tp tunnel it doesn't. Clients going to whatismyip.com shows the wan external IP address rather than the external address of our datacenter..

Any thoughts?

(Last edited by arfett on 17 Sep 2012, 19:28)

Maybe you could try removing these lines? Sounds silly, i know, but have seen some situations where it did matter.

        option _orig_ifname 'ppp0'
        option _orig_bridge 'false'

Or maybe you could go for the complete openwrt l2tp configuration: http://wiki.openwrt.org/doc/uci/network … ire.tunnel . This has worked for me.

(Last edited by Adze on 16 Sep 2012, 22:16)

Adze wrote:

Maybe you could try removing these lines? Sounds silly, i know, but have seen some situations where it did matter.

        option _orig_ifname 'ppp0'
        option _orig_bridge 'false'

Or maybe you could go for the complete openwrt l2tp configuration: http://wiki.openwrt.org/doc/uci/network … ire.tunnel . This has worked for me.

Those are added automatically after configuring it sometimes and they are not currently in the config.

Also we are using openl2tp.

Are there supposed to be routes in these?

root@OpenWrt:~# ip route show table 1001
root@OpenWrt:~# ip route show table 1002
default via 192.168.1.1 dev eth0.1
root@OpenWrt:~# ip route show table 1003

Edit: I updated the original post with the output of "iptables -L -t mangle -v -n"
Interestingly enough it doesn't look like the rule for 127.0.0.0/8 traffic is working either as I can unplug my wan connection and mwan3 sends an ifdown but the router traffic does not failover to wan2. This was working on saturday with the same config file.

(Last edited by arfett on 17 Sep 2012, 19:31)

arfett wrote:

Interestingly enough it doesn't look like the rule for 127.0.0.0/8 traffic is working either as I can unplug my wan connection and mwan3 sends an ifdown but the router traffic does not failover to wan2. This was working on saturday with the same config file.

I think i've made a mistake somewhere with the last update... will get back asap.

Adze,  I gave your mwan3 a try based on r8 of your svn repo.  I observed that my wan2 (dhcp) got virtually all the traffic, over wan (static), despite my attempt to direct all but a specific UDP stream over wan:

config 'rule'
    option 'dest_ip' 'hostname.example.com'
    option 'proto' 'udp'
    option 'dest_port' '1194'
    option 'use_policy' 'wan2_only'   

config 'rule'
    option 'dest_ip' '0.0.0.0/0'
    option 'use_policy' 'wan_only'

(full mwan3 config here: http://www.pastebin.ca/2205638)

Also, I had trouble with some previously working (in single-wan) iptables DNAT rules, passing ssh through to internal hosts from static ips on the gateway.  The traffic arrived at wan, but then disappeared, never forwarded inside.  The DNAT rule was never triggerred (judging by the packet/byte counts).  When I turn off mwan3, my DNAT rules start working again.

Hi RussellSenior,


Your config looks OK. I'd like to get your setup working. Could you please paste me the outcome of:

- ip rule
- route -n
- iptables -L -t mangle -v -n
- cat /etc/config/network
- ip route list table 1001
- ip route list table 1002
- cat /etc/config/firewall


Thanks.

Adze, here is a slightly anonymized version.  Note that I am not seeing table 1001, but some others, which I've included.  Also, I am not using the OpenWrt firewall script, but have substituted my own init script (included):

  http://www.pastebin.ca/2205823

It looks like maybe the dhcp is stomping on the static default gateway route I've specified for the wan interface.

Oh, actually, I neglected to assign a metric to the wan interfaces, that explains the static iface's default gateway going away.

RussellSenior wrote:

Oh, actually, I neglected to assign a metric to the wan interfaces, that explains the static iface's default gateway going away.

And that explains why mwan3 isn't working... After you have changed this, is it working? If not could you paste the same info again after setting different metric to your wan's?

Thnx!

Now, it seems things are mostly working.  New wrinkle is olsrd derived routes are not being honored.  All the olsrd routes are going into 'table main'

RussellSenior wrote:

Now, it seems things are mostly working.  New wrinkle is olsrd derived routes are not being honored.  All the olsrd routes are going into 'table main'

You can work-around this by adding a rule to tell mwan3 for which destinations to look in the default routing table.

config rule
 option 'dest_ip' '10.0.0.0/8'
 option 'use_policy' 'default'
Adze wrote:
RussellSenior wrote:

Now, it seems things are mostly working.  New wrinkle is olsrd derived routes are not being honored.  All the olsrd routes are going into 'table main'

You can work-around this by adding a rule to tell mwan3 for which destinations to look in the default routing table.

config rule
 option 'dest_ip' '10.0.0.0/8'
 option 'use_policy' 'default'

That works, cool!

However, my rule:

config 'rule'
    option 'dest_ip' 'donk.personaltelco.net'
    option 'proto' 'udp'
    option 'dest_port' '1194'
    option 'use_policy' 'wan2_only'   

isn't working (packets go out eth1.1 (wan) instead of wan2, while this one does work:

config 'rule'
    option 'dest_ip' '4.2.2.2'
    option 'use_policy' 'wan2_only'

(Last edited by RussellSenior on 20 Sep 2012, 13:08)

Hostnames in rules work, but are kinda borky... When mwan3 fills the iptables rules, it resolves the hostname. If it has multiple A records, it will only use the first one. It will not change until an interface has changed its status again. My advice would be to use ip addresses only.

To verify that the rule is actually correct, please use the command "iptables -L mwan3_rules -t mangle -v -n" and see if it's there.

I wrote a Luci web interface, but I don't know how to upload

mexia wrote:

I wrote a Luci web interface, but I don't know how to upload

I have also created one shown here. It just needs the final part which is a status page showing real-time data on whether interfaces are up/down and also a troubleshooting page which spits out iptables information related to mwan3 (this part is done.)

https://forum.openwrt.org/viewtopic.php?pid=178312

Adze wrote:

Hostnames in rules work, but are kinda borky... When mwan3 fills the iptables rules, it resolves the hostname. If it has multiple A records, it will only use the first one. It will not change until an interface has changed its status again. My advice would be to use ip addresses only.

To verify that the rule is actually correct, please use the command "iptables -L mwan3_rules -t mangle -v -n" and see if it's there.

Yeah, it resolves the ipaddr numbers correctly.  Looks like it's an issue of where the data is originating.  That is, if I run a traceroute -U -p 1194 donk.personaltelco.net from a host on my LAN, it routes the right way.  If I do that from the router running mwan3, it doesn't route the right way.

RussellSenior wrote:

Yeah, it resolves the ipaddr numbers correctly.  Looks like it's an issue of where the data is originating.  That is, if I run a traceroute -U -p 1194 donk.personaltelco.net from a host on my LAN, it routes the right way.  If I do that from the router running mwan3, it doesn't route the right way.

I wasn't having much luck getting router generated traffic routing correctly so I made my own hotplug script which runs right after his that adds ip rules for loopback traffic based on ifup/ifdown hotplug events for wan interfaces.

ip rule add dev lo table 100x

If I could get it working via rules instead I would be ecstatic.

(Last edited by arfett on 21 Sep 2012, 00:36)

The way i got my router originated traffic to load-balance is by adding an ip alias on the loopback interface. Then create a default route with the ip alias as source in the main routing table with a lower metric than the lowest wan interface. You will have something like this:

root@mercurius:~# ip addr show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.1.1/32 brd 255.255.255.255 scope global lo   <-- This is the added ip alias.
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

root@mercurius:~# ip route
default via 192.168.1.1 dev lo  src 192.168.1.1  metric 5    <-- This is the added loopback route.
default via 95.97.227.169 dev eth0.1  proto static  metric 10 
default via 213.154.232.9 dev eth0.2  proto static  metric 20 
95.97.227.168/29 dev eth0.1  proto static  scope link  metric 10 
192.168.33.0/24 dev br-lan  proto kernel  scope link  src 192.168.33.2 
213.154.232.8/29 dev eth0.2  proto static  scope link  metric 20

And proof that it is load-balancing:

root@mercurius:~# traceroute -4 -l -n -q 1 -w 1 213.136.13.52
traceroute to 213.136.13.52 (213.136.13.52), 30 hops max, 38 byte packets
 1  213.154.232.9  0.722 ms (64)
 2  213.136.1.10  16.243 ms (254)
 3  212.142.61.37  15.958 ms (253)
 4  84.116.244.21  11.587 ms (252)
 5  84.116.135.182  12.166 ms (251)
 6  195.69.144.35  16.226 ms (249)
 7  213.154.229.54  32.742 ms (249)
 8  213.136.13.52  12.850 ms (57)

root@mercurius:~# traceroute -4 -l -n -q 1 -w 1 213.136.13.52
traceroute to 213.136.13.52 (213.136.13.52), 30 hops max, 38 byte packets
 1  95.97.227.169  0.395 ms (64)
 2  213.136.1.10  17.067 ms (254)
 3  213.136.1.233  17.746 ms (252)
 4  84.116.244.21  10.208 ms (252)
 5  84.116.132.218  12.596 ms (251)
 6  213.136.13.52  18.374 ms (60)

root@mercurius:~# traceroute -4 -l -n -q 1 -w 1 213.136.13.52
traceroute to 213.136.13.52 (213.136.13.52), 30 hops max, 38 byte packets
 1  213.154.232.9  0.642 ms (64)
 2  213.136.1.10  16.267 ms (254)
 3  213.136.1.233  17.590 ms (252)
 4  84.116.244.21  11.194 ms (252)
 5  213.136.13.52  17.959 ms (60)

root@mercurius:~# traceroute -4 -l -n -q 1 -w 1 213.136.13.52
traceroute to 213.136.13.52 (213.136.13.52), 30 hops max, 38 byte packets
 1  95.97.227.169  0.431 ms (64)
 2  10.15.180.129  7.107 ms (254)
 3  213.136.1.233  17.650 ms (252)
 4  213.154.229.54  17.460 ms (252)
 5  84.116.135.202  13.082 ms (251)
 6  195.69.144.35  17.767 ms (249)
 7  213.154.229.54  16.024 ms (249)
 8  213.136.13.52  17.117 ms (60)
Adze wrote:

The way i got my router originated traffic to load-balance is by adding an ip alias on the loopback interface. Then create a default route with the ip alias as source in the main routing table with a lower metric than the lowest wan interface. You will have something like this:

I think where I messed up was not using a lower metric. I will try again and see what happens. And just to clarify again you did not need to create a rule for this traffic?

arfett wrote:

And just to clarify again you did not need to create a rule for this traffic?

You must create a rule for this traffic. In above example i have only one default load-balancing rule with equalize active. The rules both apply to hosts on the lan and to the router itself.

config 'interface' 'wan1'
    option 'enabled' '1'

config 'interface' 'wan2'
    option 'enabled' '1'

config 'member' 'wan1_m1_w3'
    option 'interface' 'wan1'
    option 'metric' '1'
    option 'weight' '3'

config 'member' 'wan2_m1_w2'
    option 'interface' 'wan2'
    option 'metric' '1'
    option 'weight' '2'

config 'policy' 'wan1_wan2_loadbalanced'
    list 'use_member' 'wan1_m1_w3'
    list 'use_member' 'wan2_m1_w2'

config 'rule'
    option 'dest_ip' '0.0.0.0/0'
    option 'equalize' '1'
    option 'use_policy' 'wan1_wan2_loadbalanced'

(Last edited by Adze on 21 Sep 2012, 20:00)