OpenWrt Forum Archive

Topic: mwan3; multi-wan policy routing (general topic)

The content of this topic has been archived between 22 May 2013 and 6 May 2018. Unfortunately there are posts – most likely complete pages – missing.

jmarlin wrote:

Hey Adze, the new marking bits you noted work much better with nodogsplash and qos-scripts. Thanks so much! I've added both JohnV's findings and this latest issue to the wiki now.

Nice to hear and thnx for reporting back!

Hi Adze!
Thank you for your great job!
I am using your mwan3 package in my OpenWRT router, and it works very well.

But I have a problem:
In my country, people could not access google or youtube or facebook( many websites ) directly. We need proxy or VPN to access this websites.
And I am using dnsmasq + ipset + iptables mark + OpenVPN in the router to let the network packets to google.com through VPN automatically.
1. Create ipset named vpn.
2. Add google.com into conf file of dnsmasq, so when I resolve this domain name, dnsmasq will add the ip into ipset vpn.
3. In iptables, if the target IP is in ipset vpn, then mark with 1
4. All packet which mark with 1 goto route table vpn
5. Set the route table vpn default rule go through OpenVPN.

It works very well in my WNDR 4300, but when I install your mwan3 package and configure to 2 wan load balance, it dose not work anymore.

So, I think there are some conflicts between mwan3 and my settings. And I could not find the conflict...
I have to use 2 routers, one UBNT edgerouter lite using your mwan3 and WNDR4300 using my setting.

Is possible to let mwan3 work with my setting together?

ps. sorry for my poor english.

Hi fovecifer,

fovecifer wrote:

In my country, people could not access google or youtube or facebook( many websites ) directly. We need proxy or VPN to access this websites.

Booo to cencorship! Booo to the fascists in power, that think their moral compass is better then someone else, with their hunger for control!

What i would do to fix this, is to configure your OpenVPN tunnel also as a wan interface in mwan3. This way you will have three wan interfaces (2x real and 1x vpn). With the help of rules you can steer traffic out the right interface. So in summary:

1. Create 3 wan interfaces (2x real, 1x vpn)
2. Add google.com to ipset vpn into conf file of dnsmasq.
3. Create mwan3 rules with ipset vpn to traverse over wan vpn.

Done.

(Last edited by Adze on 20 Mar 2015, 08:22)

Thank you Adze!
I think your solution is perfect! I will change my setting when I back home.
Thank you again Adze! I am trying to fix this for several days.

Adze wrote:

Booo to censorship! Booo to the fascists in power, that think their moral compass is better then someone else, with their hunger for control!

I can't agree more!  I love google services I hate censorship!

Minor thing, and maybe I am missing something.

When I checkout for-14.07 branch from git the version of mwan is 1.5 r10.

When i checkout master from git the version of mwan is 1.5 r8.

Just a heads up to the maintainers as maybe master needs to be updated, or maybe the release difference is not important......

JohnV wrote:

Minor thing, and maybe I am missing something.

When I checkout for-14.07 branch from git the version of mwan is 1.5 r10.

When i checkout master from git the version of mwan is 1.5 r8.

Just a heads up to the maintainers as maybe master needs to be updated, or maybe the release difference is not important......

It's more like we are just forgetting to update the other repository since it's not really as relevant as the OpenWrt Packages feed.

In the "packages" git repo:

I assumed the "for-14.07" is the feed (source) for BB.  Or at least this is what one should use when building a custom BB image.

And "master" is the feed (source) for CC.  I assumed that master should be used when building a custom CC image.

I build all my own images, so I don't really know how the pre-built stuff lines up, but (perhaps naively) assumed there would be packages of mwan3, one for BB and one for CC (and probably one for AA).

Am I wrong? A bit off-topic and I apologize.

I just realized what you're talking about. The version in master is for trunk and supports features only available in trunk (for now.)

Master is 1.6-1
for-14.07 is 1.5-10

(Last edited by arfett on 27 Mar 2015, 00:20)

arfett wrote:

I just realized what you're talking about. The version in master is for trunk and supports features only available in trunk (for now.)

Master is 1.6-1
for-14.07 is 1.5-10

You are correct, my mistake in my working copy form git.  I apologize for wasting your time....

Still learning git.

Hello. Thank for mwan3.

Right now we have working Linux-server (external IPs 1.1.1.1 and 2.2.2.2) with OpenVPN server and remote office with TP-Link MR-3420 with OpenWRT, mwan3, 3Gmodem (dynamic external IP), USBFlash, USBhub, wan-Internet-provider (external IP 9.9.9.1 gw 9.9.9.9 )

OpenVPN server ccd for MR-3420 client:
iroute 192.168.70.0 255.255.255.0
push "redirect-gateway def1"

OpenVPN client on MR-3420:
remote 1.1.1.1 1194
remote 2.2.2.2 1194
persist-key
;persist-tun

OpenVPN is the default route on MR-3420:
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.8.0.5        128.0.0.0       UG    0      0        0 tun0
0.0.0.0         9.9.9.9  0.0.0.0         UG    1      0        0 pppoe-wan
0.0.0.0         10.64.64.64     0.0.0.0         UG    2      0        0 3g-wan2
10.8.0.0        10.8.0.5        255.255.255.0   UG    0      0        0 tun0
10.8.0.5        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan2
9.9.9.9  0.0.0.0         255.255.255.255 UH    0      0        0 pppoe-wan
1.1.1.1    9.9.9.9  255.255.255.255 UGH   0      0        0 pppoe-wan
128.0.0.0       10.8.0.5        128.0.0.0       UG    0      0        0 tun0
192.168.29.0     10.8.0.5        255.255.0.0     UG    0      0        0 tun0
192.168.70.0    0.0.0.0         255.255.255.0   U     0      0        0 br-lan

Can this work correct with mwan3 failover (wan 9.9.9.1 + wan2 3G)?
Will mwan3 tracking work with default gw set to OpenVPN?
How we can acces router (over 9.9.9.1) when default gw is OpenVPN (add ip rule add from 9.9.9.1 lookup 1)?
Can be tracking IPs the same for wan and wan2 for mwan3?

What else you can recomend to do with this configuration?

Thank you.

(Last edited by altuhovd on 27 Mar 2015, 10:58)

Hi altuhovd,


mwan3 is compatible with openvpn tunnels, but you have to use 0.0.0.0/0 routes only. In your current output tun0 has no default route. There is more info on this in this thread.

Adze wrote:

Hi altuhovd,
mwan3 is compatible with openvpn tunnels, but you have to use 0.0.0.0/0 routes only. In your current output tun0 has no default route. There is more info on this in this thread.

Thank you, I will read.

Adze wrote:

Hi altuhovd,


mwan3 is compatible with openvpn tunnels, but you have to use 0.0.0.0/0 routes only. In your current output tun0 has no default route. There is more info on this in this thread.

I found one problem:

1) I have wan = (wired) internet-provider with static IP 9.9.9.9 (gw 9.9.9.1) and wan2 = 3G-usb-modem with PPP
2) When wan is down (tracking down, not pinging) and wan2 is active I see this:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         9.9.9.9     0.0.0.0         UG    1      0        0 eth0
0.0.0.0         10.64.64.64     0.0.0.0         UG    2      0        0 3g-wan2
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan2
9.9.9.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.70.0    0.0.0.0         255.255.255.0   U     0      0        0 br-lan
3) All trafic is going to gw 10.64.64.64 (3G modem). Internet working

4) But when I do /etc/init.d/openvpn start and starting OpenVPN  tunnel with redirect-gateway def1 I see this:
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.8.0.117      128.0.0.0       UG    0      0        0 tun0
0.0.0.0         9.9.9.1     0.0.0.0         UG    1      0        0 eth0
0.0.0.0         10.64.64.64     0.0.0.0         UG    2      0        0 3g-wan2
10.8.0.0        10.8.0.117      255.255.255.0   UG    0      0        0 tun0
10.8.0.117      0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan2
9.9.9.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0
1.1.1.1    9.9.9.1     255.255.255.255 UGH   0      0        0 eth0
128.0.0.0       10.8.0.117      128.0.0.0       UG    0      0        0 tun0
192.168.0.0     10.8.0.117      255.255.0.0     UG    0      0        0 tun0
192.168.70.0    0.0.0.0         255.255.255.0   U     0      0        0 br-lan

After OpenVPN is started and connected to OpenVPN server (1.1.1.1) it change default gateway and add route to OpenVPN-server via prevous default gw, from OpenVPN man:
(1) Create a static route for the --remote address which forwards to the pre-existing default gateway. This is done so that (3) will not create a routing loop.

(2) Delete the default gateway route.

(3) Set the new default gateway to be the VPN endpoint address

What can I do when wan (gw 9.9.9.1) is not working and OpenVPN select this as prevous default gw?
Why OpenVPN-client may not select default route with metric 2 (currently active)?

Can mwan3 delete all default gateways (with metrics 1,2,3,...) and specify only one default gateway (with metric 0)?
Why we need many default gateways (with different metrics) instead of one default gw with metric 0? For pinging from wan and wan2 we can use ip rule add ... lookup ?

(Last edited by altuhovd on 27 Mar 2015, 13:20)

Hi altuhovd,


These openvpn routes result in that all traffic traverses tun0.

0.0.0.0         10.8.0.117      128.0.0.0       UG    0      0        0 tun0
128.0.0.0       10.8.0.117      128.0.0.0       UG    0      0        0 tun0

Do not use "redirect-gateway def1", as it creates above routes.

Can you describe what you want to achieve? Maybe i can help you with that.

(Last edited by Adze on 27 Mar 2015, 13:22)

Adze wrote:

Hi altuhovd,

These openvpn routes result in that all traffic traverses tun0.

0.0.0.0         10.8.0.117      128.0.0.0       UG    0      0        0 tun0
128.0.0.0       10.8.0.117      128.0.0.0       UG    0      0        0 tun0

Do not use "redirect-gateway def1", as it creates above routes.

Can you describe what you want to achieve? Maybe i can help you with that.

I want two internet channes (wan=static IP + wan2=3G-modem) + OpenVPN client + route all traffic to OpenVPN server + mwan3 (failover internet-channels) on TP-Link MR-3420.

Ok. I can temporary disable redirect-gateway def1.
Now testing without it.
Switch to backup channel working fine (openvpn reconnects), but switching back to primary not working for established connections (like openvpn).
After primary channel is up, openvpn is still going to backup channel. And all established connection (lice icq, https) too.

(Last edited by altuhovd on 27 Mar 2015, 14:11)

What i would do is setup mwan3 with 3 wan interfaces (2x real + 1x openvpn). Setup a rule for upd 1194 to use the real wans. Add a second rule that routes all traffic over OpenVPN tunnel. Done.

Adze wrote:

What i would do is setup mwan3 with 3 wan interfaces (2x real + 1x openvpn). Setup a rule for upd 1194 to use the real wans. Add a second rule that routes all traffic over OpenVPN tunnel. Done.

Thank you Adze. You helped so many peoples.

As I did not find such a simple solution, sorry. I will try it.

But why after switching from backup to primary channel some traffic (established) still going over backup channel (monitored by tcpdump)?
What manual should I read?

(Last edited by altuhovd on 27 Mar 2015, 18:27)

When a primary link comes back, all already established sessions will continue to traverse the backup wan. Only new sessions will route over the primary wan. This works as designed, but maybe not what you want..

You can however create a custom script which kills all active sessions on the backup link if the primary comes back to life. Create a script in /etc/hotplug.d/iface. Name it something like 16-mwan3-kill-sessions. With conntrack tools you can terminate all udp 1194 sessions.

(Last edited by Adze on 27 Mar 2015, 18:32)

First i need to work with OpenVPN and mwan3 is default route for tun0 (OpenVPN tunnel)

I fixed ccd-config on OpenVPN server:
#push "redirect-gateway def1"
push "route 0.0.0.0 0.0.0.0 vpn_gateway 3"

Now route table on router:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         9.9.9.1        0.0.0.0         UG    1      0        0 pppoe-wan
0.0.0.0         10.64.64.64     0.0.0.0         UG    2      0        0 3g-wan2
0.0.0.0         10.8.0.5        0.0.0.0         UG    3      0        0 tun0
10.8.0.0        10.8.0.5        255.255.255.0   UG    0      0        0 tun0
10.8.0.5        0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan2
9.9.9.1        0.0.0.0         255.255.255.255 UH    0      0        0 pppoe-wan
192.168.0.0     10.8.0.5        255.255.0.0     UG    0      0        0 tun0
192.168.66.0    0.0.0.0         255.255.255.0   U     0      0        0 br-lan

But will it work? Right now I see, that ovpn interface (OpenVPN tunnel) is up and tracking active, but in mwan3 LuCI configuration:
MWAN Interface Configuration - ovpn
WARNING: this interface has no metric configured in /etc/config/network!

Do I need specify metric for ovpn interface in /etc/config/network? Current protocol is "none" for ovpn.

(Last edited by altuhovd on 27 Mar 2015, 20:13)

Hi altuhovd,


Your routing table looks ok to me! The warning will go away when you configure the openvpn (tun0) interface in /etc/config/network.

config 'interface' 'openvpn'
        option 'proto' 'none'
        option 'ifname' 'tun0'
        option 'metric' '3'

Fixed. Current mwan3 status (except know networks):

Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)
Interface ovpn is online (tracking active)

Policy balanced:
 wan2 (40%)
 wan (60%)

Policy wan2_only:
 wan2 (100%)

Policy wan2_wan:
 wan2 (100%)

Policy wan_only:
 wan (100%)

Policy wan_wan2:
 wan (100%)

Active rules:
source             destination        proto  src-port      dest-port     policy          hits     
--------------------------------------------------------------------------------------------------
0.0.0.0/0          0.0.0.0/0          udp    0:65535       1194          wan_wan2        0        
0.0.0.0/0          0.0.0.0/0          all                                wan_wan2        14

"0.0.0.0/0          0.0.0.0/0          udp    0:65535       1194          wan_wan2        0" = zero (0) hits on OpenVPN tunnel it because it's already established, i'm think. Will try to restart. If hits will grow, then I will fix 0.0.0.0/0 rule (ovpn->wan-wan2 failover)

(sorry for bad English)

Hi Adze
I don't want to be pushy, but have had a chance to look at the quota option yet..
Thanks

Testing mwan3 over OpenVPN->WAN->WAN2

1) Right now all working fine from LAN at router (192.168.70.0/24).
Can ping any host.

2) But not working at router itself (can not ping any host, nslookup not working).
from router I can not ping 8.8.8.8, but can ping IPs in OpenVPN (LAN at office).

3) Should interface metrics at "MWAN Interface Configuration" page be the same as at "MWAN Member Configuration" page?
At the "MWAN Interface Configuration" page metrics is:
wan = 1
wan2 = 2
ovpn =3

At "MWAN Member Configuration" page metrics:
wan = 10
wan2 = 20
ovpn = 9

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         9.9.9.1     0.0.0.0         UG    1      0        0 eth0
0.0.0.0         10.64.64.64     0.0.0.0         UG    2      0        0 3g-wan2
0.0.0.0         10.8.0.117      0.0.0.0         UG    3      0        0 tun0
10.8.0.0        10.8.0.117      255.255.255.0   UG    0      0        0 tun0
10.8.0.117      0.0.0.0         255.255.255.255 UH    0      0        0 tun0
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan2
9.9.9.0     0.0.0.0         255.255.255.0   U     1      0        0 eth0
192.168.0.0     10.8.0.117      255.255.0.0     UG    0      0        0 tun0
192.168.70.0    0.0.0.0         255.255.255.0   U     0      0        0 br-lan

0:      from all lookup local
1001:   from all iif eth0 lookup main
1002:   from all iif 3g-wan2 lookup main
1003:   from all iif tun0 lookup main
2001:   from all fwmark 0x100/0xff00 lookup 1
2002:   from all fwmark 0x200/0xff00 lookup 2
2003:   from all fwmark 0x300/0xff00 lookup 3
2253:   from all fwmark 0xfd00/0xff00 blackhole
2254:   from all fwmark 0xfe00/0xff00 unreachable
32766:  from all lookup main
32767:  from all lookup default

Interface status:
Interface wan is online (tracking active)
Interface wan2 is online (tracking active)
Interface ovpn is online (tracking active)

Policy balanced:
 wan2 (40%)
 wan (60%)

Policy ovpn_wan2_wan:
 ovpn (100%)

Policy wan2_only:
 wan2 (100%)

Policy wan2_wan:
 wan2 (100%)

Policy wan_only:
 wan (100%)

Policy wan_wan2:
 wan (100%)

Active rules:
source             destination        proto  src-port      dest-port     policy          hits
--------------------------------------------------------------------------------------------------
0.0.0.0/0          0.0.0.0/0          udp    0:65535       1197          wan_wan2        0
0.0.0.0/0          0.0.0.0/0          all                                ovpn_wan2_wan   1139
altuhovd wrote:

2) But not working at router itself (can not ping any host, nslookup not working).
from router I can not ping 8.8.8.8, but can ping IPs in OpenVPN (LAN at office).

Probably because replies are not routed back over openvpn tunnel wink. If you ping from router, your router will create an ip packet based on main routing table. So if you ping 8.8.8.8, it will create an icmp packet with source 9.9.9.9 (ip address on interface eth0) and destination 8.8.8.8, which will be routed over OpenVPN tunnel. Please try "ping -I tun0 8.8.8.8" to confirm.

altuhovd wrote:

3) Should interface metrics at "MWAN Interface Configuration" page be the same as at "MWAN Member Configuration" page?
At the "MWAN Interface Configuration" page metrics is:
wan = 1
wan2 = 2
ovpn =3

At "MWAN Member Configuration" page metrics:
wan = 10
wan2 = 20
ovpn = 9

Whut ?? mwan3 interfaces configuration has no metric config options ?? If you mean general network config, then no, they serve a different purpose and don't have to be the same.

(Last edited by Adze on 30 Mar 2015, 09:23)

Adze wrote:

What i would do is setup mwan3 with 3 wan interfaces (2x real + 1x openvpn). Setup a rule for upd 1194 to use the real wans. Add a second rule that routes all traffic over OpenVPN tunnel. Done.

I'd like to replicate altuhovd's setup, but it doesn't seem to work for me. Here are excerpts from my config files:

/etc/config/network

config interface 'vpn1'
        option ifname 'tun1'
        option proto 'none'
        option metric '110'

/etc/config/mwan3

config interface 'vpn1'
        option enabled '1'
#       list track_ip '10.99.99.1'
        option reliability '1'
        option count '1'
        option timeout '4'
        option interval '5'
        option down '3'
        option up '8'

config member 'vpn1_m1_w2'
        option interface 'vpn1'
        option metric '1'
        option weight '2'

config policy 'vpn1_only'
        list use_member 'vpn1_m1_w2'

config rule 'vpn_endpoint1'
        option dest_ip 'remote_ovpn_server_ip'
        option dest_port '1194'
        option proto 'udp'
        option use_policy 'wan2_wan'

config rule 'vpn_pbr1'
        option dest_ip '172.21.22.0/255.255.255.0'
        option use_policy 'vpn1_only'

config rule 'vpn_pbr2'
        option dest_ip '192.168.101.0/255.255.255.0'
        option use_policy 'vpn1_only'

I used openvpn's route-nopull to ignored any routes pushed by the OpenVPN server and manually added policy-based routing (vpn_pbrX) rules.

However, mwan3 complains that vpn1/tun1 doesn't have a default route. I tried a separate "config route ..." in /etc/config/network, but it didn't seem to work either. Any suggestions ?

PS: One use scenario would be to send all traffic of an internal LAN IP via the VPN tunnel, to overcome geo-ip limitations (e.g. watch BBC news videos by tunneling all traffic via a UK-based VPN server)