OpenWrt Forum Archive

Topic: mwan3; multi-wan policy routing (general topic)

The content of this topic has been archived between 22 May 2013 and 6 May 2018. Unfortunately there are posts – most likely complete pages – missing.

This is my mwan3 configuration

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option enabled '1'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'
    option last_resort 'default'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'
    option last_resort 'default'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'
    list use_member 'wan3g_m2_w4'
    option last_resort 'unreachable'

config interface 'wan3g'
    option enabled '1'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '5'
    option up '5'

config member 'wan3g_m2_w4'
    option interface 'wan3g'
    option metric '3'
    option weight '4'

If wan3g up, then about one in ten of the site does not open. I think when the site does not open, dns don't resolve the name. If wan3g is down, everything works fine. Wan and wan2 is from same provider, wan3g from other provider. I want to use wan3g as a backup when the wan and wan2 will not be available. Please help solve this problem.

P.S.
Sorry for my english, i will use google translate.

(Last edited by Vahe91 on 7 Jul 2015, 08:03)

Vahe91 based on that configuration mwan3 isn't doing anything at all since you have no rules configured.

arfett wrote:

Vahe91 based on that configuration mwan3 isn't doing anything at all since you have no rules configured.

I'm sorry, i forgot to add these lines:

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

(Last edited by Vahe91 on 3 Jul 2015, 18:41)

Vahe91 can you post more troubleshooting information like page 1 of this thread suggests?

Network config, iptables, etc.

If you use the LuCI app you can just paste what's on the troubleshooting page into a code block.

(Last edited by arfett on 4 Jul 2015, 18:18)

arfett wrote:

Vahe91 can you post more troubleshooting information like page 1 of this thread suggests?

Network config, iptables, etc.

If you use the LuCI app you can just paste what's on the troubleshooting page into a code block.

nothing is displayed, only error in log


Sun Jul  5 21:32:20 2015 daemon.err uhttpd[1161]: /usr/lib/lua/luci/controller/mwan3.lua:229: attempt to index field 'version' (a nil value)

Vahe91 wrote:
arfett wrote:

Vahe91 can you post more troubleshooting information like page 1 of this thread suggests?

Network config, iptables, etc.

If you use the LuCI app you can just paste what's on the troubleshooting page into a code block.

nothing is displayed, only error in log


Sun Jul  5 21:32:20 2015 daemon.err uhttpd[1161]: /usr/lib/lua/luci/controller/mwan3.lua:229: attempt to index field 'version' (a nil value)

I would like to upload a new version of the LuCI app that fixes another issue in the latest trunk release and also allows the page to still load if OpenWrt doesn't feel like returning the version properly.

Can you please try editing your /usr/lib/lua/luci/controller/mwan3.lua lines 229-240 with the below and see if this fixes the error?

    local wrtRelease = luci.version.distversion
        if wrtRelease ~= nil and wrtRelease ~= "" then
            wrtRelease = "OpenWrt - " .. ut.trim(wrtRelease)
        else
            wrtRelease = "OpenWrt - unknown"
        end
    local luciRelease = luci.version.luciversion
        if luciRelease ~= nil and luciRelease ~= "" then
            luciRelease = "\nLuCI - " .. ut.trim(luciRelease)
        else
            luciRelease = "\nLuCI - unknown"
        end

(Last edited by arfett on 5 Jul 2015, 20:01)

arfett wrote:

I would like to upload a new version of the LuCI app that fixes another issue in the latest trunk release and also allows the page to still load if OpenWrt doesn't feel like returning the version properly.

Can you please try editing your /usr/lib/lua/luci/controller/mwan3.lua lines 229-240 with the below and see if this fixes the error?

    local wrtRelease = luci.version.distversion
        if wrtRelease ~= nil and wrtRelease ~= "" then
            wrtRelease = "OpenWrt - " .. ut.trim(wrtRelease)
        else
            wrtRelease = "OpenWrt - unknown"
        end
    local luciRelease = luci.version.luciversion
        if luciRelease ~= nil and luciRelease ~= "" then
            luciRelease = "\nLuCI - " .. ut.trim(luciRelease)
        else
            luciRelease = "\nLuCI - unknown"
        end

Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]: /usr/lib/lua/luci/dispatcher.lua:433: Failed to execute call dispatcher target for entry '/admin/network/mwan/advanced/troubleshooting_display'.
Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]: The called action terminated with an exception:
Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]: /usr/lib/lua/luci/controller/mwan3.lua:229: attempt to index field 'version' (a nil value)
Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]: stack traceback:
Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]:     [C]: in function 'assert'
Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]:     /usr/lib/lua/luci/dispatcher.lua:433: in function 'dispatch'
Mon Jul  6 10:26:17 2015 daemon.err uhttpd[1161]:     /usr/lib/lua/luci/dispatcher.lua:168: in function </usr/lib/lua/luci/dispatcher.lua:167>

mwan3-luci (luci-app-mwan3) updated to 1.4-3 with fixes for Chaos Calmer and trunk:

https://github.com/openwrt/packages/pull/1546



Vahe91 if you upgrade your LuCI application to 1.4-3 when the pull request is accepted or manually apply the changes from the pull request on your files then you should be able to view the troubleshooting page and paste all the information to us. Please use a code block on the forum so it doesn't eat up a whole page with the information.

(Last edited by arfett on 6 Jul 2015, 19:38)

arfett wrote:

mwan3-luci (luci-app-mwan3) updated to 1.4-3 with fixes for Chaos Calmer and trunk:

https://github.com/openwrt/packages/pull/1546



Vahe91 if you upgrade your LuCI application to 1.4-3 when the pull request is accepted or manually apply the changes from the pull request on your files then you should be able to view the troubleshooting page and paste all the information to us. Please use a code block on the forum so it doesn't eat up a whole page with the information.

How to upgrade ?

Vahe91 wrote:
arfett wrote:

mwan3-luci (luci-app-mwan3) updated to 1.4-3 with fixes for Chaos Calmer and trunk:

https://github.com/openwrt/packages/pull/1546



Vahe91 if you upgrade your LuCI application to 1.4-3 when the pull request is accepted or manually apply the changes from the pull request on your files then you should be able to view the troubleshooting page and paste all the information to us. Please use a code block on the forum so it doesn't eat up a whole page with the information.

How to upgrade ?

1. Manually edit the files in the same places as in the pull request
2. Reinstall after the pull request has been accepted

Software versions : 

OpenWrt - OpenWrt Chaos Calmer 15.05-rc2
LuCI - git-15.146.54948-a497fba

mwan3 - 1.6-1
mwan3-luci - 1.4-2

Output of "cat /etc/config/mwan3" : 

config rule 'seasonvar'
    option dest_ip '178.236.136.0/23'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'seasonvar2'
    option dest_ip '146.255.192.0/22'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'seasonvar1'
    option dest_ip '176.58.32.0/24'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'akumb'
    option dest_ip '37.139.24.0/21'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'watchis'
    option dest_ip '5.61.34.63'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'watchis1'
    option dest_ip '178.218.208.0/20'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'filmix'
    option dest_ip '37.48.64.0/18'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'filmix1'
    option dest_ip '109.200.11.224/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'filmix2'
    option dest_ip '213.186.127.0/24'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'filmix3'
    option dest_ip ' 213.186.126.240/28'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'filmix4'
    option dest_ip '212.113.32.0/21'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'filmix5'
    option dest_ip '213.186.122.0/24'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'onlinefilm'
    option dest_ip '50.7.0.0/16'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda2'
    option dest_ip ' 173.193.242.254'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda6'
    option dest_ip '109.73.68.248/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda8'
    option dest_ip ' 88.150.129.0/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda9'
    option dest_ip '149.3.131.88/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda10'
    option dest_ip '173.192.225.175'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda11'
    option dest_ip '109.200.21.160/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda14'
    option dest_ip '173.193.197.52'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda15'
    option dest_ip '50.22.223.21'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda16'
    option dest_ip '109.200.25.192/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda17'
    option dest_ip '149.3.134.80/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda18'
    option dest_ip '109.200.11.208/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda19'
    option proto 'all'
    option use_policy 'wan_only'
    option dest_ip ' 217.199.213.64/26'

config rule 'kinobanda21'
    option dest_ip ' 184.173.146.25'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda22'
    option dest_ip ' 173.192.217.6'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda23'
    option dest_ip '5.152.198.0/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda24'
    option dest_ip '37.220.30.144/29'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda25'
    option dest_ip '173.193.202.103'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda26'
    option dest_ip ' 173.193.214.232'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda27'
    option dest_ip '50.22.217.195'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda28'
    option dest_ip '208.43.81.144'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda29'
    option dest_ip '173.192.224.116'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'kinobanda30'
    option dest_ip '173.193.202.111'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'list'
    option dest_ip '190.93.241.127'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'list1'
    option dest_ip '190.93.242.127'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'list2'
    option dest_ip '190.93.243.127'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'list3'
    option dest_ip '141.101.112.128'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'list4'
    option dest_ip '141.101.113.128'
    option proto 'all'
    option use_policy 'wan_only'

config rule '2shared'
    option dest_ip '83.222.124.0/22'
    option proto 'all'
    option use_policy 'wan_only'

config rule '2shared1'
    option dest_ip '204.155.149.58'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'rt'
    option proto 'all'
    option use_policy 'wan2_only'
    option dest_ip '46.19.98.224/27'

config rule 'arca'
    option dest_ip '91.199.226.0/24'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'ucom'
    option proto 'all'
    option use_policy 'wan_only'
    option dest_ip '46.162.192.84'

config rule 'ucom1'
    option dest_ip '46.162.192.85'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'ineco'
    option dest_ip '93.94.222.160/28'
    option proto 'all'
    option use_policy 'wan_only'

config rule 'wot'
    option dest_ip '92.223.18.0/23'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'wot2'
    option dest_ip '92.223.4.0/20'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'wot3'
    option dest_ip '92.223.24.0/22'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo'
    option dest_ip '193.109.240.0/23'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo8'
    option dest_ip '104.20.7.84'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo9'
    option dest_ip '104.20.8.84'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo10'
    option dest_ip '104.20.9.84'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo11'
    option dest_ip '104.20.10.84'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo12'
    option dest_ip '104.20.11.84'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo13'
    option dest_ip '104.24.121.89'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'kinogo14'
    option dest_ip '104.24.120.89'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'idram'
    option dest_ip '77.72.131.0/24'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'videochart'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'
    option dest_ip '78.140.183.0/24'

config rule 'videochart1'
    option dest_ip '78.140.184.0/24'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'air1'
    option dest_ip ' 74.208.63.239'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'air'
    option dest_ip ' 193.104.87.0/24'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'autoam'
    option dest_ip ' 95.140.196.157'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan_only'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option enabled '1'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'
    option last_resort 'default'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'
    option last_resort 'default'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'
    list use_member 'wan3g_m2_w4'
    option last_resort 'unreachable'

config interface 'wan3g'
    option enabled '1'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '5'
    option up '5'

config member 'wan3g_m2_w4'
    option interface 'wan3g'
    option metric '3'
    option weight '4'

Output of "cat /etc/config/network" : 

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'

config interface 'lan'
    option ifname 'eth1'
    option force_link '1'
    option type 'bridge'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'
    option delegate '0'
    option igmp_snooping '0'

config interface 'wan'
    option proto 'dhcp'
    option _orig_ifname 'eth0'
    option _orig_bridge 'false'
    option ifname 'eth0.2'
    option metric '10'
    option macaddr '5E:DF:40:6C:59:C2'

config switch
    option name 'switch0'
    option reset '1'
    option enable_vlan '1'
    option mirror_source_port '0'
    option mirror_monitor_port '0'

config switch_vlan
    option device 'switch0'
    option vlan '1'
    option vid '1'
    option ports '0 2 3 4'

config switch_vlan
    option device 'switch0'
    option vlan '2'
    option vid '2'
    option ports '5 6t'

config switch_vlan
    option device 'switch0'
    option vlan '3'
    option vid '3'
    option ports '1 6t'

config interface 'wan2'
    option proto 'dhcp'
    option ifname 'eth0.3'
    option metric '20'
    option macaddr '4A:E3:07:02:42:0A'

config interface 'wan3g'
    option proto '3g'
    option delegate '0'
    option service 'umts_only'
    option apn 'internet'
    option pincode '1111'
    option dialnumber '*99***1#'
    option device '/dev/ttyUSB2'
    option metric '30'

Output of "ifconfig" : 

3g-wan3g  Link encap:Point-to-Point Protocol  
          inet addr:5.77.232.28  P-t-P:10.64.64.64  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:66 errors:0 dropped:0 overruns:0 frame:0
          TX packets:76 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3 
          RX bytes:2012 (1.9 KiB)  TX bytes:2176 (2.1 KiB)

br-lan    Link encap:Ethernet  HWaddr E8:94:F6:69:07:CA  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::ea94:f6ff:fe69:7ca/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3683 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4788 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:479598 (468.3 KiB)  TX bytes:2791448 (2.6 MiB)

eth0      Link encap:Ethernet  HWaddr E8:94:F6:69:07:CB  
          inet6 addr: fe80::ea94:f6ff:fe69:7cb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2620 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2730 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:936013 (914.0 KiB)  TX bytes:396002 (386.7 KiB)
          Interrupt:4 

eth0.2    Link encap:Ethernet  HWaddr 5E:DF:40:6C:59:C2  
          inet addr:46.162.216.55  Bcast:46.162.219.255  Mask:255.255.252.0
          inet6 addr: fe80::5cdf:40ff:fe6c:59c2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2093 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2154 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:788481 (770.0 KiB)  TX bytes:293947 (287.0 KiB)

eth0.3    Link encap:Ethernet  HWaddr 4A:E3:07:02:42:0A  
          inet addr:46.162.216.53  Bcast:46.162.219.255  Mask:255.255.252.0
          inet6 addr: fe80::48e3:7ff:fe02:420a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:527 errors:0 dropped:0 overruns:0 frame:0
          TX packets:568 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:100372 (98.0 KiB)  TX bytes:90279 (88.1 KiB)

eth1      Link encap:Ethernet  HWaddr E8:94:F6:69:07:CA  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3719 errors:0 dropped:3 overruns:0 frame:0
          TX packets:4797 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:536211 (523.6 KiB)  TX bytes:2793235 (2.6 MiB)
          Interrupt:5 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:106 errors:0 dropped:0 overruns:0 frame:0
          TX packets:106 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:8235 (8.0 KiB)  TX bytes:8235 (8.0 KiB)

Output of "route -n" : 

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         46.162.216.1    0.0.0.0         UG    10     0        0 eth0.2
0.0.0.0         46.162.216.1    0.0.0.0         UG    20     0        0 eth0.3
0.0.0.0         10.64.64.64     0.0.0.0         UG    30     0        0 3g-wan3g
10.64.64.64     0.0.0.0         255.255.255.255 UH    0      0        0 3g-wan3g
46.162.216.0    0.0.0.0         255.255.252.0   U     10     0        0 eth0.2
46.162.216.0    0.0.0.0         255.255.252.0   U     20     0        0 eth0.3
46.162.216.1    0.0.0.0         255.255.255.255 UH    10     0        0 eth0.2
46.162.216.1    0.0.0.0         255.255.255.255 UH    20     0        0 eth0.3
192.168.1.0     0.0.0.0         255.255.255.0   U     0      0        0 br-lan

Output of "ip rule show" : 

0:    from all lookup 128 
1:    from all lookup local 
1001:    from all iif eth0.2 lookup main 
1002:    from all iif eth0.3 lookup main 
1003:    from all iif 3g-wan3g lookup main 
2001:    from all fwmark 0x100/0xff00 lookup 1 
2002:    from all fwmark 0x200/0xff00 lookup 2 
2003:    from all fwmark 0x300/0xff00 lookup 3 
2253:    from all fwmark 0xfd00/0xff00 blackhole
2254:    from all fwmark 0xfe00/0xff00 unreachable
32766:    from all lookup main 
32767:    from all lookup default

Output of "ip route list table 1-250" : 

1
default via 46.162.216.1 dev eth0.2 
2
default via 46.162.216.1 dev eth0.3 
3
default via 10.64.64.64 dev 3g-wan3g

Firewall default output policy (must be ACCEPT) : 

ACCEPT

Output of "iptables -L -t mangle -v -n" : 

Chain PREROUTING (policy ACCEPT 318 packets, 21429 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  448 32965 mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  318 21429 fwmark     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 277 packets, 15451 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 41 packets, 5978 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   41  5978 mssfix     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 419 packets, 182K bytes)
 pkts bytes target     prot opt in     out     source               destination         
  591  292K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 460 packets, 188K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain fwmark (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain mssfix (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 TCPMSS     tcp  --  *      eth0.2  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* wan (mtu_fix) */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      eth0.3  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* wan2 (mtu_fix) */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      3g-wan3g  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* wan3g (mtu_fix) */ TCPMSS clamp to PMTU

Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   39  6169 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected dst MARK or 0xff00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 1039  325K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xff00
  171  7175 mwan3_ifaces  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  157  6165 mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  152  5679 mwan3_track  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
   24  1583 mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
 1039  325K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff00
   90  9433 mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0xff00/0xff00

Chain mwan3_iface_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0xff00 /* default */ MARK or 0xff00
   11   833 MARK       all  --  eth0.2 *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan */ MARK xset 0x100/0xff00

Chain mwan3_iface_wan2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  eth0.3 *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0xff00 /* default */ MARK or 0xff00
    3   177 MARK       all  --  eth0.3 *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan2 */ MARK xset 0x200/0xff00

Chain mwan3_iface_wan3g (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  3g-wan3g *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0xff00 /* default */ MARK or 0xff00
    0     0 MARK       all  --  3g-wan3g *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan3g */ MARK xset 0x300/0xff00

Chain mwan3_ifaces (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  170  7135 mwan3_iface_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  138  5269 mwan3_iface_wan2  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00
  123  4535 mwan3_iface_wan3g  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00

Chain mwan3_policy_balanced (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    3   295 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 statistic mode random probability 0.39999999991 /* wan2 2 5 */ MARK xset 0x200/0xff00
    4   304 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_policy_wan2_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan2 2 2 */ MARK xset 0x200/0xff00

Chain mwan3_policy_wan_only (63 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* wan 3 3 */ MARK xset 0x100/0xff00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            178.236.136.0/23     mark match 0x0/0xff00 /* seasonvar */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            146.255.192.0/22     mark match 0x0/0xff00 /* seasonvar2 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            176.58.32.0/24       mark match 0x0/0xff00 /* seasonvar1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            37.139.24.0/21       mark match 0x0/0xff00 /* akumb */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            5.61.34.63           mark match 0x0/0xff00 /* watchis */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            178.218.208.0/20     mark match 0x0/0xff00 /* watchis1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            37.48.64.0/18        mark match 0x0/0xff00 /* filmix */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            109.200.11.224/29    mark match 0x0/0xff00 /* filmix1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            213.186.127.0/24     mark match 0x0/0xff00 /* filmix2 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            213.186.126.240/28   mark match 0x0/0xff00 /* filmix3 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            212.113.32.0/21      mark match 0x0/0xff00 /* filmix4 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            213.186.122.0/24     mark match 0x0/0xff00 /* filmix5 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            50.7.0.0/16          mark match 0x0/0xff00 /* onlinefilm */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.193.242.254      mark match 0x0/0xff00 /* kinobanda2 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            109.73.68.248/29     mark match 0x0/0xff00 /* kinobanda6 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            88.150.129.0/29      mark match 0x0/0xff00 /* kinobanda8 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            149.3.131.88/29      mark match 0x0/0xff00 /* kinobanda9 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.192.225.175      mark match 0x0/0xff00 /* kinobanda10 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            109.200.21.160/29    mark match 0x0/0xff00 /* kinobanda11 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.193.197.52       mark match 0x0/0xff00 /* kinobanda14 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            50.22.223.21         mark match 0x0/0xff00 /* kinobanda15 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            109.200.25.192/29    mark match 0x0/0xff00 /* kinobanda16 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            149.3.134.80/29      mark match 0x0/0xff00 /* kinobanda17 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            109.200.11.208/29    mark match 0x0/0xff00 /* kinobanda18 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            217.199.213.64/26    mark match 0x0/0xff00 /* kinobanda19 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            184.173.146.25       mark match 0x0/0xff00 /* kinobanda21 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.192.217.6        mark match 0x0/0xff00 /* kinobanda22 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            5.152.198.0/29       mark match 0x0/0xff00 /* kinobanda23 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            37.220.30.144/29     mark match 0x0/0xff00 /* kinobanda24 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.193.202.103      mark match 0x0/0xff00 /* kinobanda25 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.193.214.232      mark match 0x0/0xff00 /* kinobanda26 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            50.22.217.195        mark match 0x0/0xff00 /* kinobanda27 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            208.43.81.144        mark match 0x0/0xff00 /* kinobanda28 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.192.224.116      mark match 0x0/0xff00 /* kinobanda29 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            173.193.202.111      mark match 0x0/0xff00 /* kinobanda30 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            190.93.241.127       mark match 0x0/0xff00 /* list */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            190.93.242.127       mark match 0x0/0xff00 /* list1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            190.93.243.127       mark match 0x0/0xff00 /* list2 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            141.101.112.128      mark match 0x0/0xff00 /* list3 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            141.101.113.128      mark match 0x0/0xff00 /* list4 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            83.222.124.0/22      mark match 0x0/0xff00 /* 2shared */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            204.155.149.58       mark match 0x0/0xff00 /* 2shared1 */
    0     0 mwan3_policy_wan2_only  all  --  *      *       0.0.0.0/0            46.19.98.224/27      mark match 0x0/0xff00 /* rt */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            91.199.226.0/24      mark match 0x0/0xff00 /* arca */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            46.162.192.84        mark match 0x0/0xff00 /* ucom */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            46.162.192.85        mark match 0x0/0xff00 /* ucom1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            93.94.222.160/28     mark match 0x0/0xff00 /* ineco */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            92.223.18.0/23       mark match 0x0/0xff00 /* wot */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            92.223.0.0/20        mark match 0x0/0xff00 /* wot2 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            92.223.24.0/22       mark match 0x0/0xff00 /* wot3 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            193.109.240.0/23     mark match 0x0/0xff00 /* kinogo */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.20.7.84          mark match 0x0/0xff00 /* kinogo8 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.20.8.84          mark match 0x0/0xff00 /* kinogo9 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.20.9.84          mark match 0x0/0xff00 /* kinogo10 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.20.10.84         mark match 0x0/0xff00 /* kinogo11 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.20.11.84         mark match 0x0/0xff00 /* kinogo12 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.24.121.89        mark match 0x0/0xff00 /* kinogo13 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            104.24.120.89        mark match 0x0/0xff00 /* kinogo14 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            77.72.131.0/24       mark match 0x0/0xff00 /* idram */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            78.140.183.0/24      mark match 0x0/0xff00 /* videochart */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            78.140.184.0/24      mark match 0x0/0xff00 /* videochart1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            74.208.63.239        mark match 0x0/0xff00 /* air1 */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            193.104.87.0/24      mark match 0x0/0xff00 /* air */
    0     0 mwan3_policy_wan_only  all  --  *      *       0.0.0.0/0            95.140.196.157       mark match 0x0/0xff00 /* autoam */
    7   599 mwan3_policy_balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xff00 /* default_rule */

Chain mwan3_track (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  128  4096 MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_track_wan dst icmptype 8 length 32 MARK or 0xff00
   86  2752 MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_track_wan2 dst icmptype 8 length 32 MARK or 0xff00
   49  1568 MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_track_wan3g dst icmptype 8 length 32 MARK or 0xff00

(Last edited by Vahe91 on 7 Jul 2015, 08:02)

What are the differences, benefits or disadvantages between the multiwan package and the mwan3 package? I can't find a single word about that...

I have two interface,wan and wwan.I don't want wan have access to 80 and 443 port.How to set?

(Last edited by axishero on 12 Jul 2015, 09:08)

axishero wrote:

I have two interface,wan and wwan.I don't want wan have access to 80 and 443 port.How to set?

I would imagine the standard OpenWrt firewall traffic rules is what you need.

arfett wrote:
axishero wrote:

I have two interface,wan and wwan.I don't want wan have access to 80 and 443 port.How to set?

I would imagine the standard OpenWrt firewall traffic rules is what you need.

Please give example

(Last edited by axishero on 14 Jul 2015, 17:47)

dir2cas wrote:
Adze wrote:
dir2cas wrote:

In fact after this correction, I do not see drops on the operational wan2, but the wan1 tracking (that is interrupted) works correctly only for the wan1's gateway. The other target hosts do not get detected.

Can you confirm with tcpdump that the ping requests do leave the correct interface?

All the tests done so far are pointing to the same problem.

Here is the latest one, that proves it. If you want tcpdump tests, I will have to check it doing some port mirroring on the switch, because this is e test router that has no tcpdump installed and has no free space because of the custom image that is full with different stiff. I hope that iftop is enough.

1) Switch port facing ISP1 (wan1) disabled

sw1#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
sw1(config)#int fa 0/1
sw1(config-if)#shu
sw1(config-if)#shutdown
sw1(config-if)#
sw1(config-if)#^Z
sw1#sh int desc | i WAN
Fa0/1                          admin down     down     WAN1_/24Mbps/
Fa0/2                          up             up       WAN2_/45Mbps/

2) Ping test using the same parameters like mwan3track (I restored the initial code). Here I used one of the host targets that is tracked only via wan1.
The purpose is that we would like the ping to be treated by mwan3 the same way as it is done by the real tracking tool (mwan3track). You can see that ping is successful, despite the wan1 link is interrupted.

[root@RTR-TEST:~]# $(which ping) -c 5 -W 1 -s 4 -I eth0.2 212.73.140.119
PING 212.73.140.119 (212.73.140.119) from 46.40.125.XX eth0.2: 4(32) bytes of data.
12 bytes from 212.73.140.119: icmp_req=1 ttl=59
12 bytes from 212.73.140.119: icmp_req=2 ttl=59
12 bytes from 212.73.140.119: icmp_req=3 ttl=59
12 bytes from 212.73.140.119: icmp_req=4 ttl=59
12 bytes from 212.73.140.119: icmp_req=5 ttl=59

--- 212.73.140.119 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms

[root@RTR-TEST:~]# $(which ping) -c 5 -W 1 -s 4 -I eth0.2 212.73.140.119
PING 212.73.140.119 (212.73.140.119) from 46.40.125.XX eth0.2: 4(32) bytes of data.
12 bytes from 212.73.140.119: icmp_req=1 ttl=59
12 bytes from 212.73.140.119: icmp_req=2 ttl=59
12 bytes from 212.73.140.119: icmp_req=3 ttl=59
12 bytes from 212.73.140.119: icmp_req=4 ttl=59
12 bytes from 212.73.140.119: icmp_req=5 ttl=59

--- 212.73.140.119 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
[root@RTR-TEST:~]# cat /etc/config/mwan3 | head -n 29

config interface 'wan1'
        option enabled '1'
        list track_ip '46.40.125.1'
        list track_ip '212.73.140.119'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option reliability '3'
        option count '5'
        option timeout '2'
        option interval '20'
        option down '1'
        option up '3'

config interface 'wan2'
        option enabled '1'
        list track_ip '46.249.80.1'
        list track_ip '87.120.130.66'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option reliability '3'
        option count '5'
        option timeout '2'
        option interval '20'
        option down '1'
        option up '3'

3) Iftop verifies that no traffic is passing via wan1(eth0.2) / ISP1 link, which should be correct

[root@RTR-TEST:~]# iftop -n -P -f "icmp" -i eth0.2

                           12.5kb                      25.0kb                     37.5kb                      50.0kb                62.5kb
└──────────────────────────┴───────────────────────────┴──────────────────────────┴───────────────────────────┴───────────────────────────

4) However the ping towards 212.73.140.119 is seen on wan2 (eth0.3). As seen above 212.73.140.119 is not part of the wan2 target hosts eligible for tracking.

[root@RTR-TEST:~]# iftop -n -P -f "icmp" -i eth0.3

                           12.5kb                      25.0kb                     37.5kb                      50.0kb                62.5kb
└──────────────────────────┴───────────────────────────┴──────────────────────────┴───────────────────────────┴───────────────────────────
46.249.80.XX                                            => 212.73.140.119                                           384b    256b    185b
                                                        <=                                                          384b    256b    185b
46.249.80.XX                                            => 8.8.8.8                                                    0b    128b     71b
                                                        <=                                                            0b    128b     71b
46.249.80.XX                                            => 192.58.128.30                                              0b    115b     64b
                                                        <=                                                            0b    115b     64b
46.249.80.XX                                            => 208.67.222.222                                             0b    102b     71b
                                                        <=                                                            0b    102b     57b

5) The log - mwan3 detects the fault only towards wan1's gateway, but not to the other trackable targets.

May 13 11:37:46 sw1 2066: May 13 11:37:45 EEST: %LINK-5-CHANGED: Interface FastEthernet0/1, changed state to administratively down
May 13 11:37:46 sw1 2067: May 13 11:37:46 EEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to down
May 13 11:37:46 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 60% packet loss
May 13 11:38:31 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:39:14 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:39:56 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:40:42 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:41:25 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:41:31 sw1 2068: May 13 11:41:30 EEST: %SYS-5-CONFIG_I: Configured from console by dir2cas on vty0 (10.0.20.200)
May 13 11:42:05 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:42:48 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:43:33 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:44:18 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:44:59 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:45:42 RTR-TEST mwan3track: Tracked host issue detected on Interface wan1 (eth0.2): 46.40.125.1 -> 100% packet loss
May 13 11:45:44 sw1 2069: May 13 11:45:43 EEST: %SYS-5-CONFIG_I: Configured from console by dir2cas on vty0 (10.0.20.200)
May 13 11:45:46 sw1 2070: May 13 11:45:45 EEST: %LINK-3-UPDOWN: Interface FastEthernet0/1, changed state to up
May 13 11:45:46 sw1 2071: May 13 11:45:46 EEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

Hi Adze, arfett,

This weekend I finally got some spare time to occupy with the problem again and carry out several experiments. I built another image in which I left the busybox ping applet untouched. Just in case I added mwan3 separately (not include it in the image) and installed the git version directly (currently 1.4...), but seems that my problem has nothing common with the package version.

Tested my configuration and works ok.
Once I remove the symlink to the ping command pointing to busybox (because I want to have available the iputils-ping), the issue observed previously started to be observed again.

So, I altered /usr/sbin/mwan3track, changing the line:

ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null

to

/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null

and mwan3 works again as expected.

I am still confused, why the iputils-ping does not work correctly with mwan3, probably it is related with an issue with the socket allocation, but probably needs some more thorough testing. I would like to ask you to try to add its support since it will make mwan3 more than an openwrt compatible instrument and add the possibility for more ping options. I would be happy to help if I can and have the time.

If no, probably it is better to commit the same change as mine in the source for now.

(Last edited by dir2cas on 26 Jul 2015, 17:33)

Hi Adze, affret
may I ask you again big_smile
I have 3 wan connection
1. cis --> pppoe
2. speedy --> pppoe
3. gsm --> umts 3g
they have failover for internet connection through "speedy" to "gsm" thanks for your incredible works
and also I have 1 tunnel openvpn, the tunnel openvpn use 1 of my 3 wan, it connect through "cis" and with script that affret wrote it can restarted automatically when interface down and it reconnected through "speedy" or "gsm"
for now I want to add one or more tunnel openvpn trough each interface, because I prefer don't want "restart" or "reconnect" openvpn, if one of my interface and tunnel openvpn down, I have other tunnel openvpn connection up.
is possible for mwan3 to failover traffic openvpn tunnel to other tunnel?
Thanks before

Vahe91 wrote:

This is my mwan3 configuration

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option enabled '1'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'
    option last_resort 'default'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'
    option last_resort 'default'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'
    list use_member 'wan3g_m2_w4'
    option last_resort 'unreachable'

config interface 'wan3g'
    option enabled '1'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '5'
    option up '5'

config member 'wan3g_m2_w4'
    option interface 'wan3g'
    option metric '3'
    option weight '4'

If wan3g up, then about one in ten of the site does not open. I think when the site does not open, dns don't resolve the name. If wan3g is down, everything works fine. Wan and wan2 is from same provider, wan3g from other provider. I want to use wan3g as a backup when the wan and wan2 will not be available. Please help solve this problem.

P.S.
Sorry for my english, i will use google translate.

please help to solve this problem

suriefkasev wrote:

is possible for mwan3 to failover traffic openvpn tunnel to other tunnel?

Yes. Create mwan3 interfaces for your tunnels as well.

Vahe91 wrote:
Vahe91 wrote:

This is my mwan3 configuration

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option enabled '1'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'
    option last_resort 'default'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'
    option last_resort 'default'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'
    list use_member 'wan3g_m2_w4'
    option last_resort 'unreachable'

config interface 'wan3g'
    option enabled '1'
    list track_ip '8.8.8.8'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '5'
    option up '5'

config member 'wan3g_m2_w4'
    option interface 'wan3g'
    option metric '3'
    option weight '4'

If wan3g up, then about one in ten of the site does not open. I think when the site does not open, dns don't resolve the name. If wan3g is down, everything works fine. Wan and wan2 is from same provider, wan3g from other provider. I want to use wan3g as a backup when the wan and wan2 will not be available. Please help solve this problem.

P.S.
Sorry for my english, i will use google translate.

please help to solve this problem

Probably DNS issue. Please use OpenDNS or Google DNS. See wiki for more info.

Adze wrote:

Probably DNS issue. Please use OpenDNS or Google DNS. See wiki for more info.

I added the rules for each dns server, and the problem was resolved. For example

config rule 'dns1'
    option dest_ip 'wan1_dns_ip'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan1_only'

config rule 'dns2'
    option dest_ip 'wan2_dns_ip'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan2_only'

config rule 'dns3'
    option dest_ip 'wan3g_dns_ip'
    option proto 'all'
    option sticky '0'
    option use_policy 'wan3g_only'

Hi, Sir

I have problem and try to use luci-app-mwan3 1.3-5 in TP-Link MR3020.

My Device: TPLINK-MR3020 + WAN = Wifi Built in
                                            + WAN2 = Wifi USB Alfa 036h

Openwrt Firmware Download : https://downloads.openwrt.org/barrier_b … actory.bin

opkg update
opkg install kmod-rtl8187
opkg install luci-app-mwan3


/etc/config/interface

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'
    option ula_prefix 'fdfa:c769:8668::/48'

config interface 'lan'
    option ifname 'eth0'
    option force_link '1'
    option type 'bridge'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'
    option ip6assign '60'

config interface 'wan'
    option proto 'dhcp'

config interface 'wan2'
    option proto 'dhcp'

/etc/config/multiwan

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan2'
    option enabled '1'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '2'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan_wan2'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_wan'
    list use_member 'wan_m2_w3'
    list use_member 'wan2_m1_w2'

config rule 'sticky_even'
    option src_ip '0.0.0.0/0.0.0.1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'wan_wan2'

config rule 'sticky_odd'
    option src_ip '0.0.0.1/0.0.0.1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'wan2_wan'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

/etc/config/firewall

config defaults
    option syn_flood '1'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'REJECT'

config zone
    option name 'lan'
    option input 'ACCEPT'
    option output 'ACCEPT'
    option forward 'ACCEPT'
    option network 'lan'

config zone
    option name 'wan'
    option output 'ACCEPT'
    option forward 'REJECT'
    option masq '1'
    option mtu_fix '1'
    option input 'ACCEPT'
    option network 'wan'

config rule
    option name 'Allow-DHCP-Renew'
    option src 'wan'
    option proto 'udp'
    option dest_port '68'
    option target 'ACCEPT'
    option family 'ipv4'

config rule
    option name 'Allow-Ping'
    option src 'wan'
    option proto 'icmp'
    option icmp_type 'echo-request'
    option family 'ipv4'
    option target 'ACCEPT'

config rule
    option name 'Allow-DHCPv6'
    option src 'wan'
    option proto 'udp'
    option src_ip 'fe80::/10'
    option src_port '547'
    option dest_ip 'fe80::/10'
    option dest_port '546'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-ICMPv6-Input'
    option src 'wan'
    option proto 'icmp'
    list icmp_type 'echo-request'
    list icmp_type 'echo-reply'
    list icmp_type 'destination-unreachable'
    list icmp_type 'packet-too-big'
    list icmp_type 'time-exceeded'
    list icmp_type 'bad-header'
    list icmp_type 'unknown-header-type'
    list icmp_type 'router-solicitation'
    list icmp_type 'neighbour-solicitation'
    list icmp_type 'router-advertisement'
    list icmp_type 'neighbour-advertisement'
    option limit '1000/sec'
    option family 'ipv6'
    option target 'ACCEPT'

config rule
    option name 'Allow-ICMPv6-Forward'
    option src 'wan'
    option dest '*'
    option proto 'icmp'
    list icmp_type 'echo-request'
    list icmp_type 'echo-reply'
    list icmp_type 'destination-unreachable'
    list icmp_type 'packet-too-big'
    list icmp_type 'time-exceeded'
    list icmp_type 'bad-header'
    list icmp_type 'unknown-header-type'
    option limit '1000/sec'
    option family 'ipv6'
    option target 'ACCEPT'

config include
    option path '/etc/firewall.user'

config zone
    option name 'wan2'
    option input 'ACCEPT'
    option forward 'REJECT'
    option output 'ACCEPT'
    option network 'wan2'
    option masq '1'
    option mtu_fix '1'

config forwarding
    option dest 'wan'
    option src 'lan'

config forwarding
    option dest 'wan2'
    option src 'lan'

MWAN Detailed Status

Interface status:
Interface wan is online (tracking active)
Interface wan2 is unknown

Policy balanced:
wan (100%)

Policy wan2_only:
unreachable

Policy wan2_wan:
wan (100%)

Policy wan_only:
wan (100%)

Policy wan_wan2:
wan (100%)

Known networks:
destination        policy             hits     
-----------------------------------------------
127.0.0.0/8        default            0       
224.0.0.0/3        default            0       
192.168.1.0/24     default            26       
192.168.43.0/24    default            1600     
127.0.0.0          default            0       
127.0.0.0/8        default            0       
127.0.0.1          default            0       
127.255.255.255    default            0       
192.168.1.0        default            0       
192.168.1.1        default            25       
192.168.1.255      default            1       
192.168.43.0       default            0       
192.168.43.149     default            1576     
192.168.43.255     default            0       

Active rules:
source             destination        proto  src-port      dest-port     policy          hits     
--------------------------------------------------------------------------------------------------
0.0.0.0/0.0.0.1    0.0.0.0/0          tcp    0:65535       443           wan_wan2        23       
0.0.0.1/0.0.0.1    0.0.0.0/0          tcp    0:65535       443           wan2_wan        0       
0.0.0.0/0          0.0.0.0/0          all                                balanced        414

hmm
Sorry, I'm basic to use.
I don't know how to edit config to begin.

Best, Regards

(Last edited by nui24711 on 2 Aug 2015, 17:07)

nui24711,

For starters put metrics on both interfaces in /etc/config/network. You can do this on LuCI. It doesn't matter what numbers you use but they must be different.

Try metric 10 on wan and metric 20 on wan2.

mwan3 with email notification and tracking verbosity enabled
/modifications applied/

Hello again,

In the process of testing the issue described in my previous posts, I did several modifications of mwan3 in order to increase the verbosity of the instrument. So, I decided to go a little further and add some hefty (according to me) features, using the great customize optional hotplug script of mwan3 together with some simple change in the mwan3 tracking engine - mwan3track. Tested and works OK on OpenWrt BB with mwan3 v1.5-10.

Probably this needs a separate topic, but here it is.

Brief description of the features added:
1) Email notification:

  • this feature is accomplished, modifying /etc/hotplug.d/iface/16-mwancustom

  • very thin email agent (mua) is used - ssmtp

  • the email agent is automatically configured for mwan3 usage by mwan3 itself

  • the only configuration needed is setting the relevant variables (in the relevant section) in /etc/hotplug.d/iface/16-mwancustom

  • the email content can be customized of course, either by changing the email variables for a basic change in the email structure, or by editing the code and alter the content that mwan3 pushes

  • the email notification is processes only on mwan3 enabled interfaces and only on interface status change

  • the email notification can be disabled by setting a preset variable (disables by default)

  • discovery and notification when the system has just been powered on / booted (up to 5 minutes after system boot) - in this case the message sent is simplified and mwan3 status reports and logs are not sent (can be disabled as well)

  • compatible with non-default logging targets (for example, if logging to /var/log/messages is configured), logread option is still available if no specific log file is configured

2) NAT table flushing on WAN switchover:

  • optional feature of  mwan3 also added in /etc/hotplug.d/iface/16-mwancustom - now it flushes the NAT table, which is highly recommended for normal and optimal NAT table operation upon WAN links switchover. This is important for services that are exposed to Internet via DNAT by the router doing the multiwan. This feature will work only if you have conntrack-tools installed, conntrack-tools depends on libnetfilter-conntrack, which are relatively small packages.

  • ex.)
    I am running a PBX behind the router with mwan3 and without NAT table flush, the sip connectivity does not work right, since the socket allocated from the primary wan somehow remains running on the other working one, after the primary wan failure and restoration as well. It is the same with openvpn instances that are terminated on an internal server (not the router itself, the router is doing simply DNAT). The main downside is that each connection from an internal host in the network (especially valid for TCP) has to be initiated again. Most services have daemons that are taking care of that, but if you are watching a video stream from your PC for example, you will have to reload it.

Here is the content of /etc/hotplug.d/iface/16-mwancustom

#!/bin/sh
. /lib/functions.sh

# to enable this script uncomment the case loop at the bottom
# to report mwan status on interface hotplug ifup/ifdown events modify the lines in the send_alert function

###################################
####### General variables: ########
###################################
# Define needed variables:
SCRIPTNAME="mwan3"
SYSTEM_LOG_FILE="/var/log/messages"             # Leave it empty if using the default syslogd configuration with logread macro
LOG="$(which logger) -t ${SCRIPTNAME} -p"
###################################
#
###################################
## Email Notification variables: ##
###################################
#
# Email client settings (needed only if EMAIL_NOTIFY="1"):
EMAIL_NOTIFY="0"                     # Enable (1) / Disable (0) email notification on failover actions
EMAIL_CLIENT="$(which ssmtp)"                 # The MUA (Mail user agent)
EMAIL_CLIENT_CONF="/etc/ssmtp/ssmtp_${SCRIPTNAME}.conf" # Where the configuration of the MUA is located
EMAIL_USER="my.example@gmail.com"             # The actual email account that is sending the email
EMAIL_PASSWD="MyEmailPassword"                 # Your email account password
SMTP_SERVER="smtp.gmail.com"                 # Outgoing mail server
DST_PORT="465"                         # Destiantion port used by the mail client (smtp port is 25 by default, without encryption)
EMAIL_DOMAIN="gmail.com"                 # Domain of the mail server
EMAIL_CLIENT_TLS="YES"                     # "YES" or "NO" are valid options
EMAIL_SUBJECT="Example, WAN status change detected"     # Subject of the email message
# List of Email notification recipients
EMAIL_TO="example1@domain1 example2@domain2 example3@domain3"
EMAIL_SIGNATURE="powered by mwan3 & OpenWrt"
###################################

send_alert()
{
        # variable "$1" stores the MWAN status information
        # insert your code here to send the contents of "$1"
     #echo "$1"

    ###################################
    local HOST="$(uci get -p /var/state system.@system[0].hostname)"
    # Check whether the email agent is installed
    if [ "${EMAIL_NOTIFY}" -ne "1" ]; then ${LOG} notice "Email alert notification is not enabled"; return 2; fi
    if [ ! -x "${EMAIL_CLIENT}" ]; then ${LOG} warning "Warning: Email client ${EMAIL_CLIENT} is not installed or configured on the system"; return 3; fi

    LOG_MESSAGE="${@}"
    sleep 1

    # Checking whether the Email client has already been configured for our account and if not, write its configuration
    grep -e "generated by ${SCRIPTNAME}" ${EMAIL_CLIENT_CONF} &> /dev/null || {
        # Backing up if there is some old configuration first
        cp ${EMAIL_CLIENT_CONF} ${EMAIL_CLIENT_CONF}.backup > /dev/null 2>&1
        # Writing the Email Client configuration (SSMTP)
        echo -e "# ${EMAIL_CLIENT_CONF} -- a configuration file for sSMTP sendmail.\n### generated by ${SCRIPTNAME}\nroot=${EMAIL_USER}\nmailhub=${SMTP_SERVER}:${DST_PORT}\nrewriteDomain=${EMAIL_DOMAIN}\nhostname=${HOST}\nFromLineOverride=YES\nUseTLS=${EMAIL_CLIENT_TLS}\nAuthUser=${EMAIL_USER}\nAuthPass=${EMAIL_PASSWD}" > ${EMAIL_CLIENT_CONF}
        }

    # Composing the Email message and pipe it to the smtp client in order to be sent
    sleep 1
    echo -e "From: <${EMAIL_USER}>\nSubject: ${EMAIL_SUBJECT}\n\n${HOST} alert notification message.\n\n$(echo -e "${LOG_MESSAGE}")\n\n\n${EMAIL_SIGNATURE}" | ${EMAIL_CLIENT} -C ${EMAIL_CLIENT_CONF} ${EMAIL_TO}
    sleep 1
    ${LOG} notice "Sending email alert notification to $(echo "${EMAIL_TO}"|sed "s/ /,/g")"
    unset EMAIL_PASSWD
    unset LOG_MESSAGE
    ###################################

}

gather_event_info()
{
    local device enabled
    config_load mwan3
    config_get enabled "${INTERFACE}" enabled 0
    device=$(uci get -p /var/state network.${INTERFACE}.ifname) &> /dev/null
    #echo -e "Interface is: ${INTERFACE}; Device is: ${device}|($DEVICE); enabled = ${enabled};"
    if [ -n "$device" ] ; then
        #ACTION=ifup INTERFACE=$1 DEVICE=$device /sbin/hotplug-call iface
        # The interface is not enabled in mwan3 configuration, so no message alerting should be initiated
        [ "${enabled}" -eq "1" ] || { echo "Warning: Interface ${INTERFACE} not enabled for message alerting"; exit 1; }
    else
        echo "Warning: Invalid interface call"; exit 1;
    fi

    # create event information message
    local EVENT_INFO="Interface [ "$INTERFACE" ($DEVICE) ] on router [ "$(uci get -p /var/state system.@system[0].hostname)" ] has triggered a hotplug [ "$ACTION" ] event on "$(date +"%a %b %d %Y %T %Z")""

    # get current interface, policy and rule status
    local CURRENT_STATUS="$(/usr/sbin/mwan3 status)"

    # get last 50 MWAN systemlog messages
    if [ -e ${SYSTEM_LOG_FILE} ] ; then
        local MWAN_LOG="$(echo -e "Last 50 MWAN systemlog entries. Newest entries sorted at the top:\n$($(which cat) ${SYSTEM_LOG_FILE} | grep mwan3 | tail -n 50 | sed 'x;1!H;$!d;x')")"
    else
        local MWAN_LOG="$(echo -e "Last 50 MWAN systemlog entries. Newest entries sorted at the top:\n$(logread | grep mwan3 | tail -n 50 | sed 'x;1!H;$!d;x')")"
    fi

    # get the uptime in minutes and send different notification if the system uptime is less than 5 mins
    local UPTIME_MINUTES=$($(which uptime) | sed s/^.*up// | awk -F, '{ if ( $3 ~ /user/ ) { print $1 $2 } else { print $1 }}' | sed 's/^ *//' | awk '{ if ( $2=="min" ) { print $1 } }' | cut -d ":" -f1)

    # pass event info to send_alert function
    [ ! -z "${UPTIME_MINUTES}" -a "${UPTIME_MINUTES}" -le "5" ] && { send_alert "$(echo -e "The system has just been powered on and started ${UPTIME_MINUTES} min ago!\n\n$EVENT_INFO")"; exit 0; }
    send_alert "$(echo -e "$EVENT_INFO\n\n$CURRENT_STATUS\n\n$MWAN_LOG")"

    # Flushing the NAT table
    [ -e $(which conntrack) ] && { ${LOG} notice "Flushing NAT Table"; sleep 1; $(which conntrack) -F &> /dev/null; sleep 1; };
}


case "$ACTION" in
       ifup)
               gather_event_info
       ;;

       ifdown)
               gather_event_info
       ;;
esac

exit 0

3) Tracking verbosity:

  • this feature is accomplished with small modification of /usr/sbin/mwan3track, which is not recommended at all, so a backup of the original file is obligatory

  • mwan3track now detects the failure/recovery based on packet loss values, comparing each result with a predefined packet loss threshold that we can set and consider crucial for the connectivity

  • if packet loss above the defined threshold is observed to any of the tracked hosts, a log message is produced in the system log

  • the above rule is valid while the interface is NOT marked as inactive/offline, in order to avoid the system log to be flooded with similar messages in the periods during which the connectivity via the particular interface is down

  • this feature helps to keep an eye on the link quality as well, because we can detect packet loss (even if it is not 100%) to the destinations tracked by mwan3. If we choose the monitored hosts in the right way, we can detect problems not only via a specific interface, but related to a global network issues, end destinations outages, etc

  • busybox ping applet is now hardcoded in order to avoid the issue described in my previous posts

Here is the content of the modified /usr/sbin/mwan3track

#!/bin/sh

[ -z "$9" ] && echo "Error: should not be started manually" && exit 0

if [ -e /var/run/mwan3track-$1.pid ] ; then
    kill $(cat /var/run/mwan3track-$1.pid) &> /dev/null
    rm /var/run/mwan3track-$1.pid &> /dev/null
fi

echo "$$" > /var/run/mwan3track-$1.pid

score=$(($7+$8))
track_ips=$(echo $* | cut -d ' ' -f 9-99)
host_up_count=0
lost=0

######
# Custom variables:
MUTE_WHILE_DOWN="0"     #Do not edit - prevents logging of packet loss results while the connection vie the interface is down
PLOSS_DEFAULT="100"     #Default Packet loss rate in %
PLOSS_THRESHOLD="5"     #Packet loss rate in %, that triggers host condition change (up|down)
######

while true; do

    for track_ip in $track_ips; do
        ######
        #/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
        #/usr/bin/ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
        PING_RESULT=$(/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip | grep "packet loss") &> /dev/null
        PLOSS=$(echo "${PING_RESULT}" | sed -e 's/^.*received, //' -e 's/^.*errors, //' -e 's/% packet loss.*//')
        if [ -z "${PLOSS}" ]; then PLOSS=${PLOSS_DEFAULT}; fi
        
        #if [ $? -eq 0 ]; then
        if [ "${PLOSS}" -le "${PLOSS_THRESHOLD}" ]; then
        ######
            let host_up_count++
        else
            let lost++
            ######
            [ "${MUTE_WHILE_DOWN}" -eq "0" ] && logger -t mwan3track -p notice "Network issue detected on Interface $1 ($2): $track_ip -> ${PLOSS}% packet loss";
            ######
        fi
    done
        
    if [ $host_up_count -lt $3 ]; then
        let score--

        if [ $score -lt $8 ]; then score=0 ; fi
        if [ $score -eq $8 ]; then

            logger -t mwan3track -p notice "Interface $1 ($2) is offline"
            env -i ACTION=ifdown INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
            score=0
            ######
            MUTE_WHILE_DOWN="1"
            ######
        fi

    else

        if [ $score -lt $(($7+$8)) ] && [ $lost -gt 0 ]; then

            logger -t mwan3track -p info "Lost $(($lost*$4)) ping(s) on interface $1 ($2)"

        fi

        let score++
        lost=0

        if [ $score -gt $8 ]; then score=$(($7+$8)); fi
        if [ $score -eq $8 ]; then

            logger -t mwan3track -p notice "Interface $1 ($2) is online"
            env -i ACTION=ifup INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
            ######
            MUTE_WHILE_DOWN="0"
            ######
            rm /var/run/mwan3track-$1.pid
            exit 0
        fi
    fi

    host_up_count=0
    sleep $6
done

exit 1

And diff against the original file

[root@Core-Sf:~]# diff /usr/sbin/mwan3track /usr/sbin/mwan3track.original 
--- /usr/sbin/mwan3track
+++ /usr/sbin/mwan3track.original
@@ -14,32 +14,14 @@
 host_up_count=0
 lost=0
 
-######
-# Custom variables:
-MUTE_WHILE_DOWN="0"     #Do not edit - prevents logging of packet loss results while the connection vie the interface is down
-PLOSS_DEFAULT="100"     #Default Packet loss rate in %
-PLOSS_THRESHOLD="5"     #Packet loss rate in %, that triggers host condition change (up|down)
-######
-
 while true; do
 
     for track_ip in $track_ips; do
-        ######
-        #/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
-        #/usr/bin/ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
-        PING_RESULT=$(/bin/busybox ping -I $2 -c $4 -W $5 -s 4 -q $track_ip | grep "packet loss") &> /dev/null
-        PLOSS=$(echo "${PING_RESULT}" | sed -e 's/^.*received, //' -e 's/^.*errors, //' -e 's/% packet loss.*//')
-        if [ -z "${PLOSS}" ]; then PLOSS=${PLOSS_DEFAULT}; fi
-        
-        #if [ $? -eq 0 ]; then
-        if [ "${PLOSS}" -le "${PLOSS_THRESHOLD}" ]; then
-        ######
+        ping -I $2 -c $4 -W $5 -s 4 -q $track_ip &> /dev/null
+        if [ $? -eq 0 ]; then
             let host_up_count++
         else
             let lost++
-            ######
-            [ "${MUTE_WHILE_DOWN}" -eq "0" ] && logger -t mwan3track -p notice "Network issue detected on Interface $1 ($2): $track_ip -> ${PLOSS}% packet loss";
-            ######
         fi
     done
         
@@ -52,9 +34,7 @@
             logger -t mwan3track -p notice "Interface $1 ($2) is offline"
             env -i ACTION=ifdown INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
             score=0
-            ######
-            MUTE_WHILE_DOWN="1"
-            ######
+
         fi
 
     else
@@ -73,9 +53,6 @@
 
             logger -t mwan3track -p notice "Interface $1 ($2) is online"
             env -i ACTION=ifup INTERFACE=$1 DEVICE=$2 /sbin/hotplug-call iface
-            ######
-            MUTE_WHILE_DOWN="0"
-            ######
             rm /var/run/mwan3track-$1.pid
             exit 0
         fi

The options and the variables are self-explanatory, but several comments were added as well.

Here is a snippet of the mwan3 logging, showing how it reacts on a network issue affecting one of the wan links. This is replicated, by disabling a phy link (connected to a manageable l2 switch) throughout the path to the ISP, but not physically connected to the mwan3 router itself, in order to avoid the generic netifd mechanisms to kick in, thus emulating a connectivity problem.

Aug 17 00:31:30 switch 786: Aug 17 00:31:27 EEST: %LINK-5-CHANGED: Interface FastEthernet0/2, changed state to administratively down
Aug 17 00:31:30 switch 787: Aug 17 00:31:28 EEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to down
Aug 17 00:31:34 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 208.67.220.220 -> 100% packet loss
Aug 17 00:31:41 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 208.67.222.222 -> 100% packet loss
Aug 17 00:31:47 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 8.8.8.8 -> 100% packet loss
Aug 17 00:31:53 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 87.120.130.66 -> 100% packet loss
Aug 17 00:31:59 Core-Sf mwan3track: Network issue detected on Interface wan2 (eth0.3): 46.249.80.1 -> 100% packet loss
Aug 17 00:31:59 Core-Sf mwan3track: Interface wan2 (eth0.3) is offline
Aug 17 00:31:59 Core-Sf mwan3: ifdown interface wan2 (eth0.3)
Aug 17 00:32:06 Core-Sf sSMTP[5675]: Creating SSL connection to host
Aug 17 00:32:07 Core-Sf sSMTP[5675]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Aug 17 00:32:10 Core-Sf sSMTP[5675]: Sent mail for my.email@gmail.com (221 2.0.0 closing connection lk16sm13747142wic.6 - gsmtp) uid=0 username=root outbytes=8273
Aug 17 00:32:11 Core-Sf mwan3: Sending email alert notification to email1@domain1,email2@domain2,email3@domain3
Aug 17 00:32:11 Core-Sf mwan3: Flushing NAT Table
Aug 17 00:32:13 Core-Sf root: stopping ntpclient


########## bringing the switch port up after 3-4 minutes ########## 


Aug 17 00:34:08 switch 788: Aug 17 00:34:07 EEST: %LINK-3-UPDOWN: Interface FastEthernet0/2, changed state to up
Aug 17 00:34:08 switch 789: Aug 17 00:34:08 EEST: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/2, changed state to up
Aug 17 00:35:22 Core-Sf mwan3track: Lost 95 ping(s) on interface wan2 (eth0.3)
Aug 17 00:36:42 Core-Sf mwan3track: Interface wan2 (eth0.3) is online
Aug 17 00:36:43 Core-Sf mwan3: ifup interface wan2 (eth0.3)
Aug 17 00:36:50 Core-Sf sSMTP[6647]: Creating SSL connection to host
Aug 17 00:36:50 Core-Sf sSMTP[6647]: SSL connection using ECDHE-RSA-AES128-GCM-SHA256
Aug 17 00:36:53 Core-Sf sSMTP[6647]: Sent mail for my.email@gmail.com (221 2.0.0 closing connection ir5sm14765639wjb.23 - gsmtp) uid=0 username=root outbytes=8678
Aug 17 00:36:54 Core-Sf mwan3: Sending email alert notification to email1@domain1,email2@domain2,email3@domain3
Aug 17 00:36:54 Core-Sf mwan3: Flushing NAT Table
Aug 17 00:36:57 Core-Sf firewall: Reloading firewall due to ifup of wan2 (eth0.3)
Aug 17 00:36:57 Core-Sf mwan3track: Network issue detected on Interface wan1 (eth0.2): 212.73.140.119 -> 100% packet loss
Aug 17 00:36:58 Core-Sf root: starting ntpclient

I believe that most of these modifications can be added to mwan3 project, especially the notification feature which can be imported in the mwan3 uci configuration file.

Download link for the the mwan3 files that were modified.

Hope, someone will find this useful.

Thanks for your work. I never really put any time into making the email stuff easier on the custom hotplug script. I'll look over your suggestions this upcoming weekend and see about adding them to the source.