Mwan3 - interface wan is offline and tracking is down

yurtesen · November 12, 2019, 12:54pm

I have setup metric in interfaces. The wan interface is working and in mwan3 diagnostics I am able to ping the gateway using it. The wwan interface is with ncm/qmi and on 4G, the wan interface is the wan port of the router.

However I see this:

Interface status:
 interface wan is offline and tracking is down
 interface wwan is online and tracking is active

If I re/start wan interface, it registers as online/active. But when I apply some settings to mwan3, it goes offline/down.

I checked the interface configs in /etc/config/mwan3 for wwan and wan and they are both same. On Interfaces menu of OpenWRT the wan interface has IPv4 address and seems to be working.

Any ideas why this might be happening?

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'dhcp'
        option metric '20'

config interface 'wwan'
        option ifname 'wwan0'
        option proto 'dhcp'
        option metric '10'

Logs show::;

Tue Nov 12 14:50:38 2019 user.notice mwan3track[10261]: Stopping mwan3track for interface "wwan"
Tue Nov 12 14:50:39 2019 user.notice mwan3track[30537]: Stopping mwan3track for interface "wan"
Tue Nov 12 14:50:40 2019 user.notice mwan3track[32104]: Stopping mwan3track for interface "wan"
Tue Nov 12 14:50:44 2019 user.notice mwan3[31909]: Using firewall mask 0x3F00
Tue Nov 12 14:50:44 2019 user.notice mwan3[31909]: Max interface count is 60
Tue Nov 12 14:50:44 2019 user.notice mwan3[31909]: Execute ifup event on interface wwan (wwan0)
Tue Nov 12 14:50:45 2019 user.notice mwan3[31909]: Starting tracker on interface wwan (wwan0)
Tue Nov 12 14:50:46 2019 user.info mwan3[31909]: connection tracking not flushed on interface wwan (wwan0) ifup
Tue Nov 12 14:50:46 2019 user.notice firewall: Reloading firewall due to ifup of wwan (wwan0)
Tue Nov 12 14:50:46 2019 user.notice root: iface wwan up detected...
Tue Nov 12 14:50:48 2019 user.notice mwan3[32570]: Execute ifup event on interface wan (wwan0)
Tue Nov 12 14:50:49 2019 user.notice mwan3[32570]: Starting tracker on interface wan (wwan0)
Tue Nov 12 14:50:51 2019 user.info mwan3[32570]: connection tracking not flushed on interface wan (wwan0) ifup
Tue Nov 12 14:50:51 2019 user.notice firewall: Reloading firewall due to ifup of wan (wwan0)

The mwan3 config is:

config globals 'globals'
        option mmx_mask '0x3F00'
        option local_source 'lan'

config member 'wan_m1_w3'
        option interface 'wan'
        option metric '1'
        option weight '3'

config member 'wan_m2_w3'
        option interface 'wan'
        option metric '2'
        option weight '3'

config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'wwan_wan'

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option proto 'all'
        option sticky '0'
        option use_policy 'wwan_wan'

config interface 'wwan'
        option enabled '1'
        option family 'ipv4'
        option track_method 'ping'
        option reliability '2'
        option count '1'
        option size '56'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option flush_conntrack 'never'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.222.222'
        list track_ip '208.67.220.220'
        option up '8'
        option initial_state 'online'

config member 'wwan_m1_w2'
        option interface 'wwan'
        option metric '1'
        option weight '2'

config member 'wwan_m2_w2'
        option interface 'wwan'
        option metric '2'
        option weight '2'

config member 'wan_m1_2'
        option interface 'wan'
        option metric '1'
        option weight '2'

config policy 'wan_wwan'
        list use_member 'wan_m1_w3'
        list use_member 'wwan_m2_w2'
        option last_resort 'unreachable'

config policy 'wwan_wan'
        list use_member 'wwan_m1_w2'
        list use_member 'wan_m2_w3'
        option last_resort 'unreachable'

config policy 'balanced'
        list use_member 'wwan_m1_w2'
        list use_member 'wan_m1_2'
        option last_resort 'unreachable'

config interface 'wan'
        option enabled '1'
        option initial_state 'online'
        option family 'ipv4'
        list track_ip '8.8.4.4'
        list track_ip '8.8.8.8'
        list track_ip '208.67.220.220'
        list track_ip '208.67.222.222'
        option track_method 'ping'
        option reliability '1'
        option count '1'
        option size '56'
        option check_quality '0'
        option timeout '2'
        option interval '5'
        option failure_interval '5'
        option recovery_interval '5'
        option down '3'
        option up '8'
        option flush_conntrack 'never'

mk24 · November 12, 2019, 2:15pm

It is configured to prefer wwan and use wan only if wwan fails. Is that what you want?

yurtesen · November 12, 2019, 2:27pm

@mk4 I think it has nothing to do with this problem. It behaves exactly same if I set it up to use wan first then wwan. I tried both ways. Furthermore, it works if I restart the router, until I apply a setting at Load Balancing menu.

When I boot the router I see:

Tue Nov 12 16:15:19 2019 user.notice mwan3[3267]: Execute ifup event on interface wan (eth0.2)
Tue Nov 12 16:15:20 2019 user.notice mwan3[3267]: Starting tracker on interface wan (eth0.2)

However when I change some settings in Load Balancing I see:

Tue Nov 12 16:18:14 2019 user.notice mwan3[26274]: Execute ifup event on interface wan (wwan0)
Tue Nov 12 16:18:14 2019 user.notice mwan3[26274]: Starting tracker on interface wan (wwan0)

I am not sure why it thinks wan is mapped to wwan0 because it gets it correctly first. If I restart wan interface manually from Interfaces then I see:

Tue Nov 12 16:26:53 2019 user.notice mwan3[28035]: Execute ifup event on interface wan (eth0.2)
Tue Nov 12 16:26:54 2019 user.notice mwan3[28035]: Starting tracker on interface wan (eth0.2)
Tue Nov 12 16:26:56 2019 user.info mwan3[28035]: connection tracking not flushed on interface wan (eth0.2) ifup

I guess the question is why mwan3 is making this mistake?

yurtesen · November 12, 2019, 3:12pm

If I execute ifdown and ifup myself on wan then I see the correct interface name.
But when mwan3 does that, there is always wwan0. How does mwan3 do this?
I think it may be a bug...

yurtesen · November 12, 2019, 3:30pm

Looks like truely strange things are happening:

root@OpenWrt:~# /usr/sbin/mwan3 ifup 'wan'
Failed to parse message data
sh: out of range
root@OpenWrt:~# /usr/sbin/mwan3 ifup 'wwan'
uci: Entry not found
uci: Entry not found
grep: /var/etc/miniupnpd.conf: No such file or directory
uci: Entry not found
uci: Entry not found
grep: /var/etc/miniupnpd.conf: No such file or directory
root@OpenWrt:~#

yurtesen · November 12, 2019, 3:58pm

@mk24 well there was a bug in mwan3 I wonder if it really worked for anybody

It was a small problem and I made a PR for it.

Now it seems to work fine as can be seen below:

Tue Nov 12 17:56:12 2019 user.notice mwan3track[3217]: Stopping mwan3track for interface "wwan"
Tue Nov 12 17:56:13 2019 user.notice mwan3track[4744]: Stopping mwan3track for interface "wan"
Tue Nov 12 17:56:16 2019 user.notice mwan3[12624]: Using firewall mask 0x3F00
Tue Nov 12 17:56:16 2019 user.notice mwan3[12624]: Max interface count is 60
Tue Nov 12 17:56:17 2019 user.notice mwan3[12624]: Execute ifup event on interface wwan (wwan0)
Tue Nov 12 17:56:17 2019 user.notice mwan3[12624]: Starting tracker on interface wwan (wwan0)
Tue Nov 12 17:56:18 2019 user.info mwan3[12624]: connection tracking not flushed on interface wwan (wwan0) ifup
Tue Nov 12 17:56:21 2019 user.notice mwan3[13132]: Execute ifup event on interface wan (eth0.2)
Tue Nov 12 17:56:21 2019 user.notice mwan3[13132]: Starting tracker on interface wan (eth0.2)
Tue Nov 12 17:56:23 2019 user.info mwan3[13132]: connection tracking not flushed on interface wan (eth0.2) ifup