Mwan3 first 4 pings drop on wlanb

madPL · March 14, 2023, 8:19pm

Hi,

After configuring mwan3 (https://openwrt.org/docs/guide-user/network/wan/multiwan/mwan3) I have problem with ping using second interface. First 4 pings are dropped.

root@OpenWrt:~# ping -I lan1 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=4 ttl=56 time=23.396 ms
64 bytes from 1.1.1.1: seq=5 ttl=56 time=23.594 ms

When I switch assing policy to wanb_only it works correctly. The same if I stop mwan3. Also If I use mwan3 use wanb ping 1.1.1.1 pings works ok.

Because of this problem I can see in logs that mwan3track can't ping target.

Tue Mar 14 20:07:12 2023 user.info mwan3track[10991]: Check (ping) failed for target "1.1.1.1" on interface wanb (lan1). Current score: 10
Tue Mar 14 20:09:36 2023 user.info mwan3track[10991]: Check (ping) failed for target "1.1.1.1" on interface wanb (lan1). Current score: 10

OpenWrt version: OpenWrt 22.03.2 r19803-9a599fee93

This looks like similar issue Mwan3 - wanb stops working as soon as primary wan interfaces goes down

trendy · March 14, 2023, 9:00pm

Are these 4 packets actually sent out of lan1 interface? If not, which interface is used? Is there any response coming back even delayed?
You can verify with tcpdump.

madPL · March 14, 2023, 9:15pm

There is nothing about first 4 pings on tcpdump

root@OpenWrt:~# tcpdump -i lan1 icmp and dst host 1.1.1.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lan1, link-type EN10MB (Ethernet), capture size 262144 bytes
21:08:57.909271 IP 192.168.2.229 > one.one.one.one: ICMP echo request, id 20624, seq 4, length 64
21:08:58.910228 IP 192.168.2.229 > one.one.one.one: ICMP echo request, id 20624, seq 5, length 64

I checked other interfaces (wan) and don't see any logs there for this ping.

Anyway I checked how mwan3track works and there is no problem because it use LD_PRELOAD=/lib/mwan3/libwrap_mwan3_sockopt.so.1.0.

root@OpenWrt:~# mwan3 use wanb ping 1.1.1.1
Running 'ping 1.1.1.1' with DEVICE=lan1 SRCIP=192.168.2.229 FWMARK=0x3f00 FAMILY=ipv4
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=56 time=25.914 ms
64 bytes from 1.1.1.1: seq=1 ttl=56 time=28.807 ms
64 bytes from 1.1.1.1: seq=2 ttl=56 time=22.932 ms

But still not sure why ping -I lan1 1.1.1.1 drops 4 packages.

sscheib · March 14, 2023, 11:04pm

For me a simple reboot fixed the issue temporarily. Just yesterday when wanA went down I had no fail over. After rebooting the router and retrying a failover, everything worked. This seems to be only an issue after a while - for whatever reason - and only if wanA goes down in my case. If wanB goes down, the failover is smooth.

And yes, I also couldn't see any paket before the 5th one in tcpdump when using plain ping.

sscheib · March 25, 2023, 8:16pm

I did a bit more digging, and this really makes no sense to me. I think it is an issue with mwan3, but I cannot say for sure. Especially, since failover does not work at all, when wanA goes down.
When mwan3 is disabled, pings go through both wanA and wanB properly.

wanA aka pppoe-wan (in my case) when mwan3 is stopped

root@openwrt.dmz.int.example.com:~# ping -I pppoe-wan heise.de -4 -c 5
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: seq=0 ttl=248 time=15.008 ms
64 bytes from 193.99.144.80: seq=1 ttl=248 time=14.940 ms
64 bytes from 193.99.144.80: seq=2 ttl=248 time=15.073 ms
64 bytes from 193.99.144.80: seq=3 ttl=248 time=14.691 ms
64 bytes from 193.99.144.80: seq=4 ttl=248 time=15.061 ms

--- heise.de ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 14.691/14.954/15.073 ms
root@openwrt.dmz.int.example.com:~# 

# tcpdump of the pings in seperate SSH session
root@openwrt.dmz.int.example.com:~# tcpdump -enni any dst host heise.de -vvv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
18:07:19.640195 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 51366, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 9550, seq 0, length 64
18:07:20.640436 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 51443, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 9550, seq 1, length 64
18:07:21.640573 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 51518, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 9550, seq 2, length 64
18:07:22.640707 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 51576, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 9550, seq 3, length 64
18:07:23.640848 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 51666, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 9550, seq 4, length 64

wanb aka lan4 (in my case) when mwan3 is stopped

root@openwrt.dmz.int.example.com:~# ping -I lan4 heise.de -4 -c 5
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: seq=0 ttl=245 time=17.560 ms
64 bytes from 193.99.144.80: seq=1 ttl=245 time=9.586 ms
64 bytes from 193.99.144.80: seq=2 ttl=245 time=11.785 ms
64 bytes from 193.99.144.80: seq=3 ttl=245 time=10.242 ms
64 bytes from 193.99.144.80: seq=4 ttl=245 time=10.854 ms

--- heise.de ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 9.586/12.005/17.560 ms
root@openwrt.dmz.int.example.com:~# 

# tcpdump of the pings in seperate SSH session
root@openwrt.dmz.int.example.com:~# tcpdump -enni any dst host heise.de -vvv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
18:11:28.554726 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 12898, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 0, length 64
18:11:28.554747 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 12898, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 0, length 64
18:11:29.554816 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 12930, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 1, length 64
18:11:29.554827 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 12930, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 1, length 64
18:11:30.554874 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 12967, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 2, length 64
18:11:30.554884 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 12967, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 2, length 64
18:11:31.554927 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 13067, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 3, length 64
18:11:31.554935 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 13067, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 3, length 64
18:11:32.554979 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 13127, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 4, length 64
18:11:32.554987 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 13127, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 9573, seq 4, length 64

Then, when mwan3 is started, the following can be observed:
wanA aka pppoe-wan (in my case) when mwan3 is started - looks exactly the same

root@openwrt.dmz.int.example.com:~# /etc/init.d/mwan3 start
root@openwrt.dmz.int.example.com:~# ping -I pppoe-wan heise.de -4 -c 5
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: seq=0 ttl=248 time=15.083 ms
64 bytes from 193.99.144.80: seq=1 ttl=248 time=15.037 ms
64 bytes from 193.99.144.80: seq=2 ttl=248 time=15.151 ms
64 bytes from 193.99.144.80: seq=3 ttl=248 time=15.014 ms
64 bytes from 193.99.144.80: seq=4 ttl=248 time=14.704 ms

--- heise.de ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 14.704/14.997/15.151 ms
root@openwrt.dmz.int.example.com:~# 

# tcpdump of the pings in seperate SSH session
root@openwrt.dmz.int.example.com:~# tcpdump -enni any dst host heise.de -vvv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
20:44:30.465624 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 55839, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 11232, seq 0, length 64
20:44:31.465882 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 55932, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 11232, seq 1, length 64
20:44:32.466057 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 56032, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 11232, seq 2, length 64
20:44:33.466134 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 56065, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 11232, seq 3, length 64
20:44:34.466202 Out ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 56146, offset 0, flags [DF], proto ICMP (1), length 84)
    10.2.174.182 > 193.99.144.80: ICMP echo request, id 11232, seq 4, length 64

wanB aka lan4 (in my case) when mwan3 is started - strange things happen

root@openwrt.dmz.int.example.com:~# ping -I lan4 heise.de -4 -c 5
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: seq=4 ttl=245 time=14.976 ms

--- heise.de ping statistics ---
5 packets transmitted, 1 packets received, 80% packet loss
round-trip min/avg/max = 14.976/14.976/14.976 ms
root@openwrt.dmz.int.example.com:~# 

# tcpdump of the pings in seperate SSH session
root@openwrt.dmz.int.example.com:~# tcpdump -enni any dst host heise.de -vvv
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
20:46:25.910403 Out AA:BB:CC:DD:EE:FF ethertype ARP (0x0806), length 44: Ethernet (len 6), IPv4 (len 4), Request who-has 193.99.144.80 tell 10.1.217.230, length 28
20:46:25.910423 Out AA:BB:CC:DD:EE:FF ethertype ARP (0x0806), length 44: Ethernet (len 6), IPv4 (len 4), Request who-has 193.99.144.80 tell 10.1.217.230, length 28
20:46:26.964544 Out AA:BB:CC:DD:EE:FF ethertype ARP (0x0806), length 44: Ethernet (len 6), IPv4 (len 4), Request who-has 193.99.144.80 tell 10.1.217.230, length 28
20:46:26.964554 Out AA:BB:CC:DD:EE:FF ethertype ARP (0x0806), length 44: Ethernet (len 6), IPv4 (len 4), Request who-has 193.99.144.80 tell 10.1.217.230, length 28
20:46:28.014542 Out AA:BB:CC:DD:EE:FF ethertype ARP (0x0806), length 44: Ethernet (len 6), IPv4 (len 4), Request who-has 193.99.144.80 tell 10.1.217.230, length 28
20:46:28.014551 Out AA:BB:CC:DD:EE:FF ethertype ARP (0x0806), length 44: Ethernet (len 6), IPv4 (len 4), Request who-has 193.99.144.80 tell 10.1.217.230, length 28
20:46:29.910651 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 39265, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 11935, seq 4, length 64
20:46:29.910664 Out AA:BB:CC:DD:EE:FF ethertype IPv4 (0x0800), length 100: (tos 0x0, ttl 64, id 39265, offset 0, flags [DF], proto ICMP (1), length 84)
    10.1.217.230 > 193.99.144.80: ICMP echo request, id 11935, seq 4, length 64

Instead of the outgoing ICMP pakets there are suddenly ARP requests, which I don't understand. I am not an expert in networking at all, so it might be only a symptom of a totally different issue. What I can say for sure is, that it only happens when mwan3 is started; It happens consistently. Everytime I start pinging with mwan3 started, lan4 is not receiving the first four pings, but instead only shows ARP requests.

Now, when mwan3 is instructed to issue the ping, it works perfectly fine for both interfaces (both when mwan3 is started and stopped):

root@openwrt.dmz.int.example.com:~# mwan3 use wanb ping heise.de -4 -c 5
Running 'ping heise.de -4 -c 5' with DEVICE=lan4 SRCIP=10.1.217.230 FWMARK=0x3f00 FAMILY=ipv4
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: seq=0 ttl=245 time=17.473 ms
64 bytes from 193.99.144.80: seq=1 ttl=245 time=13.638 ms
64 bytes from 193.99.144.80: seq=2 ttl=245 time=11.961 ms
64 bytes from 193.99.144.80: seq=3 ttl=245 time=13.294 ms
64 bytes from 193.99.144.80: seq=4 ttl=245 time=10.490 ms

--- heise.de ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 10.490/13.371/17.473 ms
root@openwrt.dmz.int.example.com:~# mwan3 use wan ping heise.de -4 -c 5
Running 'ping heise.de -4 -c 5' with DEVICE=pppoe-wan SRCIP=10.2.174.182 FWMARK=0x3f00 FAMILY=ipv4
PING heise.de (193.99.144.80): 56 data bytes
64 bytes from 193.99.144.80: seq=0 ttl=248 time=15.124 ms
64 bytes from 193.99.144.80: seq=1 ttl=248 time=14.895 ms
64 bytes from 193.99.144.80: seq=2 ttl=248 time=15.044 ms
64 bytes from 193.99.144.80: seq=3 ttl=248 time=14.717 ms
64 bytes from 193.99.144.80: seq=4 ttl=248 time=14.880 ms

--- heise.de ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 14.717/14.932/15.124 ms
root@openwrt.dmz.int.example.com:~#

This symptom alone is not really an issue (at least for me), but whenever wanA goes down, there is no failover to wanB. Of course, when I stop mwan3 (when wanA is gone), the only possible way for the pakets to get routed is through wanB. This then obviously works. That, however, defeats the complete purpose of mwan3 to me and is not want I am looking for.
Does anybody have an idea how to troubleshoot this further?
I can say with confidence it worked with the exact same configuration on 19.07 (setup and configuration of my OpenWrt devices is automated, so I am 100% confident it is the same configuration)

I am grateful for any pointers, thoughts or hints.

Thanks!

Just in case it helps, below the output of mwan3 internal:

root@openwrt.dmz.int.example.com:~# mwan3 internal
Software-Version
-------------------------------------------------
OpenWrt - 21.02.5

Output of "ip -4 a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: lan4@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc cake state UP group default qlen 1000
    inet 10.1.217.230/29 brd 10.2.217.231 scope global lan4
       valid_lft forever preferred_lft forever
11: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 172.31.2.1/24 brd 172.31.2.255 scope global br-lan
       valid_lft forever preferred_lft forever
35: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc cake state UNKNOWN group default qlen 3
    inet 10.2.174.182 peer 10.3.244.18/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever

Output of "ip -4 route show"
-------------------------------------------------
default via 10.3.244.18 dev pppoe-wan proto static metric 10 
default via 10.2.217.229 dev lan4 proto static metric 20 
10.3.244.18 dev pppoe-wan proto kernel scope link src 10.2.174.182 
10.2.217.224/29 dev lan4 proto static scope link metric 20 
172.31.2.0/24 dev br-lan proto kernel scope link src 172.31.2.1 
172.31.3.0/24 via 172.31.2.3 dev br-lan proto static 
172.31.4.0/24 via 172.31.2.4 dev br-lan proto static 
172.31.5.0/24 via 172.31.2.5 dev br-lan proto static 
172.31.6.0/24 via 172.31.2.6 dev br-lan proto static 
172.31.10.0/24 via 172.31.2.3 dev br-lan proto static 
172.31.100.0/24 via 172.31.2.3 dev br-lan proto static 

Output of "ip -4 rule show"
-------------------------------------------------
0:      from all lookup local
1001:   from all iif pppoe-wan lookup 1
1002:   from all iif lan4 lookup 2
2001:   from all fwmark 0x100/0x3f00 lookup 1
2002:   from all fwmark 0x200/0x3f00 lookup 2
2061:   from all fwmark 0x3d00/0x3f00 blackhole
2062:   from all fwmark 0x3e00/0x3f00 unreachable
3001:   from all fwmark 0x100/0x3f00 unreachable
3002:   from all fwmark 0x200/0x3f00 unreachable
18381:  from all fwmark 0x3ffd/0x3fff blackhole
18382:  from all fwmark 0x3ffe/0x3fff unreachable
32766:  from all lookup main
32767:  from all lookup default

Output of "ip -4 route list table 1-250"
-------------------------------------------------
Routing table 1:
default via 10.3.244.18 dev pppoe-wan proto static metric 10 
10.3.244.18 dev pppoe-wan proto kernel scope link src 10.2.174.182 
172.31.2.0/24 dev br-lan proto kernel scope link src 172.31.2.1 
172.31.3.0/24 via 172.31.2.3 dev br-lan proto static 
172.31.4.0/24 via 172.31.2.4 dev br-lan proto static 
172.31.5.0/24 via 172.31.2.5 dev br-lan proto static 
172.31.6.0/24 via 172.31.2.6 dev br-lan proto static 
172.31.10.0/24 via 172.31.2.3 dev br-lan proto static 
172.31.100.0/24 via 172.31.2.3 dev br-lan proto static 

Routing table 2:
default via 10.2.217.229 dev lan4 proto static metric 20 
10.2.217.224/29 dev lan4 proto static scope link metric 20 
172.31.2.0/24 dev br-lan proto kernel scope link src 172.31.2.1 
172.31.3.0/24 via 172.31.2.3 dev br-lan proto static 
172.31.4.0/24 via 172.31.2.4 dev br-lan proto static 
172.31.5.0/24 via 172.31.2.5 dev br-lan proto static 
172.31.6.0/24 via 172.31.2.6 dev br-lan proto static 
172.31.10.0/24 via 172.31.2.3 dev br-lan proto static 
172.31.100.0/24 via 172.31.2.3 dev br-lan proto static 

Output of "iptables -t mangle -w -L -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 72079 packets, 59M bytes)
 pkts bytes target     prot opt in     out     source               destination         
72111   59M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 1888 packets, 517K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 70181 packets, 59M bytes)
 pkts bytes target     prot opt in     out     source               destination         
16964 1015K TCPMSS     tcp  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 TCPMSS clamp to PMTU
 8932  514K TCPMSS     tcp  --  *      lan4    0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 1894 packets, 653K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1899  654K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 72067 packets, 59M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
53827   55M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
73523   60M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 CONNMARK restore mask 0x3f00
 1995  191K mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1644  171K mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1527  146K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
74010   60M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
71262   59M mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  146 12939 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_iface_in_wanb (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  lan4   *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  205  7952 MARK       all  --  lan4   *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb */ MARK xset 0x200/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1974  181K mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1828  168K mwan3_iface_in_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_balanced (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 statistic mode random probability 0.50000000000 /* wanb 2 4 */ MARK xset 0x200/0x3f00
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 2 2 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   36  2844 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_wanb (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 1271  113K MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wanb_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   36  2844 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb 2 2 */ MARK xset 0x200/0x3f00

Chain mwan3_policy_wanb_wan (7 references)
 pkts bytes target     prot opt in     out     source               destination         
  163 17580 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wanb 2 2 */ MARK xset 0x200/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   10   873 mwan3_policy_wanb_wan  all  --  *      *       172.31.4.9           0.0.0.0/0            mark match 0x0/0x3f00
    0     0 mwan3_policy_wanb_wan  all  --  *      *       172.31.4.100         0.0.0.0/0            mark match 0x0/0x3f00
  111 14187 mwan3_policy_wanb_wan  all  --  *      *       172.31.4.20          0.0.0.0/0            mark match 0x0/0x3f00
    0     0 mwan3_policy_wanb_wan  all  --  *      *       172.31.3.15          0.0.0.0/0            mark match 0x0/0x3f00
    0     0 mwan3_policy_wanb_wan  all  --  *      *       172.31.4.101         0.0.0.0/0            mark match 0x0/0x3f00
    0     0 mwan3_policy_wanb_wan  all  --  *      *       172.31.3.3           0.0.0.0/0            mark match 0x0/0x3f00
   42  2520 mwan3_policy_wanb_wan  all  --  *      *       172.31.3.33          0.0.0.0/0            mark match 0x0/0x3f00
   36  2844 mwan3_policy_wan_only  all  --  *      *       172.31.2.21          0.0.0.0/0            mark match 0x0/0x3f00
   36  2844 mwan3_policy_wanb_only  all  --  *      *       172.31.2.22          0.0.0.0/0            mark match 0x0/0x3f00
 1271  113K mwan3_policy_wan_wanb  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
root@openwrt.dmz.int.example.com:~#

For reference, I created a topic a while ago, where I observed that behaviour once, but back then I thought it was just a hick-up and that a reboot fixed it (which is not the case): Mwan3 - wanb stops working as soon as primary wan interfaces goes down - #2 by sscheib

Please also not that the IPs, MACs and hostnames have been "obfuscated" with "dummy" data. I took special care that all IP addresses and MACs have been consistently replaced to ensure the general data makes still sense, although those values have been replaced with "general values".

trendy · March 26, 2023, 3:03pm

Could you create a ticket to the developers and point to the topics? It seems that it is getting quite common.

sscheib · March 26, 2023, 4:13pm

You mean in GitHub? Sure.

sscheib · March 28, 2023, 10:02pm

Created https://github.com/openwrt/openwrt/issues/12278
Let's hope for the best