Multi WAN packet loss on 21.02 (all good on 19.07)

We observed similar scenario with mwan3 v2.10.13-1 on RB750Gr3 running on OpenWrt 21.02 across various locations. Common scenario was loss of at least 3 pings on secondary WAN interface before ping starts going.

Sample ping:

root@DEVICEID:~# ping -I lan3 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes
64 bytes from 9.9.9.9: seq=4 ttl=56 time=51.615 ms
64 bytes from 9.9.9.9: seq=5 ttl=56 time=51.191 ms
64 bytes from 9.9.9.9: seq=6 ttl=56 time=50.650 ms
64 bytes from 9.9.9.9: seq=7 ttl=56 time=50.852 ms
64 bytes from 9.9.9.9: seq=8 ttl=56 time=51.223 ms
64 bytes from 9.9.9.9: seq=9 ttl=56 time=50.569 ms
64 bytes from 9.9.9.9: seq=10 ttl=56 time=50.973 ms
^C
--- 9.9.9.9 ping statistics ---
11 packets transmitted, 7 packets received, 36% packet loss
round-trip min/avg/max = 50.569/51.010/51.615 ms

Observed tcpdump at the time:

10:29:15.621915 ARP, Request who-has 9.9.9.9 tell 192.168.1.33, length 28
10:29:16.624858 ARP, Request who-has 9.9.9.9 tell 192.168.1.33, length 28
10:29:17.648771 ARP, Request who-has 9.9.9.9 tell 192.168.1.33, length 28
10:29:19.636860 IP 192.168.1.33 > 9.9.9.9: ICMP echo request, id 30037, seq 4, length 64
10:29:19.688123 IP 9.9.9.9 > 192.168.1.33: ICMP echo reply, id 30037, seq 4, length 64
10:29:20.095243 IP 192.168.1.33 > 8.8.8.8: ICMP echo request, id 30067, seq 0, length 64
10:29:20.137324 IP 8.8.8.8 > 192.168.1.33: ICMP echo reply, id 30067, seq 0, length 64
10:29:20.640865 IP 192.168.1.33 > 9.9.9.9: ICMP echo request, id 30037, seq 5, length 64
10:29:20.691728 IP 9.9.9.9 > 192.168.1.33: ICMP echo reply, id 30037, seq 5, length 64
10:29:21.644869 IP 192.168.1.33 > 9.9.9.9: ICMP echo request, id 30037, seq 6, length 64
10:29:21.695165 IP 9.9.9.9 > 192.168.1.33: ICMP echo reply, id 30037, seq 6, length 64
10:29:22.636926 IP 192.168.1.33 > 101.53.132.190: ICMP echo request, id 30057, seq 5, length 64
10:29:22.648901 IP 192.168.1.33 > 9.9.9.9: ICMP echo request, id 30037, seq 7, length 64
10:29:22.699368 IP 9.9.9.9 > 192.168.1.33: ICMP echo reply, id 30037, seq 7, length 64
10:29:23.640935 IP 192.168.1.33 > 101.53.132.190: ICMP echo request, id 30057, seq 6, length 64
10:29:23.652895 IP 192.168.1.33 > 9.9.9.9: ICMP echo request, id 30037, seq 8, length 64
10:29:23.703731 IP 9.9.9.9 > 192.168.1.33: ICMP echo reply, id 30037, seq 8, length 64

As a temp fix, reverting mwan3 to v2.8.16 from OpenWrt 19.x resolved the issue.