Two WAN connections to individual LANs

In mwan3 status it shows correctly. But since you don't have another IPv6 connection you can remove all references to IPv6 from mwan3. Might be a problem that devices tend to prefer IPv6 over IPv4, so eventually you'll be only using wanA, not wanB.

Other policies are also using wanA :slight_smile:

This is the first connection, wanA. What do you get with the second?

For the iptables, have you done what it says in the wikl?

For routers using 22.03 or above the default firewall uses firewall4/nftables, the packages iptables-nft and ip6tables-nft are needed for mwan3 functionality to work. mwan3 does not currently natively support nftables, but does function with the iptables compatibility backend which will translate rules to be compatible with nftables.

It shows wan_6 correctly because the only way it works somehow with mwan3 is:

  1. configure wanA with option ipv6 auto
  2. rename all references in mwan3 from wan6 to wan_6
    Otherwise it wasn't possible for me to get a green status for the IPv6 connection.
    I will delete all WAN_6/WAN6 configurations from mwan3.

ofc I configured policies for lan specific routing:
I want:
all traffic from lan to wanA
all traffic from lan_oi to wanB
all traffic from guest to wanB
Since there is no IPv6 on wanB I configured a default for all IPv6 traffic -> wanA

That's my wanB. device internet.10 is wanB:

root@OpenWrt-EG:~# ping -c 4 -4 -I internet.10 www.google.com
PING www.google.com (142.251.209.132): 56 data bytes
64 bytes from 142.251.209.132: seq=0 ttl=113 time=8.673 ms
64 bytes from 142.251.209.132: seq=1 ttl=113 time=8.783 ms
64 bytes from 142.251.209.132: seq=2 ttl=113 time=8.507 ms
64 bytes from 142.251.209.132: seq=3 ttl=113 time=8.522 ms

--- www.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 8.507/8.621/8.783 ms
root@OpenWrt-EG:~#

internet.7 is my pppoe-wan connection and wanA:

root@OpenWrt-EG:~# ping -c 4 -4 -I pppoe-wan www.google.com
PING www.google.com (142.251.209.132): 56 data bytes
64 bytes from 142.251.209.132: seq=0 ttl=61 time=15.551 ms
64 bytes from 142.251.209.132: seq=1 ttl=61 time=15.443 ms
64 bytes from 142.251.209.132: seq=2 ttl=61 time=15.430 ms
64 bytes from 142.251.209.132: seq=3 ttl=61 time=15.675 ms

--- www.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 15.430/15.524/15.675 ms
root@OpenWrt-EG:~# ping -c 4 -6 -I pppoe-wan www.google.com
PING www.google.com (2a00:1450:4001:830::2004): 56 data bytes
64 bytes from 2a00:1450:4001:830::2004: seq=0 ttl=119 time=23.085 ms
64 bytes from 2a00:1450:4001:830::2004: seq=1 ttl=119 time=22.956 ms
64 bytes from 2a00:1450:4001:830::2004: seq=2 ttl=119 time=22.815 ms
64 bytes from 2a00:1450:4001:830::2004: seq=3 ttl=119 time=23.340 ms

--- www.google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 22.815/23.049/23.340 ms
root@OpenWrt-EG:~#

Yes, I installed before I began these two packages:


Did I have to configure something? It only says on IPv6 it needs additional configs.

They are almost fine, the only problem I saw in the firewall configuration is the missing forwarding from lan_oi zone to wan zone.

Could you also post the nft list ruleset to verify that the rules are applied properly?

Sorry for my late reply. I checked the fw rules and it seems to be ok?

I also removed all related Wan_6 stuff in the mwan manager.

Here is the output from nft list ruleset:

root@OpenWrt-EG:~# nft list ruleset
table inet fw4 {
        chain input {
                type filter hook input priority filter; policy drop;
                iifname "lo" accept comment "!fw4: Accept traffic from loopback"
                ct state established,related accept comment "!fw4: Allow inbound established and related flows"
                tcp flags syn / fin,syn,rst,ack jump syn_flood comment "!fw4: Rate limit TCP syn packets"
                udp dport 1234 counter packets 0 bytes 0 accept comment "!fw4: Allow-Wireguard-Inbound"
                iifname { "wg0", "br-lan.99" } jump input_lan comment "!fw4: Handle lan IPv4/IPv6 input traffic"
                iifname { "pppoe-wan", "internet.10" } jump input_wan comment "!fw4: Handle wan IPv4/IPv6 input traffic"                iifname "br-lan.20" jump input_GuestZone comment "!fw4: Handle GuestZone IPv4/IPv6 input traffic"
                iifname "br-lan.100" jump input_lan_oi comment "!fw4: Handle lan_oi IPv4/IPv6 input traffic"
                jump handle_reject
        }

        chain forward {
                type filter hook forward priority filter; policy drop;
                ct state established,related accept comment "!fw4: Allow forwarded established and related flows"
                iifname { "wg0", "br-lan.99" } jump forward_lan comment "!fw4: Handle lan IPv4/IPv6 forward traffic"
                iifname { "pppoe-wan", "internet.10" } jump forward_wan comment "!fw4: Handle wan IPv4/IPv6 forward traffic"
                iifname "br-lan.20" jump forward_GuestZone comment "!fw4: Handle GuestZone IPv4/IPv6 forward traffic"
                iifname "br-lan.100" jump forward_lan_oi comment "!fw4: Handle lan_oi IPv4/IPv6 forward traffic"
                jump handle_reject
        }

        chain output {
                type filter hook output priority filter; policy accept;
                oifname "lo" accept comment "!fw4: Accept traffic towards loopback"
                ct state established,related accept comment "!fw4: Allow outbound established and related flows"
                oifname { "wg0", "br-lan.99" } jump output_lan comment "!fw4: Handle lan IPv4/IPv6 output traffic"
                oifname { "pppoe-wan", "internet.10" } jump output_wan comment "!fw4: Handle wan IPv4/IPv6 output traffic"
                oifname "br-lan.20" jump output_GuestZone comment "!fw4: Handle GuestZone IPv4/IPv6 output traffic"
                oifname "br-lan.100" jump output_lan_oi comment "!fw4: Handle lan_oi IPv4/IPv6 output traffic"
        }

        chain prerouting {
                type filter hook prerouting priority filter; policy accept;
                iifname { "wg0", "br-lan.99" } jump helper_lan comment "!fw4: Handle lan IPv4/IPv6 helper assignment"
                iifname "br-lan.20" jump helper_GuestZone comment "!fw4: Handle GuestZone IPv4/IPv6 helper assignment"
                iifname "br-lan.100" jump helper_lan_oi comment "!fw4: Handle lan_oi IPv4/IPv6 helper assignment"
        }

        chain handle_reject {
                meta l4proto tcp reject with tcp reset comment "!fw4: Reject TCP traffic"
                reject comment "!fw4: Reject any other traffic"
        }

        chain syn_flood {
                limit rate 25/second burst 50 packets return comment "!fw4: Accept SYN packets below rate-limit"
                drop comment "!fw4: Drop excess packets"
        }

        chain input_lan {
                meta nfproto ipv4 meta l4proto igmp counter packets 50490 bytes 1617712 accept comment "!fw4: ubus:igmpproxy[instance1] rule 3"
                jump accept_from_lan
        }

        chain output_lan {
                jump accept_to_lan
        }

        chain forward_lan {
                jump accept_to_wan comment "!fw4: Accept lan to wan forwarding"
                jump accept_to_lan_oi comment "!fw4: Accept lan to lan_oi forwarding"
                jump accept_to_lan
        }

        chain helper_lan {
        }

        chain accept_from_lan {
                iifname { "wg0", "br-lan.99" } counter packets 141532 bytes 17241604 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
        }

        chain accept_to_lan {
                oifname { "wg0", "br-lan.99" } counter packets 199586 bytes 10735721 accept comment "!fw4: accept lan IPv4/IPv6 traffic"
        }

        chain drop_to_lan {
                oifname { "wg0", "br-lan.99" } counter packets 0 bytes 0 drop comment "!fw4: drop lan IPv4/IPv6 traffic"        }

        chain input_wan {
                meta nfproto ipv4 meta l4proto igmp counter packets 3351 bytes 120636 accept comment "!fw4: ubus:igmpproxy[instance1] rule 0"
                meta nfproto ipv4 udp dport 68 counter packets 1 bytes 28 accept comment "!fw4: Allow-DHCP-Renew"
                icmp type echo-request counter packets 3127 bytes 179151 accept comment "!fw4: Allow-Ping"
                meta nfproto ipv4 meta l4proto igmp counter packets 0 bytes 0 accept comment "!fw4: Allow-IGMP"
                meta nfproto ipv6 udp dport 546 counter packets 4 bytes 708 accept comment "!fw4: Allow-DHCPv6"
                ip6 saddr fe80::/10 icmpv6 type . icmpv6 code { mld-listener-query . no-route, mld-listener-report . no-route, mld-listener-done . no-route, mld2-listener-report . no-route } counter packets 0 bytes 0 accept comment "!fw4: Allow-MLD"
                icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply, nd-router-solicit, nd-router-advert } limit rate 1000/second counter packets 339 bytes 45724 accept comment "!fw4: Allow-ICMPv6-Input"
                icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, nd-neighbor-solicit . no-route, nd-neighbor-advert . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Input"
                meta nfproto ipv4 meta l4proto udp counter packets 4312 bytes 460063 accept comment "!fw4: Allow-UPD-IPTV"
                jump reject_from_wan
        }

        chain output_wan {
                jump accept_to_wan
        }

        chain forward_wan {
                meta l4proto udp ip daddr 239.255.255.250 counter packets 0 bytes 0 jump drop_to_lan comment "!fw4: ubus:igmpproxy[instance1] rule 1"
                meta l4proto udp ip daddr 224.0.0.0/4 counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: ubus:igmpproxy[instance1] rule 2"
                icmpv6 type { destination-unreachable, time-exceeded, echo-request, echo-reply } limit rate 1000/second counter packets 261 bytes 15608 accept comment "!fw4: Allow-ICMPv6-Forward"
                icmpv6 type . icmpv6 code { packet-too-big . no-route, parameter-problem . no-route, parameter-problem . admin-prohibited } limit rate 1000/second counter packets 0 bytes 0 accept comment "!fw4: Allow-ICMPv6-Forward"
                meta l4proto esp counter packets 0 bytes 0 jump accept_to_lan comment "!fw4: Allow-IPSec-ESP"
                udp dport 500 counter packets 15 bytes 3600 jump accept_to_lan comment "!fw4: Allow-ISAKMP"
                jump reject_to_wan
        }

        chain accept_to_wan {
                meta nfproto ipv4 oifname { "pppoe-wan", "internet.10" } ct state invalid counter packets 4797 bytes 217852 drop comment "!fw4: Prevent NAT leakage"
                oifname { "pppoe-wan", "internet.10" } counter packets 258568 bytes 38117570 accept comment "!fw4: accept wan IPv4/IPv6 traffic"
        }

        chain reject_from_wan {
                iifname { "pppoe-wan", "internet.10" } counter packets 60507 bytes 6545263 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
        }

        chain reject_to_wan {
                oifname { "pppoe-wan", "internet.10" } counter packets 1106 bytes 69622 jump handle_reject comment "!fw4: reject wan IPv4/IPv6 traffic"
        }

        chain input_GuestZone {
                tcp dport { 53, 67, 68 } counter packets 0 bytes 0 accept comment "!fw4: Allow-Guest-DHCP-DNS"
                udp dport { 53, 67, 68 } counter packets 1107 bytes 84736 accept comment "!fw4: Allow-Guest-DHCP-DNS"
                jump reject_from_GuestZone
        }

        chain output_GuestZone {
                jump accept_to_GuestZone
        }

        chain forward_GuestZone {
                jump accept_to_wan comment "!fw4: Accept GuestZone to wan forwarding"
                jump reject_to_GuestZone
        }

        chain helper_GuestZone {
        }

        chain accept_to_GuestZone {
                oifname "br-lan.20" counter packets 11 bytes 2796 accept comment "!fw4: accept GuestZone IPv4/IPv6 traffic"
        }

        chain reject_from_GuestZone {
                iifname "br-lan.20" counter packets 8 bytes 468 jump handle_reject comment "!fw4: reject GuestZone IPv4/IPv6 traffic"
        }

        chain reject_to_GuestZone {
                oifname "br-lan.20" counter packets 0 bytes 0 jump handle_reject comment "!fw4: reject GuestZone IPv4/IPv6 traffic"
        }

        chain input_lan_oi {
                jump accept_from_lan_oi
        }

        chain output_lan_oi {
                jump accept_to_lan_oi
        }

        chain forward_lan_oi {
                jump accept_to_lan comment "!fw4: Accept lan_oi to lan forwarding"
                jump accept_to_wan comment "!fw4: Accept lan_oi to wan forwarding"
                jump accept_to_lan_oi
        }

        chain helper_lan_oi {
        }

        chain accept_from_lan_oi {
                iifname "br-lan.100" counter packets 0 bytes 0 accept comment "!fw4: accept lan_oi IPv4/IPv6 traffic"
        }

        chain accept_to_lan_oi {
                oifname "br-lan.100" counter packets 0 bytes 0 accept comment "!fw4: accept lan_oi IPv4/IPv6 traffic"
        }

        chain input_wan_oi {
        }

        chain output_wan_oi {
        }

        chain forward_wan_oi {
        }

        chain accept_to_wan_oi {
        }

        chain reject_from_wan_oi {
        }

        chain reject_to_wan_oi {
        }

        chain dstnat {
                type nat hook prerouting priority dstnat; policy accept;
        }

        chain srcnat {
                type nat hook postrouting priority srcnat; policy accept;
                oifname { "pppoe-wan", "internet.10" } jump srcnat_wan comment "!fw4: Handle wan IPv4/IPv6 srcnat traffic"
        }

        chain srcnat_wan {
                meta nfproto ipv4 masquerade comment "!fw4: Masquerade IPv4 wan traffic"
        }

        chain srcnat_wan_oi {
        }

        chain raw_prerouting {
                type filter hook prerouting priority raw; policy accept;
        }

        chain raw_output {
                type filter hook output priority raw; policy accept;
        }

        chain mangle_prerouting {
                type filter hook prerouting priority mangle; policy accept;
        }

        chain mangle_postrouting {
                type filter hook postrouting priority mangle; policy accept;
        }

        chain mangle_input {
                type filter hook input priority mangle; policy accept;
        }

        chain mangle_output {
                type route hook output priority mangle; policy accept;
        }

        chain mangle_forward {
                type filter hook forward priority mangle; policy accept;
                iifname { "pppoe-wan", "internet.10" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 ingress MTU fixing"
                oifname { "pppoe-wan", "internet.10" } tcp flags syn tcp option maxseg size set rt mtu comment "!fw4: Zone wan IPv4/IPv6 egress MTU fixing"
        }
}
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                counter packets 29221 bytes 45661338 jump mwan3_hook
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
                counter packets 3084 bytes 249188 jump mwan3_hook
        }

        chain mwan3_ifaces_in {
                meta mark & 0x00003f00 == 0x00000000 counter packets 7929 bytes 8674745 jump mwan3_iface_in_wan
                meta mark & 0x00003f00 == 0x00000000 counter packets 7892 bytes 8672740 jump mwan3_iface_in_wanb
        }

        chain mwan3_custom_ipv4 {
                xt match "set" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_connected_ipv4 {
                xt match "set" counter packets 7680 bytes 14660993 xt target "MARK"
        }

        chain mwan3_dynamic_ipv4 {
                xt match "set" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_rules {
                ip saddr 10.20.30.0/24 meta mark & 0x00003f00 == 0x00000000 counter packets 4512 bytes 4552416 jump mwan3_policy_wanb_only
                ip saddr 172.16.1.0/24 meta mark & 0x00003f00 == 0x00000000 counter packets 0 bytes 0 jump mwan3_policy_wanb_only
                ip saddr 192.168.1.0/24 meta mark & 0x00003f00 == 0x00000000 counter packets 132 bytes 24652 jump mwan3_policy_wan_only
        }

        chain mwan3_hook {
                meta mark & 0x00003f00 == 0x00000000 counter packets 32184 bytes 45903130 xt target "CONNMARK"
                meta mark & 0x00003f00 == 0x00000000 counter packets 7930 bytes 8674777 jump mwan3_ifaces_in
                meta mark & 0x00003f00 == 0x00000000 counter packets 5900 bytes 4669326 jump mwan3_custom_ipv4
                meta mark & 0x00003f00 == 0x00000000 counter packets 5900 bytes 4669326 jump mwan3_connected_ipv4
                meta mark & 0x00003f00 == 0x00000000 counter packets 4884 bytes 4595562 jump mwan3_dynamic_ipv4
                meta mark & 0x00003f00 == 0x00000000 counter packets 4884 bytes 4595562 jump mwan3_rules
                counter packets 32305 bytes 45910526 xt target "CONNMARK"
                meta mark & 0x00003f00 != 0x00003f00 counter packets 15109 bytes 20182396 jump mwan3_custom_ipv4
                meta mark & 0x00003f00 != 0x00003f00 counter packets 15109 bytes 20182396 jump mwan3_connected_ipv4
                meta mark & 0x00003f00 != 0x00003f00 counter packets 8445 bytes 5595167 jump mwan3_dynamic_ipv4
        }

        chain mwan3_iface_in_wan {
                iifname "pppoe-wan" xt match "set" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                iifname "pppoe-wan" xt match "set" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                iifname "pppoe-wan" xt match "set" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                iifname "pppoe-wan" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 37 bytes 2005 xt target "MARK"
        }

        chain mwan3_iface_in_wanb {
                iifname "internet.10" xt match "set" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                iifname "internet.10" xt match "set" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                iifname "internet.10" xt match "set" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                iifname "internet.10" meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 1993 bytes 4003446 xt target "MARK"
        }

        chain mwan3_policy_wan_only {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 132 bytes 24652 xt target "MARK"        }

        chain mwan3_policy_wanb_only {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 4512 bytes 4552416 xt target "MARK"
        }

        chain mwan3_policy_balanced {
                meta mark & 0x00003f00 == 0x00000000 xt match "statistic" xt match "comment" counter packets 0 bytes 0 xt target "MARK"
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_policy_wan_wanb {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_policy_wanb_wan {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }
}
# Warning: table ip6 mangle is managed by iptables-nft, do not touch!
table ip6 mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                counter packets 20961 bytes 22623799 jump mwan3_hook
        }

        chain OUTPUT {
                type route hook output priority mangle; policy accept;
                counter packets 276 bytes 45029 jump mwan3_hook
        }

        chain mwan3_ifaces_in {
        }

        chain mwan3_custom_ipv6 {
                xt match "set" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_connected_ipv6 {
                xt match "set" counter packets 140 bytes 31027 xt target "MARK"
        }

        chain mwan3_dynamic_ipv6 {
                xt match "set" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_rules {
        }

        chain mwan3_hook {
                meta l4proto ipv6-icmp xt match "icmp6" counter packets 6 bytes 328 return
                meta l4proto ipv6-icmp xt match "icmp6" counter packets 2 bytes 352 return
                meta l4proto ipv6-icmp xt match "icmp6" counter packets 115 bytes 8256 return
                meta l4proto ipv6-icmp xt match "icmp6" counter packets 36 bytes 2384 return
                meta l4proto ipv6-icmp xt match "icmp6" counter packets 0 bytes 0 return
                meta mark & 0x00003f00 == 0x00000000 counter packets 21078 bytes 22657508 xt target "CONNMARK"
                meta mark & 0x00003f00 == 0x00000000 counter packets 184 bytes 31637 jump mwan3_ifaces_in
                meta mark & 0x00003f00 == 0x00000000 counter packets 184 bytes 31637 jump mwan3_custom_ipv6
                meta mark & 0x00003f00 == 0x00000000 counter packets 184 bytes 31637 jump mwan3_connected_ipv6
                meta mark & 0x00003f00 == 0x00000000 counter packets 144 bytes 27038 jump mwan3_dynamic_ipv6
                meta mark & 0x00003f00 == 0x00000000 counter packets 144 bytes 27038 jump mwan3_rules
                counter packets 21078 bytes 22657508 xt target "CONNMARK"
                meta mark & 0x00003f00 != 0x00003f00 counter packets 256 bytes 62412 jump mwan3_custom_ipv6
                meta mark & 0x00003f00 != 0x00003f00 counter packets 256 bytes 62412 jump mwan3_connected_ipv6
                meta mark & 0x00003f00 != 0x00003f00 counter packets 156 bytes 35984 jump mwan3_dynamic_ipv6
        }

        chain mwan3_policy_wan_only {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_policy_wanb_only {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_policy_balanced {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_policy_wan_wanb {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }

        chain mwan3_policy_wanb_wan {
                meta mark & 0x00003f00 == 0x00000000 xt match "comment" counter packets 0 bytes 0 xt target "MARK"
        }
}
root@OpenWrt-EG:~#

I'm just before giving up... also a strange behavior is this:

root@OpenWrt-EG:~# ping -4 -I internet.10 www.google.de
PING www.google.de (142.250.185.195): 56 data bytes
64 bytes from 142.250.185.195: seq=4 ttl=112 time=20.017 ms
64 bytes from 142.250.185.195: seq=5 ttl=112 time=19.980 ms
64 bytes from 142.250.185.195: seq=6 ttl=112 time=19.869 ms
^C
--- www.google.de ping statistics ---
7 packets transmitted, 3 packets received, 57% packet loss
round-trip min/avg/max = 19.869/19.955/20.017 ms
root@OpenWrt-EG:~# ping -4 -I pppoe-wan www.google.de
PING www.google.de (142.250.185.195): 56 data bytes
64 bytes from 142.250.185.195: seq=0 ttl=61 time=25.005 ms
64 bytes from 142.250.185.195: seq=1 ttl=61 time=25.159 ms
64 bytes from 142.250.185.195: seq=2 ttl=61 time=25.006 ms
^C
--- www.google.de ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 25.005/25.056/25.159 ms
root@OpenWrt-EG:~# ping -4 -I internet.10 www.google.de
PING www.google.de (142.250.185.195): 56 data bytes
64 bytes from 142.250.185.195: seq=4 ttl=112 time=20.116 ms
64 bytes from 142.250.185.195: seq=5 ttl=112 time=20.012 ms
64 bytes from 142.250.185.195: seq=6 ttl=112 time=20.237 ms
^C
--- www.google.de ping statistics ---
7 packets transmitted, 3 packets received, 57% packet loss
round-trip min/avg/max = 20.012/20.121/20.237 ms
root@OpenWrt-EG:~# ping -4 -I internet.10 www.google.com
PING www.google.com (172.217.16.68): 56 data bytes
64 bytes from 172.217.16.68: seq=4 ttl=113 time=8.614 ms
64 bytes from 172.217.16.68: seq=5 ttl=113 time=8.431 ms
64 bytes from 172.217.16.68: seq=6 ttl=113 time=8.486 ms
^C
--- www.google.com ping statistics ---
7 packets transmitted, 3 packets received, 57% packet loss
round-trip min/avg/max = 8.431/8.510/8.614 ms
root@OpenWrt-EG:~#

First three packages getting dropped on wanb?!

When I disable the mwan3 service and try again, it works:

root@OpenWrt-EG:~# ping -4 -I internet.10 www.google.com
PING www.google.com (216.58.206.36): 56 data bytes
64 bytes from 216.58.206.36: seq=0 ttl=53 time=19.935 ms
64 bytes from 216.58.206.36: seq=1 ttl=53 time=19.928 ms
64 bytes from 216.58.206.36: seq=2 ttl=53 time=19.924 ms
64 bytes from 216.58.206.36: seq=3 ttl=53 time=19.775 ms
64 bytes from 216.58.206.36: seq=4 ttl=53 time=19.854 ms
^C
--- www.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 19.775/19.883/19.935 ms
root@OpenWrt-EG:~# ping -4 -I internet.10 www.google.com
PING www.google.com (216.58.206.36): 56 data bytes
64 bytes from 216.58.206.36: seq=0 ttl=53 time=20.070 ms
64 bytes from 216.58.206.36: seq=1 ttl=53 time=19.996 ms
64 bytes from 216.58.206.36: seq=2 ttl=53 time=19.944 ms
64 bytes from 216.58.206.36: seq=3 ttl=53 time=19.906 ms
64 bytes from 216.58.206.36: seq=4 ttl=53 time=19.895 ms
^C
--- www.google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 19.895/19.962/20.070 ms
root@OpenWrt-EG:~# ping -4 -I internet.10 www.google.de
PING www.google.de (142.250.181.195): 56 data bytes
64 bytes from 142.250.181.195: seq=0 ttl=113 time=8.851 ms
64 bytes from 142.250.181.195: seq=1 ttl=113 time=8.557 ms
64 bytes from 142.250.181.195: seq=2 ttl=113 time=8.762 ms
64 bytes from 142.250.181.195: seq=3 ttl=113 time=8.670 ms
^C
--- www.google.de ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 8.557/8.710/8.851 ms
root@OpenWrt-EG:~#

I'm pretty sure I have seen again this issue with the 3 lost pings somewhere here.
Maybe @pavelgl or @mk24 or @lleachii remember why is that happening?

This happens when based on the fwmark (depending on the active policy), an ip rule directs the packet to the wrong routing table.

Here is an example where the active policy is wanb_to_wan and the test is run on the (inactive) wan interface.

root@MikroTik:~# ping -I wan 8.8.8.8 -c 6
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=4 ttl=57 time=1.288 ms
64 bytes from 8.8.8.8: seq=5 ttl=57 time=1.276 ms

--- 8.8.8.8 ping statistics ---
6 packets transmitted, 2 packets received, 66% packet loss
round-trip min/avg/max = 1.276/1.282/1.288 ms

Pretty much the same results.
Running tcpdump at the same time shows some strange ARP requests.

root@MikroTik:~# tcpdump -nnti any arp and host 8.8.8.8
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
wan   Out ARP, Request who-has 8.8.8.8 tell 37.157.X.X, length 28
wan   Out ARP, Request who-has 8.8.8.8 tell 37.157.X.X, length 28
wan   Out ARP, Request who-has 8.8.8.8 tell 37.157.X.X, length 28

That's because ping is forced to use the wan interface, but the packet is directed to the wrong (wanb) routing table, where the wan network is not listed and there is no appropriate default gateway.

traceroute always returns Host unreachable

root@MikroTik:~# traceroute -i wan 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 46 byte packets
 1  37-157-X-X.net1.bg (37.157.X.X)  3143.876 ms !H  3117.337 ms !H  3119.533 ms !H

There is no problem when running the tests using the currently active (based on the wanb_to_wan policy) wanb interface.

If the policy is balanced, there is a chance that packets will be forwarded to the correct routing table.

So these kinds of tests should be run when the service is stopped, and that's probably the meaning of "before proceeding" in the guide.

2 Likes

One potential root cause for the problem seems to me to be the lack of the proper markings in nftables.

ip rule is expecting these marks to classify the traffic accordingly, but they are nowhere to be found in the nftables output.
I have no better idea but to tag here the devs (@feckert @aaronjg ) to have a look, but it would be best to open a ticket for that.

Besides that can you spot a problem in my configuration or should it work normally?

I will search the GitHub repo for mwan3 and post it there as an issue.

No other problem that I can see. I hope you have fixed already the previously mentioned mistakes. Best of luck with the problem!

So I think it finally works.
I installed the script reffered in the doc: https://github.com/openwrt/packages/issues/22474

I also changed every policy in mwan3 to use "default" as the last resort:

Now I only have issues with my IPTV (igmpproxy) on lan. I will test further.
EDIT: It seems a restart on the IPTV client fixed the problem. :slight_smile:

1 Like