Mwan3 feedback on IPv6 and missing features

Hello openWrt community,

I am the maintainer of the mwan3 package https://github.com/openwrt/packages/tree/master/net/mwan3 .

I want to know the experience if anyone is using mwan3 in a IPv6 only and in a IPv6/IPv4 mixed environment. What are the Problems to get mwan3 working on IPv6. Also I want to get an overview who is using mwan3 and what are the problems and missing feature are.

Thanks

Hi,

I'm trying to get mwan3 working with two dual stack internet connections. IPv4 seems to work but my wanb6 interface is always offline. I get IPv6 addresses from both connection on my client, is this correct? How does the client decide which address to use?

Ping6 for wan6 (this one works):

# ping6 -c 1 -I eth0.1 www.google.com
PING www.google.com (2a00:1450:4001:816::2004): 56 data bytes
ping6: sendto: Permission denied

Ping6 for wanb6 (this one doesn't work):

# ping6 -c 1 -I eth0.2 www.google.com
PING www.google.com (2a00:1450:4001:816::2004): 56 data bytes
64 bytes from 2a00:1450:4001:816::2004: seq=0 ttl=57 time=33.391 ms

--- www.google.com ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 33.391/33.391/33.391 ms
# mwan3 status
Interface status:
 interface wan is online and tracking is active
 interface wan6 is online and tracking is active
 interface wanb is online and tracking is active
 interface wanb6 is offline and tracking is active

Current ipv4 policies:
balanced:
 wanb (40%)
 wan (60%)

wan_only:
 wan (100%)

wan_wanb:
 wan (100%)

wanb_only:
 wanb (100%)

wanb_wan:
 wanb (100%)


Current ipv6 policies:
balanced:
 wan6 (100%)

wan_only:
 wan6 (100%)

wan_wanb:
 wan6 (100%)

wanb_only:
 unreachable

wanb_wan:
 wan6 (100%)


Directly connected ipv4 networks:
 224.0.0.0/3
 10.0.0.255
 127.0.0.0
 127.255.255.255
 10.0.0.0
 10.0.0.1
 192.168.179.0/24
 192.168.178.184
 192.168.178.255
 192.168.179.0
 192.168.179.255
 127.0.0.0/8
 10.0.0.0/24
 127.0.0.1
 192.168.179.25
 192.168.178.0
 192.168.178.0/24

Directly connected ipv6 networks:
 prefix:of:wanb6:address::/64
 prefix:of:wan6:address::/56
 prefix:of:wanb6:address::/56
 fe80::/64
 prefix:of:wanb6:pd::/64
 fd8b:1382:bd9::/64
 pfrefix:of:wan6:pd::/64
 prefix:of:wan6:address::/64

Active ipv4 user rules:
    2   100 S https  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport sports 0:65535 multiport dports 443 
    0     0 - balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Active ipv6 user rules:
    0     0 S https  tcp      *      *       ::/0                 ::/0                 multiport sports 0:65535 multiport dports 443 
    0     0 - balanced  all      *      *       ::/0                 ::/0                 

Running openwrt 18.06.2 on x86_64. Didn't touch the default configuration except enabling the interfaces. Both interfaces have an IPv6 address in a /64 and a deligated prefix of /58.

I would appreciate your help!

EDIT: Just installed the latest snapshot and switched wan and wanb. Now wan6 doesn't work, so it seems to be related to the router which connects to the ISP. Those are both FRITZ!boxes (different models) but the relevant configuration should be the same.

Thinking about the issue with multiple wans and ipv6. Based on which IP address is selected by the originator of the traffic, the traffic can only traverse the WAN associated with that prefix. There's not much that mwan3 can do other than trying to get fancy and use NPT6 in some tricky way that would probably cause fragility.

Anyone else have thoughts?

This situation would usually be solved by the ISP (both) accepting packets from both prefixes, but this is generally only an option for (higher end) business contracts - for consumer contracts the ISP won't even consider this request.

1 Like

Those were my thoughts too.

  • On ipv4 we have a dedicated network address which will get NATed anyway on the wan interfaces.

  • On ipv6 we have no dedicated network address so we have to use NAT66 or NPT6 on the wan interfaces.

Since much traffic comes from web access another option is to use a proxy and tell the proxy to use different source IPs for different connections explicitly

I've been trying to use MWAN3 today and I have the same problem, IPv4 works great but IPv6 doesn't and explained above by @dlakelan.
The NAT idea I believe would defeat the purpose, I would actually want to use that mainly for failover.

@feckert,
How do you handle interfaces that have both IPv4 and IPv6? In MWAN3 I can only select IPv4 or IPv6 to use with a single interface. Selecting one of both enables the selected protocol and it works. But is it possible to use IPv4 and IPv6 on a single interface somehow?

In my case I have 3 wireguard interfaces where all 3 have both IPv4 and IPv6 addresses, I'm using failover to switch between them.

I recently managed to make my LTE dongle work, so I gave mwan3 a try. As neither of my ISPs support IPv6 so far, I am using HEnet tunnel for 6in4. However IPv6 doesn't work at all.
I tried to create a set of Interface-Member-Policy-Rule for v6, I tried without, I tried to use main routing table as policy for IPv6. No joy.

mwan3 status

Summary
Interface status:
 interface wan is online and tracking is active
 interface LTE is online and tracking is active
 interface HEnet is offline and tracking is active

Current ipv4 policies:
HEnet_only:
 default

wan_LTE:
 wan (100%)

wan_only:
 wan (100%)


Current ipv6 policies:
HEnet_only:
 default

wan_LTE:
 default

wan_only:
 default


Directly connected ipv4 networks:
 127.0.0.1
 127.255.255.255
 172.17.17.0/24
 10.0.20.0
 XXX.XXX.XXX.XXX
 127.0.0.0/8
 10.0.20.0/30
 10.0.10.15
 224.0.0.0/3
 172.30.30.0
 100.117.168.212
 10.0.10.0
 10.0.1.0/24
 37.205.11.30
 127.0.0.0
 10.0.20.5
 10.0.10.0/28
 10.0.20.4/30
 172.17.17.255
 YYY.YYY.YYY.YYY
 216.66.86.122
 10.0.20.7
 10.0.2.1
 100.117.168.208/29
 10.0.10.1
 10.0.20.4
 10.0.3.0/24
 10.0.20.2
 172.30.30.0/24
 100.117.168.208
 100.117.168.215
 172.30.30.255
 172.17.17.1
 10.0.20.3
 10.0.20.8/30
 10.0.2.255
 172.30.30.1
 10.0.10.16/28
 10.0.2.0/24
 172.17.17.0
 10.0.20.10
 10.0.2.0
 10.0.20.14

Directly connected ipv6 networks:
 fd00:bbbb::/64
 2001:470:6e:765::/64
 fe80::/64
 fd00:bbbb:0:ffff::/64
 fe80::/10
 fd00:bbbb:0:bbbb::/64
 2001:470:::/64
 2001:470::/64
 fd00:cccc::/64
 2001:470:ffff::/64

Active ipv4 user rules:
  533 90136 - wan_LTE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            
    0     0 - HEnet_only  all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Active ipv6 user rules:
 1967  225K - wan_LTE  all      *      *       ::/0                 ::/0                 
    0     0 - HEnet_only  all      *      *       ::/0                 ::/0

Troubleshooting

Summary
Software-Version
-------------------------------------------------
OpenWrt - OpenWrt 18.06.4 r7808-ef686b7292
LuCI - git-19.241.48373-38f5f4b


Output of "ip a show"
-------------------------------------------------
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 10.0.2.1/32 scope global lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:11:22:33:44:57 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::215:6dff:fec3:30b7/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:11:22:33:44:57 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::15:6dff:fec3:30b7/64 scope link 
       valid_lft forever preferred_lft forever
4: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
    link/sit 0.0.0.0 brd 0.0.0.0
5: ifb0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc hfsc state UNKNOWN group default qlen 32
    link/ether 1e:21:42:85:2f:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1c21:42ff:fe85:2ffe/64 scope link 
       valid_lft forever preferred_lft forever
6: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether 9e:f4:f2:52:c3:2b brd ff:ff:ff:ff:ff:ff
14: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
    link/none 
    inet 10.0.20.2/30 brd 10.0.20.3 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::fa3:2163:60ab:f975/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
16: wwan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 00:1e:10:1f:00:00 brd ff:ff:ff:ff:ff:ff
    inet 100.117.168.212/29 brd 100.117.168.215 scope global wwan0
       valid_lft forever preferred_lft forever
    inet6 fe80::21e:10ff:fe1f:0/64 scope link 
       valid_lft forever preferred_lft forever
24: eth1.2@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:11:22:33:44:58 brd ff:ff:ff:ff:ff:ff
    inet 172.17.17.1/24 brd 172.17.17.255 scope global eth1.2
       valid_lft forever preferred_lft forever
    inet6 2001:470:59d2:bbbb:2d4f:671:df3b:58b4/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 2001:470:59d2:bbbb:e483:aadd:fd92:1271/64 scope global deprecated dynamic noprefixroute 
       valid_lft 5046sec preferred_lft 0sec
    inet6 fd00:bbbb::bbbb:9d59:8858:4c4c:f14d/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::15:6dff:fec3:30b8/64 scope link 
       valid_lft forever preferred_lft forever
25: eth1.3@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:11:22:33:44:59 brd ff:ff:ff:ff:ff:ff
    inet 172.30.30.1/24 brd 172.30.30.255 scope global eth1.3
       valid_lft forever preferred_lft forever
    inet6 2001:470:59d2:ffff:b808:e7ec:3a2e:48a3/64 scope global dadfailed tentative noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 2001:470:59d2:ffff:b2d5:fc6f:d0d9:c6c9/64 scope global dadfailed deprecated tentative dynamic noprefixroute 
       valid_lft 5046sec preferred_lft 0sec
    inet6 fd00:bbbb::ffff:1e80:7f1:4b32:3fce/64 scope global dadfailed tentative noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::15:6dff:fec3:30b9/64 scope link dadfailed tentative 
       valid_lft forever preferred_lft forever
26: eth1.1@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:11:22:33:44:57 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.1/24 brd 10.0.2.255 scope global eth1.1
       valid_lft forever preferred_lft forever
    inet6 2001:470:59d2::1/64 scope global dadfailed tentative noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fd00:bbbb::1/64 scope global dadfailed tentative noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::15:6dff:fec3:30b7/64 scope link dadfailed tentative 
       valid_lft forever preferred_lft forever
27: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1484 qdisc hfsc state UNKNOWN group default qlen 3
    link/ppp 
    inet YYY.YYY.YYY.YYY peer XXX.XXX.XXX.XXX/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever
    inet6 fe80::8875:3e1e:6934:5e65/10 scope link 
       valid_lft forever preferred_lft forever
28: elvetias: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1
    link/none 
    inet 10.0.20.5/30 brd 10.0.20.7 scope global elvetias
       valid_lft forever preferred_lft forever
    inet6 fd00:cccc::1/64 scope global 
       valid_lft forever preferred_lft forever
29: roadwarrior: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1
    link/none 
    inet 10.0.10.1/28 brd 10.0.10.15 scope global roadwarrior
       valid_lft forever preferred_lft forever
31: 6in4-HEnet@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1
    link/sit YYY.YYY.YYY.YYY peer 216.66.86.122
    inet6 2001:470::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5f50:d375/64 scope link 
       valid_lft forever preferred_lft forever


Output of "ip route show"
-------------------------------------------------
default via 10.0.2.1 dev lo 
default via XXX.XXX.XXX.XXX dev pppoe-wan proto static metric 10 
default via 100.117.168.209 dev wwan0 proto static src 100.117.168.212 metric 40 
10.0.1.0/24 via 10.0.20.1 dev tun0 proto zebra metric 20 
10.0.2.0/24 dev eth1.1 proto kernel scope link src 10.0.2.1 
10.0.3.0/24 via 10.0.10.3 dev roadwarrior proto zebra metric 20 
10.0.10.0/28 dev roadwarrior proto kernel scope link src 10.0.10.1 
10.0.10.16/28 via 10.0.20.1 dev tun0 proto zebra metric 20 
10.0.20.0/30 dev tun0 proto kernel scope link src 10.0.20.2 
10.0.20.2 via 10.0.20.1 dev tun0 proto zebra metric 20 
10.0.20.4/30 dev elvetias proto kernel scope link src 10.0.20.5 
10.0.20.8/30 via 10.0.20.1 dev tun0 proto zebra metric 20 
10.0.20.10 via 10.0.20.1 dev tun0 proto zebra metric 20 
10.0.20.14 via 10.0.20.1 dev tun0 proto zebra metric 20 
37.205.11.30 dev pppoe-wan proto static scope link metric 10 
XXX.XXX.XXX.XXX dev pppoe-wan proto kernel scope link src YYY.YYY.YYY.YYY 
100.117.168.208/29 dev wwan0 proto static scope link metric 40 
172.17.17.0/24 dev eth1.2 proto kernel scope link src 172.17.17.1 
172.30.30.0/24 dev eth1.3 proto kernel scope link src 172.30.30.1 
216.66.86.122 via XXX.XXX.XXX.XXX dev pppoe-wan proto static 
216.66.86.122 via 100.117.168.209 dev wwan0 proto static metric 40


Output of "ip -6 route show"
-------------------------------------------------
default from 2001:470::/64 dev 6in4-HEnet proto static metric 20 pref medium
default from 2001:470::/48 dev 6in4-HEnet proto static metric 20 pref medium
2001:470::/64 dev 6in4-HEnet proto static metric 20 pref medium
2001:470::/64 dev eth1.1 proto static metric 1024 pref medium
2001:470::/64 dev eth1.2 proto static metric 1024 pref medium
2001:470::/64 dev eth1.3 proto static metric 1024 pref medium
unreachable 2001:470::/48 dev lo proto static metric 2147483647 error -148 pref medium
fd00:bbbb::/64 dev eth1.1 proto static metric 1024 pref medium
fd00:bbbb:0:bbbb::/64 dev eth1.2 proto static metric 1024 pref medium
fd00:bbbb:0:ffff::/64 dev eth1.3 proto static metric 1024 pref medium
unreachable fd00:bbbb::/48 dev lo proto static metric 2147483647 error -148 pref medium
fd00:cccc::/64 dev elvetias proto kernel metric 256 pref medium
fe80::/64 dev ifb0 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev eth1.1 proto kernel metric 256 pref medium
fe80::/64 dev eth1.2 proto kernel metric 256 pref medium
fe80::/64 dev eth1.3 proto kernel metric 256 pref medium
fe80::/64 dev wwan0 proto kernel metric 256 pref medium
fe80::/64 dev 6in4-HEnet proto kernel metric 256 pref medium
fe80::/10 dev pppoe-wan metric 1 pref medium
fe80::/10 dev pppoe-wan proto kernel metric 256 pref medium


Output of "ip rule show"
-------------------------------------------------
0:	from all lookup local 
1001:	from all iif pppoe-wan lookup main 
1002:	from all iif wwan0 lookup main 
2001:	from all fwmark 0x100/0x3f00 lookup 1 
2002:	from all fwmark 0x200/0x3f00 lookup 2 
2061:	from all fwmark 0x3d00/0x3f00 blackhole
2062:	from all fwmark 0x3e00/0x3f00 unreachable
32766:	from all lookup main 
32767:	from all lookup default


Output of "ip -6 rule show"
-------------------------------------------------
0:      from all lookup local 
1003:   from all iif 6in4-HEnet lookup main 
2003:   from all fwmark 0x300/0x3f00 lookup 3 
2061:   from all fwmark 0x3d00/0x3f00 blackhole
2062:   from all fwmark 0x3e00/0x3f00 unreachable
32766:  from all lookup main 
4200000000:     from 2001:470:59d2::1/64 iif eth1.1 unreachable
4200000000:     from 2001:470:59d2:bbbb:2d4f:671:df3b:58b4/64 iif eth1.2 unreachable
4200000000:     from 2001:470:59d2:ffff:b808:e7ec:3a2e:48a3/64 iif eth1.3 unreachable
4200000001:     from all iif lo failed_policy
4200000014:     from all iif tun0 failed_policy
4200000016:     from all iif wwan0 failed_policy
4200000016:     from all iif wwan0 failed_policy
4200000024:     from all iif eth1.2 failed_policy
4200000025:     from all iif eth1.3 failed_policy
4200000026:     from all iif eth1.1 failed_policy
4200000027:     from all iif pppoe-wan failed_policy
4200000028:     from all iif elvetias failed_policy
4200000028:     from all iif elvetias failed_policy
4200000029:     from all iif roadwarrior failed_policy
4200000031:     from all iif 6in4-HEnet failed_policy


Output of "ip route list table 1-250"
-------------------------------------------------
Table 1: default via XXX.XXX.XXX.XXX dev pppoe-wan
Table 2: default via 100.117.168.209 dev wwan0


Output of "iptables -L -t mangle -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 267K packets, 324M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 302K  334M mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 10508 packets, 1489K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 245K packets, 321M bytes)
 pkts bytes target     prot opt in     out     source               destination         
  17M 4855M qos_Default  all  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0           
  181 10732 TCPMSS     tcp  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      6in4-HEnet  0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp  --  *      wwan0   0.0.0.0/0            0.0.0.0/0            tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 9779 packets, 2166K bytes)
 pkts bytes target     prot opt in     out     source               destination         
  28M 3965M qos_Default  all  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0           
20649 5550K mwan3_hook  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 255K packets, 323M bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 254K  328M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 322K  340M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0x3f00
25388 2738K mwan3_ifaces_in  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
24713 2695K mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 2436  278K mwan3_ifaces_out  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1036  160K mwan3_rules  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 322K  340M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0x3f00
 265K  329M mwan3_connected  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_LTE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  wwan0  *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all  --  wwan0  *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* LTE */ MARK xset 0x200/0x3f00

Chain mwan3_iface_in_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  103 13875 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            match-set mwan3_connected src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
  572 29863 MARK       all  --  pppoe-wan *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_iface_out_LTE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  695 58380 MARK       all  --  *      wwan0   0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* LTE */ MARK xset 0x200/0x3f00

Chain mwan3_iface_out_wan (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  705 59220 MARK       all  --  *      pppoe-wan  0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan */ MARK xset 0x100/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
25382 2738K mwan3_iface_in_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
24587 2684K mwan3_iface_in_LTE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_ifaces_out (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 2435  278K mwan3_iface_out_wan  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00
 1726  217K mwan3_iface_out_LTE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00

Chain mwan3_policy_HEnet_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan_LTE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  514 87660 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_policy_wan_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* wan 3 3 */ MARK xset 0x100/0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  514 87660 mwan3_policy_wan_LTE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default_rule */
    0     0 mwan3_policy_HEnet_only  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0x3f00 /* default_rule_v6 */

Chain qos_Default (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  45M 8820M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK restore mask 0xf
 382K   42M qos_Default_ct  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf
  30M 3908M MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf0 length 0:500 MARK xset 0x22/0xff
 195K   19M MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            MARK xset 0x11/0xff
 414K  347M MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf0 tcp spts:1024:65535 dpts:1024:65535 MARK xset 0x44/0xff
2314K 2382M MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf0 udp spts:1024:65535 dpts:1024:65535 MARK xset 0x44/0xff
  45M 8820M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff

Chain qos_Default_ct (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   78 17714 MARK       all  --  *      *       10.0.2.31            0.0.0.0/0            mark match 0x0/0xf /* spa941 */ MARK xset 0x11/0xff
 3351  204K MARK       all  --  *      *       10.0.2.32            0.0.0.0/0            mark match 0x0/0xf /* gigaset-c530 */ MARK xset 0x11/0xff
 3182  183K MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf tcp multiport ports 22,53 /* ssh, dns */ MARK xset 0x22/0xff
96434 6626K MARK       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf udp multiport ports 22,53 /* ssh, dns */ MARK xset 0x22/0xff
55588 4907K MARK       icmp --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf MARK xset 0x22/0xff
 139K 8513K MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf tcp multiport ports 20,21,25,80,110,143,443,465,993,995 /* ftp, smtp, http(s), imap, pop */ MARK xset 0x33/0xff
84179   22M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x0/0xf MARK xset 0x44/0xff
 382K   42M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0            CONNMARK save mask 0xff


Output of "ip6tables -L -t mangle -v -n"
-------------------------------------------------
Chain PREROUTING (policy ACCEPT 2351 packets, 269K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 3639  416K mwan3_hook  all      *      *       ::/0                 ::/0                

Chain INPUT (policy ACCEPT 21 packets, 3048 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 qos_Default  all      *      pppoe-wan  ::/0                 ::/0                
    0     0 TCPMSS     tcp      *      pppoe-wan  ::/0                 ::/0                 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp      *      6in4-HEnet  ::/0                 ::/0                 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU
    0     0 TCPMSS     tcp      *      wwan0   ::/0                 ::/0                 tcp flags:0x06/0x02 /* !fw3: Zone wan MTU fixing */ TCPMSS clamp to PMTU

Chain OUTPUT (policy ACCEPT 812 packets, 86596 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 4974  784K qos_Default  all      *      pppoe-wan  ::/0                 ::/0                
  878 95344 mwan3_hook  all      *      *       ::/0                 ::/0                

Chain POSTROUTING (policy ACCEPT 812 packets, 86596 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain mwan3_connected (2 references)
 pkts bytes target     prot opt in     out     source               destination         
   17  1628 MARK       all      *      *       ::/0                 ::/0                 match-set mwan3_connected dst MARK or 0x3f00

Chain mwan3_hook (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    4   224 RETURN     icmpv6    *      *       ::/0                 ::/0                 ipv6-icmptype 133
   26  6864 RETURN     icmpv6    *      *       ::/0                 ::/0                 ipv6-icmptype 134
    4   256 RETURN     icmpv6    *      *       ::/0                 ::/0                 ipv6-icmptype 135
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 ipv6-icmptype 136
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 ipv6-icmptype 137
 4483  504K CONNMARK   all      *      *       ::/0                 ::/0                 CONNMARK restore mask 0x3f00
 4390  494K mwan3_ifaces_in  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00
 4390  494K mwan3_connected  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00
 4386  494K mwan3_ifaces_out  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00
 3658  418K mwan3_rules  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00
 4483  504K CONNMARK   all      *      *       ::/0                 ::/0                 CONNMARK save mask 0x3f00
 2090  231K mwan3_connected  all      *      *       ::/0                 ::/0                 mark match ! 0x3f00/0x3f00

Chain mwan3_iface_in_HEnet (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all      6in4-HEnet *       ::/0                 ::/0                 match-set mwan3_connected_v6 src mark match 0x0/0x3f00 /* default */ MARK or 0x3f00
    0     0 MARK       all      6in4-HEnet *       ::/0                 ::/0                 mark match 0x0/0x3f00 /* HEnet */ MARK xset 0x300/0x3f00

Chain mwan3_iface_out_HEnet (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  728 75712 MARK       all      *      6in4-HEnet  ::/0                 ::/0                 mark match 0x0/0x3f00 /* HEnet */ MARK xset 0x300/0x3f00

Chain mwan3_ifaces_in (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 3093  346K mwan3_iface_in_HEnet  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00

Chain mwan3_ifaces_out (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 3092  346K mwan3_iface_out_HEnet  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00

Chain mwan3_policy_HEnet_only (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan_LTE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 2352  269K MARK       all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_policy_wan_only (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00 /* default */ MARK or 0x3f00

Chain mwan3_rules (1 references)
 pkts bytes target     prot opt in     out     source               destination         
 2352  269K mwan3_policy_wan_LTE  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00 /* default_rule */
    0     0 mwan3_policy_HEnet_only  all      *      *       ::/0                 ::/0                 mark match 0x0/0x3f00 /* default_rule_v6 */

Chain qos_Default (2 references)
 pkts bytes target     prot opt in     out     source               destination         
 4974  784K CONNMARK   all      *      *       ::/0                 ::/0                 CONNMARK restore mask 0xf
 4950  781K qos_Default_ct  all      *      *       ::/0                 ::/0                 mark match 0x0/0xf
   16  2528 MARK       udp      *      *       ::/0                 ::/0                 mark match 0x0/0xf0 length 0:500 MARK xset 0x22/0xff
    0     0 MARK       icmp     *      *       ::/0                 ::/0                 MARK xset 0x11/0xff
    0     0 MARK       tcp      *      *       ::/0                 ::/0                 mark match 0x0/0xf0 tcp spts:1024:65535 dpts:1024:65535 MARK xset 0x44/0xff
    0     0 MARK       udp      *      *       ::/0                 ::/0                 mark match 0x0/0xf0 udp spts:1024:65535 dpts:1024:65535 MARK xset 0x44/0xff
 4974  784K CONNMARK   all      *      *       ::/0                 ::/0                 CONNMARK save mask 0xff

Chain qos_Default_ct (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       tcp      *      *       ::/0                 ::/0                 mark match 0x0/0xf tcp multiport ports 22,53 /* ssh, dns */ MARK xset 0x22/0xff
    0     0 MARK       udp      *      *       ::/0                 ::/0                 mark match 0x0/0xf udp multiport ports 22,53 /* ssh, dns */ MARK xset 0x22/0xff
    0     0 MARK       icmp     *      *       ::/0                 ::/0                 mark match 0x0/0xf MARK xset 0x22/0xff
    0     0 MARK       tcp      *      *       ::/0                 ::/0                 mark match 0x0/0xf tcp multiport ports 20,21,25,80,110,143,443,465,993,995 /* ftp, smtp, http(s), imap, pop */ MARK xset 0x33/0xff
 4950  781K MARK       all      *      *       ::/0                 ::/0                 mark match 0x0/0xf MARK xset 0x44/0xff
 4950  781K CONNMARK   all      *      *       ::/0                 ::/0                 CONNMARK save mask 0xff

Did you set 0.0.0.0/0 as destination for the ipv4 rule and :: for the ipv6 rule? but its probably best to leave ipv6 out of mwan3 all together and use the main routing table for ipv6

Yes, I tried it like this as well. I believe this is the example I posted. For some reason it uses IPv4 rules for IPv6 traffic. The sequence of the rules didn't seem to make any difference.
I tried to leave IPv6 out of it, but it is still broken.

@feckert I also have a similar problem as @trendy but it may be unrelated to mwan3. I found out that when 6rd interface is up, I can manually reload firewall to get things working. (at least sometimes)

I also had ipv4 rule in ipv6 section like @trendy ...

Also the 6rd interface is created automatically so I am not able to assign a metric to it..

Also I had a strange problem. I setup the rule but way later in logs I saw this:

Wed Nov 13 01:53:41 2019 user.warn mwan3[25498]: Rule default_rule_ipv6 exceeds max of 15 chars. Not setting rule

This is just bad :slight_smile:

Perhaps it is bad that mwan3 is trying to setup ipv6 things right now. It seems broken. Perhaps ipv6 functionality should be developed separately or at least there should be a way to disable it if one desires.

It seems you are missing this route:

@feckert Can an option to disable mwan3 altogether be added? So system works like it did not exist, perhaps at least after a reboot? This may be useful for testing if mwan3 is causing troubles perhaps?

Without mwan3 it was working fine though, without the route even for the locally generated traffic.
It was when I enabled mwan3 that problems started.
Right now I have disabled mwan3 and I am trying to move to IPv6 from my ISP (more on that on a new thread), so I cannot test it anymore.

2 Likes

Stop mwan3
/etc/init.d/mwan3 stop

Disable mwan3
/etc/init.d/mwan3 disable

1 Like

@feckert I was stupid to not think about it. But I think there could be a enable/disable checkbox in luci interface. Perhaps not a very important feature request anyway :slight_smile:

1 Like

A post was split to a new topic: Mwan3track not working properly on wireguard