Very bad performance using NAT6 and mwan3

Knogle · December 14, 2020, 9:57pm

Ahoy friends.
Currently i am using my mwan3 setup again, this time having a PPPoE connection, and DHCP and DHCPv6 WAN connection through another router, with Double-NAT unfortunately. But i have provided IPv6-PD.

I have read the following page https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6 in order to enable IPv6 NATting to get IPv6 work. It is quite complicated still.
I have got one example, when i try to traceroute6 google, the result is quite weird.
Also it takes almost a minute in order to perform the command.
Same for ping google.com ,it takes almost 30 seconds between each interval. What is the problem here?
Is there maybe another solution instead of using this way to provide IPv6 support? Maybe 4in6 ? Or maybe working without IPv4 at all, and encapsulate IPv4 in IPv6, only using IPv6.

My traceroute result looks like that.

Looks like it is looping somehow.

traceroute6 google.de
traceroute to google.de (2a00:1450:4001:820::2003), 30 hops max, 80 byte packets
 1  2001-4dd0-5033-50-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:50::1)  0.271 ms  0.176 ms  0.214 ms
 2  braslns-vc1-lo0.netcologne.de (2001:4dd0:a2a:71::4acc)  14.292 ms  14.564 ms  15.262 ms
 3  ip-core-sto1-ae16.netcologne.de (2001:4dd0:a2b:112:dc30::c)  14.745 ms ip-core-eup1-ae16.netcologne.de (2001:4dd0:a2b:110:dc40::c)  15.260 ms  15.222 ms
 4  ip-core-eup1-et2-2-2.netcologne.de (2001:4dd0:a2b:20:dc40::1)  15.785 ms bdr-eup1-ae2.netcologne.de (2001:4dd0:a2b:11:dc41::b)  15.748 ms ip-core-eup2-et2-2-2.netcologne.de (2001:4dd0:a2b:21:dc40::1)  20.806 ms
 5  2001:4dd0:b2b:2::1 (2001:4dd0:b2b:2::1)  19.474 ms  19.529 ms bdr-eup1-ae2.netcologne.de (2001:4dd0:a2b:11:dc41::b)  16.638 ms
 6  2001:4dd0:b2b:2::1 (2001:4dd0:b2b:2::1)  19.682 ms  18.294 ms *
 7  * 2a00:1450:804a::1 (2a00:1450:804a::1)  18.187 ms 2001:4860:0:11e2::1 (2001:4860:0:11e2::1)  19.385 ms
 8  2001:4860:0:1::1f97 (2001:4860:0:1::1f97)  18.390 ms * 2001:4860:0:11e0::10 (2001:4860:0:11e0::10)  25.838 ms
 9  fra15s24-in-x03.1e100.net (2a00:1450:4001:820::2003)  18.767 ms 2001:4860::8:0:cb95 (2001:4860::8:0:cb95)  20.310 ms 2001:4860:0:1::1f99 (2001:4860:0:1::1f99)  19.428 ms

Thanks in advance, i think this will be quite a large topic again.

jamesmacwhite · January 1, 2021, 7:19pm

Depending on the version of mwan3 in use you might be hitting this issue: https://github.com/openwrt/packages/issues/14332

I've so far found under mwan3 2.10.x that IPv6 packets seem to being routed incorrectly 50% of the time.

Knogle · January 1, 2021, 7:34pm

Yeah i got mwan3 2.10.4-1

jamesmacwhite · January 1, 2021, 7:46pm

There is definitely something up with IPv6 and mwan3 2.10.x currently. That's why I'm under the 2.8.x version currently on 19.07, because it does work, but there are still known issues with IPv6 in that version as well.

If you want to debug, try adding these custom firewall rules:

# IPv6 mwan3 2.10.x debugging
IP="2607:f8b0:4002:c00::66"
ip6tables --table mangle -I FORWARD 1 -d $IP -j LOG --log-prefix "_forward start "
ip6tables --table mangle -I FORWARD 1 -s $IP -j LOG --log-prefix "_forward start "
ip6tables --table mangle -A FORWARD -d $IP -j LOG --log-prefix "_forward end "
ip6tables --table mangle -A FORWARD -s $IP -j LOG --log-prefix "_forward end "
ip6tables --table mangle -I POSTROUTING 1 -d $IP -j LOG --log-prefix "postroute start  "
ip6tables --table mangle -I POSTROUTING 1 -s $IP -j LOG --log-prefix "postroute start  "
ip6tables --table mangle -A POSTROUTING -d $IP -j LOG --log-prefix "postroute end  "
ip6tables --table mangle -A POSTROUTING -s $IP -j LOG --log-prefix "postroute end  "

Then ping 2607:f8b0:4002:c00::66 from a LAN client (not the router). You'll see entries in your system log from the ICMP ping traffic and post them here.

What I seem to be having (also using NAT6) is every other ICMP6 packet isn't routed correctly, which produces a 50% loss on ICMP6. So while IPv6 is working, clients basically have 50% of the IPv6 traffic going dead, so IPv6 sites will for the most part timeout or not load properly. Traceroute seems to work, but various hops will appear as * because of the timeout.

@aaronjg is looking into the issue, although more feedback and debugging on it will help identify why it's happening, assuming you have the same problem, it sounds similar though and given we are both using NAT6.

Knogle · January 2, 2021, 2:07am

jamesmacwhite:

IP="2607:f8b0:4002:c00::66"
ip6tables --table mangle -I FORWARD 1 -d $IP -j LOG --log-prefix "_forward start "
ip6tables --table mangle -I FORWARD 1 -s $IP -j LOG --log-prefix "_forward start "
ip6tables --table mangle -A FORWARD -d $IP -j LOG --log-prefix "_forward end "
ip6tables --table mangle -A FORWARD -s $IP -j LOG --log-prefix "_forward end "
ip6tables --table mangle -I POSTROUTING 1 -d $IP -j LOG --log-prefix "postroute start  "
ip6tables --table mangle -I POSTROUTING 1 -s $IP -j LOG --log-prefix "postroute start  "
ip6tables --table mangle -A POSTROUTING -d $IP -j LOG --log-prefix "postroute end  "
ip6tables --table mangle -A POSTROUTING -s $IP -j LOG --log-prefix "postroute end  "

Thanks a lot. Right now i am not encountering this issue, but the IPv6 traffic is not being balanced.
I only got one outgoing interface in the log, not both.

Second issue, but i dont know what is causing it.
When i try to ping google.com, google.de or something else, i get "No Route".
Also tried the logging above, there is no outgoing interface.
How to troubleshoot in this case?

How can i verify my NAT6 first of all?

Ping this address was quite funny. First of all, 200ms ping all the time for 1000 counts
, now after reboot, 100ms all the time. IPv4 ping delay is constant.

Now it looks like that.

02:c00::66(2607:f8b0:4002:c00::66) 56 data bytes
From 2001:4dd0:5033:1::1 icmp_seq=1 Destination unreachable: No route
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=2 ttl=104 time=114 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=3 ttl=104 time=114 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=4 ttl=104 time=114 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=5 ttl=104 time=114 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=6 ttl=104 time=115 ms

2nd try, huge latency.

ping 2607:f8b0:4002:c00::66
PING 2607:f8b0:4002:c00::66(2607:f8b0:4002:c00::66) 56 data bytes
From 2001:4dd0:5033:1::1 icmp_seq=1 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=2 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=3 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=4 Destination unreachable: No route
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=5 ttl=104 time=291 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=6 ttl=104 time=291 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=7 ttl=104 time=291 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=8 ttl=104 time=292 ms
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=9 ttl=104 time=299 ms

Traceroute doesnt work.

traceroute6 2607:f8b0:4002:c00::66
traceroute to 2607:f8b0:4002:c00::66 (2607:f8b0:4002:c00::66), 30 hops max, 80 byte packets
 1  2001-4dd0-5033-1-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:1::1)  0.230 ms !N  0.202 ms !N *

jamesmacwhite · January 2, 2021, 7:52am

Ideally you need to see the sys log entries when pinging failed to determine why. I'd report the issue here: https://github.com/openwrt/packages/issues. Post as much detailed info and logs you can as above and go from there.

aaronjg · January 3, 2021, 12:30am

Can you test this branch, and see if it fixes the issue?

If not, you may be having the same issue that James has, and we are working on resolving it.

jamesmacwhite · January 5, 2021, 6:09pm

After further debugging my issue, it would appear the issue I have is specific to L2TP, so probably not the same issue, however the branch @aaronjg mentioned was merged recently and a new version of mwan3 2.10.6 should now be available, I suggest trying that which has IPv6 routing table fixes.

Knogle · January 14, 2021, 10:41pm

I have tried the new version
Now it looks like that.

chairman@kali:~$ ping 2607:f8b0:4002:c00::66
PING 2607:f8b0:4002:c00::66(2607:f8b0:4002:c00::66) 56 data bytes
From 2001:4dd0:5033:1::1 icmp_seq=1 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=2 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=3 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=4 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=5 Destination unreachable: No route
From 2001:4dd0:5033:1::1 icmp_seq=6 Destination unreachable: No route
64 bytes from 2607:f8b0:4002:c00::66: icmp_seq=7 ttl=103 time=301 ms

jamesmacwhite · January 18, 2021, 11:37am

I'd open an issue on https://github.com/openwrt/packages titled "mwan3: Very bad performance using NAT6" for the attention of @feckert and @aaronjg. I do wonder if you have packets going out from the wrong interface as well, but you need to add more debug logging with iptables to see the specific ICMP packets and where they are coming from and exiting which interface.

aaronjg · January 18, 2021, 2:52pm

Interesting. Would be good to see the additional logging. With the other L2TP issue I believe we also saw problems with PPoE so possible there is a bug upstream. Will look for a fix or workaround.

Knogle · January 21, 2021, 6:36pm

Ahoy friends. I think my issue is slightly different, but somehow related to mwan3 i think.
To sum up, now i got 3 connections.

ethX, DHCP and DHCPv6 client as PPPoE with Virtual Dynamic IPv6 interface.
ethY, DHCP and DHCPv6 client, behind DOCSIS Router in somewhat like a bridge mode.
I have received a /30 subnet from my ISP. My DOCSIS Router is configured to be the gateway using 1st address of this subnet, my OpenWRT got the 2nd usable address assigned.
IPv6 is being passed through with Prefix Delegation, and is provided by 6in4 tunnel from ISP to DOCSIS Router.
ethZ, DHCP and DHCPv6 client behind DSL Router (NAT), without PD.

When i "restart" my router, all WAN interfaces are up, except the wanb_6 interface, which is my IPv6 DHCPv6 client interface, connected to the DOCSIS router. But having a look on "interfaces" tab the link is up, and the gateway docsis router is also reachable using IPv6. So there is something wrong, because mwan3 is listing this interface as "Down" in red.

But when i tick "Restart" on my WANB_6 interface, it is being listed as "Up" in mwan3.
Now i have some strange behaviour.
When i leave all these interfaces like they are on start, IPv6 functionallity is given somehow. I am able to ping IPv6 addresses without having "No route" messages.

DHCPv6 from the DOCSIS Router, wanb_6 is somehow not listed as IPv6 upstream on "Overview" Tab.

But having a look on IPv6-test.com, it works fine until now.

But when i tick "Restart" on my DHCPv6 interface, connected to my DOCSIS Router, it changes it's state into "Up" according to mwan3, but nothing works anymore, at least what is related to IPv6.

All interfaces "Up" on mwan3.

But ping now shows that.

chairman@debian:~$ sudo ping ipv6.google.com
PING ipv6.google.com(fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e)) 56 data bytes
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=1 Destination unreachable: No route
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=2 Destination unreachable: No route
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=3 Destination unreachable: No route
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=4 Destination unreachable: No route
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=5 Destination unreachable: No route
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=6 Destination unreachable: No route
From 2001-4dd0-5033-3-0-0-0-1.ipv6dyn.netcologne.de (2001:4dd0:5033:3::1) icmp_seq=7 Destination unreachable: No route
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=8 ttl=117 time=20.0 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=9 ttl=117 time=17.9 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=10 ttl=117 time=17.8 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=11 ttl=117 time=17.8 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=12 ttl=117 time=17.5 ms

So is there maybe a different issue, or something what might be related to mwan3?
What's the best way to troubleshoot if traffic is going out using load-balancing?

Is PD even necessary on WANB_6 when i have my prefix via WAN_6 already?

I personally think, the issue is somehow related to WANB_6, or the upstream router (DOCSIS Gateway).

When disabling WANB_6, it works fine again.

chairman@debian:~$ sudo ping ipv6.google.com
PING ipv6.google.com(fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e)) 56 data bytes
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=1 ttl=117 time=19.8 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=2 ttl=117 time=17.8 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=3 ttl=117 time=17.6 ms
64 bytes from fra16s13-in-x0e.1e100.net (2a00:1450:4001:819::200e): icmp_seq=4 ttl=117 time=17.4 ms

feckert · January 28, 2021, 9:03am

Thanks for the good issue description. At first we need the openwrt version. There where some changes in mwan3 on ipv6 between openwrt-19.07 and master. I also think this is not related to this issue. Could you please open a new issue? You could also try this changes https://github.com/openwrt/packages/pull/14473 that is staged for the openwrt-19.07 branch.