L2tp issues routing through mwan3

Installing and Using OpenWrt Network and Wireless Configuration
1

Hi

I'm using openwrt 22.03.2 with mwan3 for failover and specific routing.
I have a multi homed setup. One of the wan interface is l2tp to aaisp which provides ipv4 and ipv6 from aaisp via pppoe plusnet.

I had an puzzling issue with traffic originating from the l2tp interface.
When I set up an mwan3 rule to route a specific PC via l2tp and tested it by pinging google.co.uk from the PC only every other packet was returned. i.e. Only odd numbered packets were returned.
After much investigation I discovered that packet arriving from the internet were arriving with a mask of 0x3f, which is the mwan3 mask for default routing.
I failed to find the underlying cause for this, so I tried setting the mask to zero in the nftables raw_prerouting chain with 'ip daddr meta mark set 0x00000000'.
This fixed the issue but I wonder if anybody has a better solution.

2

The mask 0x3f is applied by mwan3, packets don't arrive with it. And the mask is an identifier for the service which uses firewall masking. In case you have some other package which uses that, you should make sure they do not conflict.

  • Did you see all packets arriving at the l2tp interface, but only half of them were forwarded to the lan host?
  • Or you did not see the response at the l2tp on half of them, although the echo was sent properly?
  • If you have any packet captures it would also help.
    Also please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:
    grafik
    Remember to redact passwords, MAC addresses and any public IP addresses you may have
ubus call system board; \
uci export network; uci export mwan3; uci export firewall; \
mwan3 status; nft list ruleset; \
ip -4 addr ; ip -4 ro li tab all ; ip -4 ru; \
ip -6 addr ; ip -6 ro li tab all ; ip -6 ru
3

This is the same as https://github.com/openwrt/packages/issues/19607 (yes that bug is about ipsec, but it doesn't matter).

Workaround suggested there:

iptables -t mangle -I PREROUTING 1 -m comment --comment "Do not inherit the mark of looped-back packets" -j MARK --set-xmark 0x0/0x3f00

...and the same for IPv6. Which is equivalent to yours.

The real fix would be to include that into the mwan3 package.

1 Like
4

I think my workaround does the same thing. In /etc/firewall.user i have:

# this is a simple fix to packets arriving via l2tp with a fwmark set
# It also has the advantage that l2tp-aaisp mwan3 rules are used as intended
L2TPADDR=$( ip addr sh l2tp-aaisp | sed -n 's/.*inet \([0-9.]*\).*/\1/p' )
if [ -n "$L2TPADDR" ] ; then
        nft add rule inet fw4 raw_prerouting ip daddr $L2TPADDR meta mark set 0
fi

but I'm not sure whether this or patrakov's iptables equivalent has the biggest hit on resources.
The real fix must be to fix whatever is adding these fwmarks in the first place.

5

Thank you all for your help.
Annoyingly, I found the l2tp issue had already been addressed in https://jamesmacwhite.medium.com/the-story-of-why-i-use-a-l2tp-relay-from-andrews-arnold-831b0de42d7b , which I had studied while I was reaching which firewall to use. At that time I think I ignored iptables entry because I was planning to use the openwrt 22.0 rc release, which had been migrated to nftables (but as I found out later not mwan3).
I now have a much better understanding of nftables and iptables and how they interact.

1 Like
6

I'm not using AAISP, but I have a very similar use-case and I'm not seeing marked packets on the l2tp interface. I'm pretty sure they're only put there once you add an mwan3 rule involving that interface. Just having mwan3 active won't add the marks unless there's a rule somewhere involving that interface.

I'm also on Virgin Media. It also irritates me they don't have ipv6. I'm however rolling my own ipv6 tunnel: I have a server in a datacentre and I've setup an l2tpv3 tunnel from my OpenWRT to my server in the datacentre, where my provider has delegated me a ipv6 prefix that I'm giving to my LAN for native ipv6 connectivity.

Like you, I use mwan3 to manage my two internet connections (in failover mode, not in load balancing mode).

I don't have any mwan3 config involving the l2tp interface. It's not routing any ipv4. So it doesn't need any mwan3 rules. Anything ipv6 goes in and out over that interface since it's the only ipv6-enabled interface on my router.

I put some catch all logging rules into my PREROUTING table to look at the traffic on that interface and literally nothing is marked. Two examples below - I SSHed over ipv6 to my server and I pinged one of its ipv6 interfaces.

Below is what it logs and it's notable for the absence of any firewall mark.

I haven't got the l2tp interface defined anywhere in mwan3 and neither do I have any mwan3 rules.

I suspect that defining the interface in mwan3 (or adding an mwan3 rule involving that interface) causes packets to be marked.

So you can fix the problem by excluding that interface from your mwan3 configuration. If you only care about the ipv6 connectivity, then this is probably the way to go.

I imagine you don't care about ipv4 connectivity via AAISP anyway, since why would you choose to tunnel an ipv4 connection if you don't have to.

Mar 23 14:30:47 openwrt kernel: 4,522967,1550640558975,-;IN=l2tpeth0 OUT= MAC=aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa SRC=xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx DST=yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy LEN=80 TC=0 HOPLIMIT=64 FLOWLBL=517643 PROTO=TCP SPT=22 DPT=58462 WINDOW=64704 RES=0x00 ACK SYN URGP=0

Mar 23 14:34:49 openwrt kernel: 4,523296,1550882454085,-;IN=l2tpeth0 OUT= MAC=aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa:aa SRC=xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx DST=yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=294880 PROTO=ICMPv6 TYPE=129 CODE=0 ID=2 SEQ=1
111
7

It's interesting that that your not seeing any marked packets. I agree, this maybe because your l2tp does not pass through mwan3.

I'm using l2tp for both ipv4 & ipv6 via the Andrew & Arnold's (AAISP) l2tp service. One of my reasons for passing ipv4 through l2tp is the poor ping response to and from virgin media, including dropped packets in both directions, compared to the l2tp (with zero packet drop and reduce ping) tunnelled over virgin media.

In my opening post for this topic, where I stated 'This fixed the issue but I wonder if anybody has a better solution', I was concerned that my fix might impact on the maximum throughput of the firewall. However I have since change to Virgin Media M350 and speed tests are showing around 372Mb over the l2tp tunnel for both ipv4 & ipv6. This has allayed my concerns.

Ideally I would like to fix the underlying cause. I suspect this is either in xl2tpd or the kernel but I'm at a loss regarding how to confirm or disprove this.

8

Ah yes, I see and I agree - latency on virgin media is terrible and occasionally there are lots of dropped packets / instability. This is in fact the very reason I have a second internet link managed by mwan3. I can't rely on a single internet connection due to VM's poor stability, since I work from home.

My second internet link has 5ms latency to my remote server, whereas VM is all over the place with about 15ms - 20mns latency to the same server, so I send latency critical traffic over the second link and bulk traffic over the VM link, otherwise I'd probably be required to do something similar to you and send ipv4 traffic over the l2tp tunnel.

I'm using l2tpv3 for the tunnel instead of 6in4 as virgin media seem to throttle 6in4 connections and I get poor throughput on a 6in4 tunnel versus wire speed on the l2tpv3 tunnel.

Sounds like you and I have a similar use-case, except I opted for a slightly different and more expensive solution - a second VDSL2 internet link with Uno and then my own tunnel to my own server for the ipv6 connectivity.

I had the virtual server already for other reasons, so it isn't really an incremental cost, but the cost of the second ISP link + the server comes to about £100 per month on top of my virgin media Gig1 price.