Mwan3 port to nftables

[dl12345]

Am I right to infer that at least one of your ISPs is delegating you only a /64?

So some of your devices have no stable L3 identity that they actually source traffic from, so you can't do per-host steering, and you have no way of doing per-class steering either unless you renumber your flat LAN on both IPv4 and IPv6 with an SSID per class and assign devices to the appropriate SSID, which is a potentially non-trivial topology adjustment.

So I can see the case for your first feature request, the MAC match. It's a simple solution to the problem and something that could probably be used by a good number of mwan3 users. Neither of the feature requests is technically challenging to implement on the face of it.

However, most Android and Apple devices randomise their MAC address anyway by default, so you're going to need to turn that feature off on the devices to be able to steer by MAC address.

The second feature request is a rather narrow use-case of VMs behind a laptop: it's a fragile hop-count match and you're the only person asking for it, so harder to justify, but I'll certainly look at it.

I followed through those links you posted and read a couple of the posts:

@patrakov

I have set up NPT on top of the original mwan3, using a firewall include similar to this.

Following the link in the above post to the thread on IPv6 WAN fail-over without IPv6 NAT, it seems like you're doing this ULA addressing and NAT66 precisely because there's no proper IPv6 failover mechanism in mwan3?

@patrakov

However, none of that should be necessary if there is no policy routing involved, i.e. if either all the traffic goes out through WANA or all traffic goes through WANB. Indeed, the prefix delegated from either WANA or from WANB should be directly announced on the LAN, and reannounced with zero validity period during the fail-over if that WAN breaks.

Agreed, and so perhaps you'll be pleased to know that the very thing you're asking for in that thread (quoted above) is one of the pieces of functionality upcoming in the beta release IPv6 extensions I referred to earlier in this thread.

RFC 8475 describes the approach and the beta mwan3 implements it: when a WAN soft fails, mwan3 will deprecate that prefix by setting preferred_lft 0 on the downstream LAN, so new connections from LAN clients will automatically failover to the other non-deprecated prefix. Then when the WAN recovers, the preferred lifetime will be restored and clients will use that prefix again. Valid lifetime is left alone so there is no re-solicitation. This requires SLAAC addressing to work and a delegated GUA prefix per WAN such that the clients get one IPv6 GUA assigned per delegated prefix.

Also in the beta mwan3 code, there's a 1:1 translation capability that auto-carves a translation pool (which can be that WAN's own delegated prefix), avoiding any mapping target collisions by auto-discovering all in-use GUA segments from that prefix, although it currently only works for GUA LAN segments since there's no auto-discovery of ULA LAN segments. ULA segments will just be masqueraded automatically.

Shouldn't be hard to extend the 1:1 feature to ULA, although that's not currently done. This 1:1 translation feature allows for full inbound reachability and uses the netmap feature. It went through a couple of iterations: I originally considered implementing it using NPT stateless translation, but that doesn't preserve hostid at a /64 granularity and so ends up being less useful than it otherwise could be as it's almost impossible to link the translated address to the originating host. So I did a roll-my-own stateless implementation that preserved the hostid portion of the address at the cost of an incremental checksum adjustment, but I baulked at releasing a stateless NPT-style translation that was not RFC 6296 compliant and converted it to netfilter's netmap, which is a standard feature, is 1:1, but is stateful. It preserves the hostid portion of the address, so translated addresses are predictable and it's just a better and simpler all-round choice to deliver the functionality.

The beta mwan3 does NAT (either 1:1 or masquerade) only when needed. So for example, if you have a GUA on your LAN and the egress WAN is the correctly source routed WAN, no translation occurs. If a packet is steered by rules to a WAN that can't route that prefix, only then will NAT of either 1:1 if defined, or masquerade if not defined, apply. Any form of steering or load balancing will always require NAT unless you have provider independent addressing with BGP.

You noted as well that

@patrakov

I think they [mwan3] have painted themselves into a corner by committing to support the situation that different uplinks could be chosen based on the destination IP - in which case that RFC is inapplicable.

I hope the beta mwan3 goes some way to fixing that as it takes over IPv6 routing and now does correct source routing, including router originated traffic with a catch-all failover rule in place, only applying NAT when a packet wants to egress a WAN with a source prefix that specific WAN doesn't route. That was a gap I think I've corrected in the beta.

Going back to your original problem, there is actually a feature in recent Android and which is supported by odhcpd, since I've tested it, that could fix your issue, but which (for no reason I can see) enforces GUA addressing and thus cannot use a ULA prefix. Android added DHCPv6-PD (IA_PD) in September 2025, per RFC 8415 and RFC 9762. The device requests its own dedicated prefix and if it obtains a prefix, it self-addresses from that prefix. So someone with a large enough delegation could potentially use this and give android clients each a /64, which then results in predictable addressing on each client that could then be used to steer based on source address, as any temporary address would fall within that prefix. In reality, it's impractical for anything other than at minimum a /56 ISP delegation.

Puzzlingly, the actual Google code rejects a ULA PD, so ULAs can't be used, making such a feature less useful in your own scenario. Not sure why it does so, since it strikes me that this would be a very useful feature to people whose ISPs give them only a small delegation.

If you're curious to test this feature, here's how I configured it on my own router (I did it on a VLANed test interface)

uci set network.test.ip6addr='<YOUR GUA>/60'
uci set dhcp.test.ra='server'
uci set dhcp.test.ra_slaac='1'
uci set dhcp.test.ra_default='2'
uci set dhcp.test.dhcpv6='server'
uci set dhcp.test.dhcpv6_pd='1'
uci set dhcp.test.dhcpv6_pd_min_len='64'
uci set dhcp.test.dhcpv6_pd_preferred='1'
uci commit network
uci commit dhcp
/etc/init.d/odhcpd reload
ifup test

I noticed when testing that when I flipped from GUA to ULA and vice versa in the example above, every time I did the flip, it actually caused my phone to reboot. Probably just a Samsung bug (I'm on Android 16)

[dl12345]

@moeller0

It is not encouraging that folks over in the ietf's ipv6 working groups mostly seem to have given up on simple multi homing in the network and seem to be pushing multi path TCP and QUIC. All the while still sneering that NAT is evil

Wonderful to be a purist, but it doesn't survive contact with the realities of consumer internet connections

[dl12345]

@patrakov

I think that, with two uplinks, there is no way a stateless setup can work.

I agree with you here.

Even in the stateless translation variant that I abandoned, while the translation mechanism was stateless, the setup itself was not, since mwan3 itself is by design stateful, a routing engine that uses ct_marks to pin a flow to a WAN.

For a reply to an inbound-initiated connection, a truly stateless setup cannot disambiguate two uplinks since it has no idea which prefix the connection arrived on.

A stateless translation mechanism in mwan3 is never going to buy a stateless setup.

[patrakov]

@dl12345

Am I right to infer that at least one of your ISPs is delegating you only a /64?

This is true, but I would say that it would be a stretch to infer that from what I wrote.

Two SSIDs/VLANs with separate IPv4 and IPv6 ranges, one routed through WAN only and one through the VPN only, would be doable, even without mwan3, but then the "for some devices, use the VPN if it works and fall back to WAN if it doesn't" part would be something I don't know how to solve.

@dl12345

Following the link in the above post to the thread on IPv6 WAN fail-over without IPv6 NAT, it seems like you're doing this ULA addressing and NAT66 precisely because there's no proper IPv6 failover mechanism in mwan3?

Not really. I have two networks that I maintain in separate locations. One uses mwan3 NAT66 with ULAs to switch between the DHCP-based WAN and the VPN, and that's where I would need the MAC-based approach and selective VPN use. The other one, with PPPoE fiber and LTE connections, previously used a similar mwan3-based approach, but now uses a whole-network fail-over mechanism not based on mwan3, described in the thread you linked to.

Regarding the recent support for DHCPv6-PD in Android, unfortunately, this is not going to be ported to the Android version that the smart TV is using.

[dl12345]

@patrakov

the "for some devices, use the VPN if it works and fall back to WAN if it doesn't" part would be something I don't know how to solve.

This would work with mwan3's beta. You'd need a 3rd VLAN / SSID / IP range and 3 different mwan3 policies, assuming you want the VPN-only policy to be a strict one that "unreachables" traffic if the VPN is down.

The first policy would contain only the WAN interface, the 2nd policy only the VPN interface and the 3rd policy would contain two members, the first being VPN with the lowest metric and the 2nd being WAN with a higher metric. Steer by source in all cases.

Since you're using ULA's, there'd be an automatic masquerade that would occur on the egress interface. It wouldn't do 1:1 as currently implemented, unless I extend it to cover ULAs too, which can be done (and is actually already on an unprioritised list of tasks, since it's a useful feature).

The IPv4 part already works for this configuration in the current release. It's just the IPv6 part for which you'd need the beta.

[the-e3n]

@dl12345 Is it possible to add ability to test dst_ip:dst_port config as well in simulator as right now it only allows to enter ip and not the port but since we can also have rules based on dst_port it will be nice to be able to test them as well.

[dl12345]

@the-e3n

Is it possible to add ability to test dst_ip:dst_port

The functionality is already there. A port specifier is only semantically valid if the protocol is UDP or TCP. So you need to drop down the Protocol list box and set it to TCP or UDP and then a data entry field for port will appear.

[the-e3n]

Oh ok, thanks didn't think of that

[ssdnvv]

@ssdnvv

It is unfortunately Cortex A73 for BananaPi R4.

Could you provide the mwan3 for this target as well?

@dl12345 ?

[dl12345]

@ssdnvv

Could you provide the mwan3 for this target as well?

You'll have to wait until a new release for it. Pushing a new release tag to the repo is what runs the build. I'm not going to do a full release cycle just to update the release.yml with a new target.

[Axxel]

Sorry if this is covered elsewhere in the thread, but search is failing me...

I have a Wireguard server configured like the wiki. This worked with iptables mwan3, but I believe is failing with this version. I cannot be sure because I use the inbound VPN infrequently and I upgraded to this version of mwan3 a while ago.

The basic symptom is that Wireguard WAN handshakes fail. Its clear this is because the inbound connection is arriving over one WAN interface, but the handshake reply is routed to the other WAN route with the lower metric. If I take the lower metric interface down clients are able to connect.

My old notes indicate that a similar problem in iptables was resolved by adding a mwan3 rule that matches on the Wireguard destination port. I believe this meant it became part of mwan's conntrack tagging that meant traffic was routed back to the inbound interface. These rules are still in place, but seem to have no effect (the do not log).

My goal is to accept Wireguard over either WAN interface on the same port (so clients don't need to worry about it). Is this possible in the new version? Am I incorrect that this is a change in mwan3 behavior?

[egc]

If mwan3 does not provides this then there is this simple solution:

OpenWRT-egc-add-on/pbr-via-wan at main · egc112/OpenWRT-egc-add-on

[Axxel]

Thanks for the suggestion, but unless I misunderstand the script (or the alternative solution) it seems to be setting the route to a particular interface? That's not my goal. If the packet arrives over wan0 I want it to return over wan0 and similar behavior for wan1. Both WAN interfaces should accept traffic for WireGuard, even concurrently.

This works for port forwarding already. According to my primitive understanding, this is because port forwards transit the NAT tables in a way that triggers conntrack to remember and reply from the original interface. But WireGuard packets come from the router itself, and those appear to pick the lowest metric.

The old version of mwan3 had docs discussing this problem, but I believe everything described there is long obsolete, even for the iptables version of mwan3. The docs for this version mention router-originated traffic, but appear to claim (at least to me) that it should already work for IPv4 traffic.

[jamesmacwhite]

The way this was handled on the iptables mwan3 was preventing mwan3 from marking the traffic to the Wireguard server dest port e.g. assuming your Wireguard server is using UDP 51820

iptables -t mangle -I mwan3_hook 2 -p udp --dport 51820 -j MARK --set-mark 0x100

You are correct, the behaviour you are seeing is the handshake failing because the traffic entered through one WAN and exited through another. This happened with me even if using failover, rather than load balancing. The solution in the iptables version was to prevent mwan3 marking that traffic.

I also have a Wireguard server and need to test it since migrating to 25.12, my first solution would be to have a mwan3 rule that ensures 51820 traffic is routed through a specific WAN to control the traffic flow, see if that works.

I remember needing to ensure any Wireguard clients didn't use 51820 to avoid conflicts.

[Axxel]

More searching pointed me to this discussion (where I see you participated). That issue makes it sound like this never should have worked, but again, it must have at some point and I wasn't using any of the techniques discussed there (including your iptables rule).

I have tried setting a mark in the firewall rule for the incoming port, but its not clear to me that's equivalent to the iptables command above. I have also tried adding mwan3 rules that would route all both incoming and outgoing (two rules, one for source and one for dest port) over the "right" WAN interface, but that made no change.

conntrack -L shows the incoming connection from WAN1 IP and the reply with the src set to the WAN2 IP (lower metric) which seems consistent with the discussion in the issue. Basically the claim is that the outbound interface is set before mwan3 ever has a chance to fix it.

The robots (Claude in this case) insist that the problem is always going to be that the source interface is set too early for mwan3 to rewrite it. It suggests that, excluding the patch in this issue, the best solution is to set the mark on the Wireguard interface to the same mark for the WAN interface as mwan3 (0x100, 0x200, etc.). This does work, but again locks me to a particular interface.

There's a more elaborate solution where I maintain my own mark and table with inverted metrics and my own hotplug script. Effectively a mini mwan3. This doesn't allow replies to clients on any interface, but would at least allow failover at the same time as mwan3. To the extent that I'm failing over by redirecting a CNAME I think this mostly works?

But I'm still hoping someone has a workaround that doesn't involve a kernel patch. Some version of the mangle command you suggested.

[jamesmacwhite]

For what it is worth, I tested a Wireguard server with the latest mwan3 nftables port and it works without any specific rule but the following scenarios may avoid the problem for my case:

1. The external WAN IPv4 to connect is the primary WAN used in mwan3 and always the primary WAN used for traffic i.e. the default rule
2. I do not have a published IPv4 DNS record for the Wireguard server using my secondary WAN

While I could publish a second IPv4 record on the alternative WAN, it seems overkill for a VPN used infrequently but also introduces potentially the issue you are referencing and I never bothered with it.

I do remember on the mwan3 iptables I needed that fwmark for traffic to work generally even just one WAN.

While it might be Wireguard specific in this case, I have services running over Cloudflare which use dual WAN and even use the Cloudflare load balancing feature with two pools configured for both WANs and this works fine. This however runs in failover not load balancing mode, so again likely prevents issues of traffic entering through one WAN but exiting another, when only one pool will be used at a time.

Maybe @dl12345 can help further.

[wackejohn]

As I mentioned in that issue, the key issue is wireguard itself: It's working in kernel space and cannot bind to a specific interface (e.g. lan). The wireguard ignores the ip address or fwmark of incoming packets (just reset the incomming address to 0.0.0.0), and then using the default route to select the source address for reply packets instead, meaning no conntrack association is established while mwan3 relies on conntrack to work.
The souce code from wireguard:

	if (!rt) {
		security_sk_classify_flow(sock, flowi4_to_flowi_common(&fl));
		if (unlikely(!inet_confirm_addr(sock_net(sock), NULL, 0,
						fl.saddr, RT_SCOPE_HOST))) {
			endpoint->src4.s_addr = 0;
			endpoint->src_if4 = 0;
			fl.saddr = 0;
			if (cache)
				dst_cache_reset(cache);
		}
		rt = ip_route_output_flow(sock_net(sock), &fl, sock);
		if (unlikely(endpoint->src_if4 && ((IS_ERR(rt) &&
			     PTR_ERR(rt) == -EINVAL) || (!IS_ERR(rt) &&
			     rt->dst.dev->ifindex != endpoint->src_if4)))) {
			endpoint->src4.s_addr = 0;
			endpoint->src_if4 = 0;
			fl.saddr = 0;
			if (cache)
				dst_cache_reset(cache);
			if (!IS_ERR(rt))
				ip_rt_put(rt);
			rt = ip_route_output_flow(sock_net(sock), &fl, sock);
		}
		if (IS_ERR(rt)) {
			ret = PTR_ERR(rt);
			net_dbg_ratelimited("%s: No route to %pISpfsc, error %d\n",
					    wg->dev->name, &endpoint->addr, ret);
			goto err;
		}
		if (cache)
			dst_cache_set_ip4(cache, &rt->dst, fl.saddr);
	}

So, there won't have a workaround without the kernel patch, unless the wireguard developer fix this issue.

[dl12345]

@Axxel

My goal is to accept Wireguard over either WAN interface on the same port

[Just noticed @wackejohn's post that he posted while I was writing this reply. He's correct except for it needing a kernel patch to fix. There is a workaround: the source-reset in the Wireguard code is conditional on the first route lookup egressing a non-arrival interface, and a source-based ip rule will keep that first lookup on the arrival interface. See the mwan3.user script at the end of this post for the workaround.]

So this does happen and it isn't an mwan3 bug. It's the Wireguard kernel code doing it on purpose. Wireguard's explicit logic is to select the main routing table's route in preference to the arrival interface when the two disagree, discarding the arrival source and resourcing from the default WAN's address.

In Wireguard's send4() on line 27 in drivers/net/wireguard/socket.c, Wireguard builds its own flow with .flowi4_mark = wg->fwmark and then sets skb->mark = wg->fwmark on line 36, using the default fwmark-0 that is a main table lookup.

The source it starts with is the address the handshake initiation arrived on: .saddr = endpoint->src4.s_addr (line 24).

endpoint->src4 is set from the received packet's destination in wg_socket_endpoint_from_skb() on line 243 by endpoint->src4.s_addr = ip_hdr(skb)->daddr with endpoint->src_if4 = skb->skb_iif on line 244, which is the arrival interface.

Wireguard does its own route lookup, rt = ip_route_output_flow(sock_net(sock), &fl, sock) on line 61 and caches it per-peer in a dst_cache (lines 48-49, 80-81).

After the first lookup, Wireguard checks whether the route it got egresses the same interface the packet arrived on

        if (unlikely(endpoint->src_if4 && ((IS_ERR(rt) &&
                 PTR_ERR(rt) == -EINVAL) || (!IS_ERR(rt) &&
                 rt->dst.dev->ifindex != endpoint->src_if4)))) {
            endpoint->src4.s_addr = 0;
            endpoint->src_if4 = 0;
            fl.saddr = 0;
            if (cache)
                dst_cache_reset(cache);
            if (!IS_ERR(rt))
                ip_rt_put(rt);
            rt = ip_route_output_flow(sock_net(sock), &fl, sock);
        }

In a multi-wan case, the fwmark-0 lookup returns the default WAN's route, whose device is not the arrival interface, so this code fires and Wireguard throws away the arrival source and arrival interface

        endpoint->src4.s_addr = 0;
        endpoint->src_if4 = 0;
        fl.saddr = 0;

and then re-routes with source selection using ip_route_output_flow(). The second lookup lands on the default WAN and selects that WAN's address.

So it's Wireguard itself explicitly choosing to send the response out of the default WAN, sourced from the default WAN's IP.

That re-sourcing breaks the conntrack match: the reply's source address no longer matches the inbound entry's reply tuple in the conntrack table, so it's considered an unrelated flow with connmark 0. Thus mwan3's connmark restore has nothing to restore.

I reproduced this with IPv4 and tried it with IPv6, which interestingly doesn't suffer from this problem.

The reason for that is that Wireguard's send6() does not have the src_if4 interface-mismatch fallback that send4() has. It only resets the source if the source is not a valid local address. It never ignores the arrival source just because the fwmark-0 route egresses a different interface. So in IPv6, Wireguard keeps the arrival source, the conntrack reply tuple still matches, and mwan3's mark restore works.

I really don't want to make mwan3 aware of specific applications / subsystems and build in workarounds if I can help it, but there is actually a clean way that would generalise to any other package that does something similar and is fairly complementary, and that is to add in a source-based ip rule, one that mwan3 can actually override if it wishes that will force Wireguards own fwmark-0 lookup to resolve to the arrival WAN and make Wireguard route the reply correctly.

It's useful enough that I'll add it to mwan3 in a future release. In the meantime, you can do the same thing by adding it via the mwan3.user script. Make sure mwan3.user is executable. You'll need to ifdown/ifup your WAN interfaces to install the rule.

#!/bin/sh
#
# /etc/mwan3.user - workaround for inbound Wireguard server return routing 
#
# The kernel Wireguard module routes its own reply packets using its interface
# fwmark (0 by default), so a reply to a client that reached the server on a
# non-default WAN is sent out the default WAN and the handshake fails. mwan3
# cannot correct this from its output hook, because Wireguard re-sources the
# reply from the default WAN's address, which breaks the conntrack match mwan3
# relies on.
#
# This installs, per enabled IPv4 mwan3 interface, an ip rule that routes any
# locally-generated traffic whose source is that WAN's own address out that
# WAN's routing table. A Wireguard reply is sourced from the address the client
# connected to (the arrival WAN's address), so this makes Wireguard's own route
# lookup select the arrival WAN and the reply returns correctly, for either WAN
# at once. Added on ifup, removed on ifdown. IPv6 does not need this as the kernel
# Wireguard v6 send path keeps the arrival source, so mwan3's normal handling works.

. /lib/functions.sh
. /lib/functions/network.sh
. /lib/mwan3/mwan3.sh

# One rule per interface at BASE + interface-id, clear of mwan3's own 1000/2000/
# 3000 bands and below the main table. Change BASE if it collides with another
# package's ip rules on your system or if you've changed priority bases in mwan3.

BASE=4000

config_load mwan3

config_get family "$INTERFACE" family ipv4
[ "$family" = "ipv4" ] || exit 0

mwan3_get_iface_id id "$INTERFACE"
[ -n "$id" ] || exit 0
pref=$((BASE + id))

case "$ACTION" in
	ifup|connected)
		while ip -4 rule del pref "$pref" 2>/dev/null; do :; done
		mwan3_get_src_ip src_ip "$INTERFACE"
		case "$src_ip" in
			""|0.0.0.0) ;;
			*) ip -4 rule add from "$src_ip" lookup "$id" pref "$pref" ;;
		esac
		;;
	ifdown)
		while ip -4 rule del pref "$pref" 2>/dev/null; do :; done
		;;
esac

exit 0

[dl12345]

@egc

there is this simple solution

Your solution pins to one WAN.

The desired end-goal of same-port-either-WAN needs a per-WAN ip rule keyed on each WAN's source address, updated if the WAN IP changes.

[dl12345]

mwan3 version 3.6.11

Fix the nft rule rendering order such that it preserves strict config order.

@jamesmacwhite this fixes the issue you raised on github.

@ssdnvv the current set of builds already contains one suitable for you: all Filogic builds are covered by aarch64_cortex-a53