1. What I think is your problem
Based on your description and the article you linked earlier in the thread, it seems that you want full transparent failover for all traffic, both LAN clients and router-originated traffic (nextdns DoH, wireguard, tailscale) when the primary wan goes down and this isn't happening as you expect.
You're reporting that without the mwan3.user routing metric manipulations, new router-originated connections don't failover, while LAN-originated traffic fails over correctly.
Manipulating the main routing table metrics in mwan3.user fixes everything for you and you want to know why this should be necessary.
2. My observations
You should not need to manipulate routing table metrics. Failover should happen correctly as mwan3 handles both forwarded and router originated traffic.
You appear to be treating a symptom and not the root cause.
You're getting behaviour that approximates your expectations, but in reality isn't what you've explicitly configured in mwan3.
You have ip rules at higher priority than mwan3 that are very likely overriding mwan3's processing and could be the cause.
Fix the root cause first.
2.1 IPv4 Policy Rules apparently interfering with mwan3
Your diagnostic output shows the following:
0: from all lookup 255
1001: from all iif eth1 lookup 1
1002: from all iif VLANs.1 lookup 2
1003: from all iif vpn_unl lookup 3
1005: from all iif wg_vpn lookup 5
1006: from all iif wg_awh lookup 6
1310: from all fwmark 0x80000/0xff0000 lookup 254
1330: from all fwmark 0x80000/0xff0000 lookup 253
1350: from all fwmark 0x80000/0xff0000 unreachable
1370: from all lookup 52
2001: from all fwmark 0x100/0x3f00 lookup 1
2002: from all fwmark 0x200/0x3f00 lookup 2
2003: from all fwmark 0x300/0x3f00 lookup 3
2005: from all fwmark 0x500/0x3f00 lookup 5
2006: from all fwmark 0x600/0x3f00 lookup 6
2061: from all fwmark 0x3d00/0x3f00 blackhole
2062: from all fwmark 0x3e00/0x3f00 unreachable
3001: from all fwmark 0x100/0x3f00 unreachable
3002: from all fwmark 0x200/0x3f00 unreachable
3003: from all fwmark 0x300/0x3f00 unreachable
3005: from all fwmark 0x500/0x3f00 unreachable
3006: from all fwmark 0x600/0x3f00 unreachable
32766: from all lookup 254
32767: from all lookup 253
90050: from all iif lo lookup 200
Note the rules at priority 1310 to 1370. These rules will override mwan3's processing.
These rules do not come from mwan3
mwan3 only installs:
iif-based rules at priorities 1001-1006fwmark 0xN00/0x3f00 rules at priorities 2001-2006 and 2061-2062fwmark 0xN00/0x3f00 unreachable fallbacks at priorities 3001-3006
The 0xff0000 mask and 0x80000 value are not from mwan3's 0x3f00 bit field, so these rules were installed by another application.
A vanilla openwrt installation has two unconditional ip rules: 32766: from all lookup main (table 254) and 32767: from all lookup default (table 253). These are normal and expected. Any packet not claimed by a higher-priority rule falls through to lookup main at 32766, which is the normal forwarding path.
The rules at 1310-1350 are an additional fwmark-conditional set of rules that have been inserted at a much higher priority and that intercept packets carrying 0x80000 in the 0xff0000 bits and route them directly to the main table at priority 1310.
Without those rules, a packet carrying 0x80000 would continue down the rule list, pass through mwan3's fwmark rules at 2001-2006, and be routed correctly by mwan3. The rules at 1310-1350 are the problem - not because they reference tables 254 and 253 (which is normal), but because they do so at a priority that pre-empts mwan3 for any packet carrying the 0x80000 mark.
I don't know the source but speculate that it might come from pbr as the fwmark mask 0xff0000 matches the fw_mask default configuration value of pbr. pbr's default installation behavior, however, does not match this rule pattern:
- pbr places rules at priority ~30000 by default (
option uplink_ip_rules_priority '30000'), not 1310-1350, however this can be overridden in pbr's configuration quite easily, so the difference in priority order here doesn't rule out pbr
- pbr installs one rule per configured uplink pointing to a named table (
pbr_IFACE). It does not install explicit rules pointing to tables 254 or 253.
- pbr does not install three rules per fwmark. The (lookup 254, lookup 253, unreachable) pattern for the same fwmark is likely not pbr's.
- pbr's global uplink rule uses
lookup main suppress_prefixlength N, not a bare lookup 254.
So there is some indication that they could come from pbr, but there's also some contradictory data that says they're not.
The critical unknown is what is applying that mark and to what that mark is being applied.
If pbr was previously configured with non-default priorities and then removed or reconfigured without a clean stop, ip rules installed at those priorities could persist. However, even under that scenario, the three-rule pattern does not match any pbr rule-installation path in the code, so the question as to what installed them remains, in my mind, unresolved.
You need to investigate where these rules come from.
UPDATE: on second review, the pattern in the priority numbering, starting at 1310 and incrementing by 20 for each rule until the final rule at priority 1370, which we know is a tailscale rule, now leads me to believe that tailscale itself could be the source of the rules. The sequence of the numbering is too predictable to be a coincidence and these rules are probably installed by tailscale in a single block
2.2 Reason for failover path for router-originated traffic not working
- Something marks router-originated packets, including nextdns DoH, with fwmark
0x80000 in the 0xff0000 bit range.
- mwan3's
mwan3_output hook runs and applies the failover policy mark 0x200 (wan_lte's mark, in the 0x3f00 bits). Since 0xff0000 and 0x3f00 are non-overlapping bit fields, both marks coexist: the final fwmark on the packet is 0x80200.
- The kernel walks the ip rule list. Rule 1310 (
fwmark 0x80000/0xff0000 lookup 254) fires before mwan3's rule 2002 (fwmark 0x200/0x3f00 lookup 2), because 1310 < 2002.
- Rule 1310 sends the packet to table 254 (main). The main table still has
default via wan-GW dev eth1 metric 10. Traffic hits eth1, which has no connectivity and thus your dns / router-originated traffic fails.
- mwan3's mark of
0x200 is completely ignored because of these higher priority rules.
2.3 Reason for failover path for LAN forwarded traffic working correctly
The ip rules at 1310-1350 are fwmark-conditional. They fire for any packet carrying 0x80000 in the 0xff0000 bits, regardless of whether it is forwarded or locally generated. There is nothing in those rules that inherently restricts them to router-originated traffic.
LAN traffic probably works because LAN forwarded packets are likely not carrying the 0x80000 mark when the routing decision is made.
So a plausible explanation is that the marking is applied in your nftables OUTPUT chain, which processes only router-generated packets.
Forwarded LAN traffic passes through PREROUTING and FORWARD but never through OUTPUT, so it arrives at the routing decision without 0x80000 set. Rule 1310 does not fire, mwan3's rule 2002 fires, and LAN traffic is routed to table 2 (wan_lte) correctly.
This is my speculation only and nft list ruleset would help to confirm it.
2.4 Probable reason why your mwan3.user metric workaround fixes it
By raising wan's main-table route metric from 10 to 100, wan_lte (metric 20) becomes the lowest-metric default in table 254.
When rule 1310 sends traffic to table 254, it now exits via wan_lte instead of eth1.
Rule 1310 still fires and overrides mwan3's mark, sending it to the main table, but it no longer causes harm because the main table's preference has changed to wan_lte.
Your mwan3.user workaround is fixing the symptom, not the cause.
2.5 The 1370: from all lookup 52 rule
Table 52 is shown in your diagnostics to contain host routes and a /22 subnet, all via tailscale0.
The rule at priority 1370 is unconditional: every packet is looked up in table 52. If the destination is a tailscale virtual IP present in table 52, the packet is routed via tailscale0 regardless of mwan3's marks. For non-tailscale destinations, table 52 has no matching route and the kernel falls through to the next rule.
My attribution of the from all lookup 52 rule to tailscale is based on the fact that all routes in table 52 are via tailscale0.
This rule is not the cause of your failover problem for non-tailscale destinations.
Whether it fully explains why tailscale virtual IP traffic "works fine" during failover depends on whether the 0x80000 marking (from the unknown source of rules 1310-1350) also affects tailscale-destined traffic or not.
If tailscale packets carry 0x80000, rule 1310 fires before rule 1370, routing them to table 254 rather than table 52. If they do not carry 0x80000, rule 1370 intercepts them and routes via tailscale0 correctly. This cannot be determined without knowing the source and scope of any 0x80000 marking that might be occurring.
2.6 10.0.0.0/8 in mwan3_connected_v4
Your diagnostics shows mwan3_connected_v4 contains a /8 entry. This is 10.0.0.0/8, added because wg_vpn has interface address 10.1.1.59/8 (a /8 prefix length), which generates a 10.0.0.0/8 connected route in the main routing table.
mwan3_set_connected_ipv4 reads all CIDR routes from the main table and adds them to mwan3_connected_v4.
Any traffic destined for 10.0.0.0/8, the wg_vps peers at 10.0.50.x, wg_awh at 10.0.31.x, vpn_unl at 10.103.209.125, or any internal DoH endpoint using a 10.x.x.x address will match the mwan3_connected chain and receive the MMX_DEFAULT mark, bypassing mwan3_rules entirely.
For destinations with specific host routes in the main table (e.g., 10.0.50.1/32 dev wg_vps), traffic still arrives correctly even with MMX_DEFAULT, because the main table's specific route is more specific than the default gateway. This is why LAN pings to wireguard tunnel IPs work during failover even though they receive MMX_DEFAULT.
If nextdns is configured to use a DoH endpoint at a 10.x.x.x address via a VPN tunnel, this is a contributing cause of its failure. If nextdns uses standard public IPs for DoH, the 10.0.0.0/8 issue does not affect it and the higher-priority fwmark rules at 1310-1350 are the sole primary cause.
2.7 What I recommend you do to fix it
2.7.1 Identify and neutralise the 1310-1350 rules
Identify the application or script that installed the 0x80000/0xff0000 ip rules and is marking packets with 0x80000.
Once identified, either stop and cleanly uninstall it, then verify ip rules are removed, or reconfigure it to use priorities above mwan3's range (above 3006).
If the rules are orphans from a package that is no longer active, flush them manually with ip rule del priority 1310; ip rule del priority 1330; ip rule del priority 1350 and confirm nothing reinstalls them on reboot.
Frankly, I'd just go ahead and manually remove them, reboot and then see if they reappear. If they don't, then see if things work. If they do, then find what's installing those rules and stop it from installing them.
2.7.2 Fix the wg_vpn prefix
Change wg_vpn's wireguard interface configuration to use a more appropriately specific prefix (e.g., /24 or /32) instead of /8. This removes the very big 10.0.0.0/8 block from mwan3_connected_v4 and ensures traffic to 10.x.x.x addresses is classified by mwan3_rules rather than having the entire /8 block simply bypassed by mwan3 because it's classified as a directly connected network.
2.8 Additional information you need to be sure of the diagnosis
- What installed the ip rules at 1310-1350?
- What is marking router-originated packets with
0x80000/0xff0000? Start by searching init scripts, config files and hotplug scripts for 80000 and ff0000
- What nftables chains are marking packets with the
0xff0000 fwmark range? Run: nft list ruleset | grep -B4 -A4 "ff0000". This identifies what is setting 0x80000 on packets, which is required for the ip rules at 1310-1350 to have any effect.
- What is in routing table 200? Run:
ip -4 route list table 200 (there is a rule 90050: from all iif lo lookup 200 with no corresponding table section in the diagnostics I got from you since it doesn't dump that table as part of the diagnostics run).
- What is nextdns configured to use as its DoH upstream? Is is a public IP or one routed to a private IP over a VPN tunnel / proxy?
- Is the
/8 prefix in the wireguard config intentional, or is it a misconfiguration (I suspect an unintentional misconfiguration because you're unaware that the /8 essentially bypasses mwan3 for anything at all with a 10.x.x.x destination address)?
)