I hate to beat the PBR vs mwan3 dead horse, but I'm at my wit's end. I've run mwan3 since roughly Chaos Calmer, primarily for automatic failover to a backup ISP. I was also using it to implement policy routing, but that stopped working around 18.06 or 19.07 (in hindsight I probably just needed iptables -t mangle -I PREROUTING 1 -m comment --comment "Do not inherit the mark of encrypted packets" -j MARK --set-xmark 0x0/0x3f00 but I didn't know that trick at the time), so I installed vpn-policy-routing, which was fine until 22.03.3 when that package went away.
My most concise question, before we dive into my config, is: given a LAN br-ch and OpenVPN WAN tun-ch, what is the shortest config to:
- route all traffic from
br-chtotun-ch - but still be able to SSH to a host on
br-chfrom my primary LAN - while having mwan3 still manage failover of my primary WAN interface
? Because that's all I want to do: I have a secondary LAN br-ch tied to a VLAN, alternate SSID, and LXC instance that I want to send to a VPN exclusive of everything else except be able to login to a host on it from my primary LAN. I don't care if I use pbr, do it all in mwan3, or something else.
Fwiw I've been through:
- PBR not working with mwan3: fine, I accept pbr and mwan3 may be incompatible. I'm okay with that and was previously capable of getting mwan3 to implement my policy routing years ago, but it just doesn't work in this layout for reasons I cannot determine (and yes, my wan interfaces all have different metrics).
- Configure OpenVPN only on some LAN ports: intriguing, but it uses vpn-policy-routing, which is gone and had worked for me previously. Since my problem is with pbr, I'm not sure this helps.
- Source-IP routing rule to vpn tunnel: seems the easiest and most concise solution for my goal, but it also doesn't work for me. The config is so small and intuitive I'm somewhat dumbfounded it doesn't work, but it doesn't.
At this point I've tried all the things, including the things that contradict the other things (literally I'm 60-80 hours on-and-off over the last 7-8 years into trying to just route a network to a VPN). At this point I have a pbr config that works if mwan3 is off and stops working as soon as mwan3 is on, which is the most progress I've had recently (and yes, I have pbr set to "Insert", which the README says makes it compatible w/ mwan3).
Again, though, I'd be happy with any config. E.g., back to Source-IP routing rule to vpn tunnel, which is now closed, is the most promising but doesn't have a concise, final solution. Could $someone (maybe @trendy or @phoedos) give me the final outcome there?
To that solution, I have (on 22.03.4):
config route
option target '0.0.0.0'
option netmask '0.0.0.0'
option metric '100'
option table 'all_ch'
option interface 'VPN_CH'
config rule
option src '192.168.5.32/28'
option lookup 'all_ch'
Which worked for me briefly, but then I couldn't connect from my primary LAN to a host on the secondary LAN br-ch (presumably no route). Then after a round of testing/debugging I reverted and rebooted, and the above stopped working (perhaps some race w/ DHCP since it needs to hand out an address and the host on the secondary LAN br-ch didn't get an address the second time).
And back to a possible mwan3-only solution, my last attempt on that config was:
config interface 'vpn_ch'
option initial_state 'online'
option family 'ipv4'
option track_method 'ping'
option reliability '1'
option count '1'
option size '56'
option max_ttl '60'
option timeout '4'
option interval '10'
option failure_interval '5'
option recovery_interval '5'
option down '5'
option up '5'
option enabled '1'
config member 'ch_m1_w1'
option metric '1'
option weight '1'
option interface 'vpn_ch'
config policy 'ch_policy'
list use_member 'ch_m1_w1'
option last_resort 'unreachable'
config rule 'ch_rule'
option family 'ipv4'
option proto 'all'
option src_ip '192.168.5.32/28'
option sticky '0'
option use_policy 'ch_policy'
option dest_ip '0.0.0.0/0'
Which also failed to route 192.168.5.32/28 to WAN vpn_ch.
Aside from 22.03.4, my related packages are:
# opkg list-installed | egrep 'iptables|pbr|mwan3|policy'
iptables-mod-conntrack-extra - 1.8.7-7
iptables-mod-ipopt - 1.8.7-7
iptables-nft - 1.8.7-7
luci-app-mwan3 - git-23.093.40772-fa4dc75
luci-app-pbr - 1.1.0-1
mwan3 - 2.11.4-1
pbr - 1.0.1-16
Thanks!
That could do it: