Need help configuring failover VPN with exceptions

Hiya fella's

I need to configure a router for a family member, that is not technical at all. So I have to make sure its stable and works. After some trial and error with the PBR app, I came to the conclusion that it is 'not' a good option in this particular case. Its slow to boot and for some reason does not always work. So I was looking for an alternate solution.

I stumbled on this, but its unclear to me if this is still the PBR app or something different.
https://openwrt.org/docs/guide-user/network/routing/pbr_netifd

The use case:

  • all wan traffic blocked with exceptions for
    • email smtp imap over ssl to myisp.com (dont know which yet, so 465 587 993 995)
  • all other traffic over vpn A
  • all other traffic over vpn B if A is down

Thanks for helping out.

The latter.
Generally, most of the conditions can be met, except the last one with the failover.
For that you either monitor it yourself with a script and change the gateway or look into mwan3.

1 Like

I finally made some time to do this, but cannot figure out what is wrong and really need some help :blush:

updated use case:

  • all wan traffic blocked with exceptions for
    • (ipset) ViaPlay that contains all streaming addresses (+ip ipleak.net)
  • all other traffic over vpn A
/etc/config/firewall:

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option drop_invalid '1'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	option family 'ipv4'
	list network 'lan'
	option masq '1'

config zone
	option name 'wan'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	option family 'ipv4'
	option input 'DROP'
	option forward 'DROP'
	list network 'wan'

config forwarding
	option src 'lan'
	option dest 'wan'

config zone
	option name 'openvpn'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	list network 'openvpn'

config forwarding
	option dest 'openvpn'
	option src 'lan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'isp-to-wan'
	option src 'lan'
	option dest 'wan'
	option target 'ACCEPT'
	option family 'ipv4'
	option ipset 'isp'
	list proto 'all'

config rule
	option name 'viaplay-to-wan'
	option src 'lan'
	option dest 'wan'
	option target 'ACCEPT'
	option family 'ipv4'
	option ipset 'viaplay'
	list proto 'all'

config rule
	option name 'lan-to-openvpn'
	option src 'lan'
	option dest 'openvpn'
	option target 'ACCEPT'
	list proto 'all'

config rule
	option name 'catch-all-reject'
	option src 'lan'
	option dest 'wan'
	option target 'REJECT'
	list proto 'all'

config ipset
	option name 'isp'
	option comment 'Internet Service Provider'
	option family 'ipv4'
	list match 'dest_ip'
	option loadfile '/etc/luci-uploads/isp.ipv4'

config ipset
	option name 'viaplay'
	option family 'ipv4'
	option loadfile '/etc/luci-uploads/viaplay.ipv4'
	option counters '1'
	option comment 'ViaPlay Streaming Service'
	list match 'dest_ip'

No need for masquerade in lan.

You probably need mtu fix in openvpn zone.

These 2 are not needed if you want to block lan->wan traffic. The absence of a forwarding means that it will be blocked, thus no need for a blocking rule.

Also not needed since you have a lan-to-openvpn rule.

But again these are just firewall rules, not ip routing rules. If you don't have the latter, you'll end up with firewall blocking the traffic that doesn't go to the correct gateway,

Thank you, that makes sense, so those need to be added for wan and openvpn under "routing/ipv4-rules" ? And do I need to add route-nopull to openvpn config ?

That's right, but you need also to provide a routing table for the non-default gateway.

The modern way is to add to the openvpn config:

pull-filter ignore "redirect-gateway"
# If you also have IPv6 then add:
#pull-filter ignore "redirect-gateway ipv6"

You need some form of PBR as already mentioned by @trendy :
https://openwrt.org/docs/guide-user/network/routing/pbr

I was looking at Route LAN to VPN by IP set, however I cannot get it to work.

What I'm reading is that it tags all packets not in ipset wan(6) and then uses the tag to route packets over table 2 that points to the vpn. Based on this I have:

  • execute commands from "Route LAN to VPN by IP set"
  • added the IP from dnsleaktest.com to ipset
  • added VPN/WAN to be allowed to forward from LAN

I expected dnsleaktest.com to show WAN-ip and ipleak.net to show VPN-ip. But it does not work. They both show WAN-ip.

# cat /etc/config/network  | grep -v 'mac\|prefix'

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1'

config device
	option name 'eth1'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option ip4table '1'
	option ip6table '1'

config device
	option name 'eth0'

config interface 'wan'
	option device 'eth0'
	option proto 'dhcp'

config interface 'vpn'
	option proto 'none'
	option device 'tun0'
	option ip4table '2'
	option ip6table '2'

config rule 'lan_vpn'
	option in 'lan'
	option mark '1'
	option lookup '2'
	option priority '30000'

config rule6 'lan_vpn6'
	option in 'lan'
	option mark '1'
	option lookup '2'
	option priority '30000'
# cat /etc/config/firewall 

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config zone 'vpn'
	option name 'vpn'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	list network 'vpn'

config ipset 'wan_set'
	option name 'wan'
	option family 'ipv4'
	list match 'net'
	list entry '23.239.16.110'

config rule 'lan_mark'
	option name 'Mark-LAN-VPN'
	option src 'lan'
	option dest '*'
	option ipset '!wan dest'
	option proto 'all'
	option family 'ipv4'
	option set_mark '0x1'
	option target 'MARK'

config ipset 'wan_set6'
	option name 'wan6'
	option family 'ipv6'
	option match 'net'

config rule 'lan_mark6'
	option name 'Mark-LAN-VPN'
	option src 'lan'
	option dest '*'
	option ipset '!wan6 dest'
	option proto 'all'
	option family 'ipv6'
	option set_mark '0x1'
	option target 'MARK'

config forwarding
	option src 'lan'
	option dest 'vpn'

config forwarding
	option src 'lan'
	option dest 'wan'

Have you checked that netifd actually has setup an openvpn routing table:

ip route show
ip route show table 2
ip route show table all

I did not :

# ip route show table 2
Error: ipv4: FIB table does not exist.
Dump terminated
#
# reserved values
#
128	prelocal
255	local
254	main
253	default
0	unspec
#
# custom values
#
2	vpn

As OpenVPN does setup its own interface it is possible that netifd cannot do that.

Several ways out:

  1. Use WireGuard instead of OpenVPN
  2. Reverse the situation, route everything by default via OpenVPN and use pbr to route via the wan
  3. Use the PBR app, that can work with OpenVPN

My preferred option is 2, so I restored from backup (basic configuration with openvpn as default gateway)

  1. Added table entry: echo '2 wan' >>/etc/iproute2/rt_tables
  2. Swapped VPN for WAN in all commands
  3. Removed negation from ipset
echo '2	wan' >>/etc/iproute2/rt_tables
for IPV in 4 6
do
	uci -q delete firewall.wan_set${IPV%4}
	uci set firewall.wan_set${IPV%4}="ipset"
	uci set firewall.wan_set${IPV%4}.name="wan${IPV%4}"
	uci set firewall.wan_set${IPV%4}.family="ipv${IPV}"
	uci set firewall.wan_set${IPV%4}.match="net"
	uci -q delete firewall.lan_mark${IPV%4}
	uci set firewall.lan_mark${IPV%4}="rule"
	uci set firewall.lan_mark${IPV%4}.name="Mark-LAN-WAN"
	uci set firewall.lan_mark${IPV%4}.src="lan"
	uci set firewall.lan_mark${IPV%4}.dest="*"
	uci set firewall.lan_mark${IPV%4}.ipset="wan${IPV%4} dest"
	uci set firewall.lan_mark${IPV%4}.proto="all"
	uci set firewall.lan_mark${IPV%4}.family="ipv${IPV}"
	uci set firewall.lan_mark${IPV%4}.set_mark="0x1"
	uci set firewall.lan_mark${IPV%4}.target="MARK"
	uci set network.lan.ip${IPV}table="1"
	uci set network.wan.ip${IPV}table="2"
	uci -q delete network.lan_wan${IPV%4}
	uci set network.lan_wan${IPV%4}="rule${IPV%4}"
	uci set network.lan_wan${IPV%4}.in="lan"
	uci set network.lan_wan${IPV%4}.mark="1"
	uci set network.lan_wan${IPV%4}.lookup="2"
	uci set network.lan_wan${IPV%4}.priority="30000"
done
uci commit firewall
uci commit network
service firewall restart
service network restart

After some more frustration, I think I see some light in the dark tunnel.

My default wan route is gone, after manually setting it with ip route add default via x.x.x.x and restarting openvpn I get what I want.

So the question is now, why does it get removed ?

That is how netifd works (and one of the reasons netifd was not used in the pbr app anymore )

The default route should go via your vpn so the main routing table should not need a default route via the wan according to the netifd paradigm.
What I am not sure about is if there will be a proper escape route for the vpn endpoint via the wan

You can check how things shoul work together with

ip route show
ip rule show

If that default route is removed, everything dies. OpenVPN is stacked on top of it isn't it ?

It is not stacked on top per se but there should be a route for the VPN endpoint (server) via the wan, not sure how that is handled by netifd and OpenVPN.

The examples of netifd are all made with WireGuard as VPN

You can have multiple gateways in the same routing table if you assign different metrics. This way it will not be overwritten by OpenVPN.

I seem to have an ipset that is not-existent called "wan dest". First I thought it was a mistake, but its not. If "wan" is used it will not work if "wan dest" is manually entered it does :crazy_face:

Very confused right now.

firewall.lan_mark.name='Mark-LAN-WAN'
firewall.lan_mark.src='lan'
firewall.lan_mark.dest='*'
firewall.lan_mark.ipset='wan dest' <-----------------
firewall.lan_mark.family='ipv4'
firewall.lan_mark.set_mark='0x1'
firewall.lan_mark.target='MARK'
firewall.lan_mark.proto='all'
firewall.@ipset[0]=ipset
firewall.@ipset[0].name='wan'
firewall.@ipset[0].family='ipv4'
firewall.@ipset[0].match='net'
firewall.@ipset[0].loadfile='/var/run/wan.ipv4'
firewall.@ipset[1]=ipset
firewall.@ipset[1].name='wan6'
firewall.@ipset[1].family='ipv6'
firewall.@ipset[1].match='net'
firewall.@ipset[1].loadfile='/var/run/wan.ipv6'
firewall.lan_mark.ipset='wan dest'

edit: its to mark the ipset to be used as destination address isn't it ?

It's used to mark packets from lan to any zone matching the addresses in the ipset.

I understand what the rule is supposed to do, what I did not understand was why

firewall.lan_mark.ipset='wan dest' <----------------- works
firewall.lan_mark.ipset='wan' <----------------- does not

When selecting an ipset in luci its purpose remains unassigned (instead of assigning it to destination address which was what I wanted.) In retrospect that seems only logical because it might as well be used as a source address ipset. Hence the added dest.

At least that is what I think is the case.

Anyway, I had it working by disabling OpenVPN on startup, setting the default route and then starting OpenVPN manually, that came with its own set of problems. I had to modify /etc/rc.local to add a default route to the modem., but that did not always work reliable. So that became as ugly as:

while ! ip route add default via <modem ip address>;do sleep 1;done
/etc/init.d/openvpn start

exit 0

Than I had to modify /etc/iproute2/rt_tables to add table 2

# custom values
#
2       wan

As of now, I have two working situations, one with OpenVPN and one with WireGuard. Now I just have to figure out exactly what steps were actually needed.