Failing to move Wireguard tunnels to "other" WAN in multihomed setup

rmappleby · May 15, 2025, 6:06pm

Apologies for a long and densely written post - unfortunately I need to provide some content for my problem to make sense.

I run a dual-stack environment supporting a small number of internet facing services. The Internet connectivity is provided by an iGLnet MT6000 router. It has a primary PPPoE connection to an ISP (Plusnet) that provides static IPv4 connectivity to the Internet via the 'wan' interface. As Plusnet don't provide IPv6, a 6in4 tunnel to Hurricane Electric (HE) runs over 'wan' to provide me with static IPv6 connectivity to the Internet via the 'wan6' interface. This setup has worked well, but I wish to replace these with a new FTTP ISP (Toob) who will provide faster and cheaper IPv4 and IPv6 connectivity.

Unfortunately even though they have an option of a chargeable static IP, they won't provide rDNS, so instead I'm considering combining the Toob FTTP service with an overlay service from another ISP (Andrews and Arnold) who can provide the features that I want. Toob would become just my bit-mover, and AAISP would become my functional ISP. I'm currently trying to set up a proof of concept of this running on my existing ISPs before committing to 2 year contracts etc.

I have added the L2TP tunnel to AAISP running over the 'wan' connection, providing an additional static IPv4 and IPv6 connection to the Internet via 'aaisp' and 'aaisp6' interfaces. The IPv4 address and IPv6 prefix associated with the AAISP network are static, and together will form my "overlay" network by which I hope all my internet-facing services will be accessed.

So right now, my internet connections are 'wan' and 'wan6' in firewall zone 'wan', and 'aaisp' and 'aaisp6' in firewall zone 'overlay'.

My internal network is then split into two VLANs with vlan filtering on the routers bridge interface, 'br-lan', giving me two devices, 'br-lan.1' and 'br-lan.10'. All my general purpose devices will connect to the router via the 'lan' interface, which is built on 'br-lan.1' (ie VLAN1) and the servers that support my internet-facing services will connect to the router via the 'lan_server' interface, which is built on 'br-lan.10' (ie VLAN10). Both the 'lan' and 'lan_server' interfaces are in the 'lan' firewall zone, along with my two Wireguard tunnels (see later).

My goal for connectivity is that devices connected to the 'lan_server' interface will use 'aaisp' and 'aaisp6' for connectivity, and devices connected to anything else (mostly the 'lan' interface) will use 'wan' and 'wan6' for connectivity. I implemented this as follows:

For IPv4, I set the priority of the 'wan' interface higher than the 'aaisp' interface using the metric option in their interface definitions in /etc/config/network. This makes all IPv4 traffic destined for the Internet prefer the 'wan' interface. I then modify this behaviour with policy based routing, using the OpenWRT PBR tool. I have defined rules to force packets destined for the internet from the 'lan_server' interface to go via the 'aaisp' interface instead.
For IPv6 I have selectively delegated prefixes from 'wan6' and 'aaisp6' such that devices on each VLAN only have one GUA, which is delegated from the interface that I want that VLAN to use for internet access. For devices on 'lan' (VLAN1) it will be part of the delegation from 'wan6', and for devices on 'lan_server' (VLAN10) it will be part of the delegation from 'aaisp6'. The router also delegates ULA prefixes to each VLAN for internal routing. The use of options such as ip6class, ip6assign, ip6hint, ip6ifaceid and prefix_filter ensures that the IPv6 environment is created predictably.

This is implemented, tested, and traffic seems to flow reliably via the correct ISPs.

Now the fun part - I need to move my internet-facing services to the 'overlay'. Mostly this is straightforward; for example, directing inbound HTTP traffic on the AAISP IPv4 address to an appropriate device on 'lan_server' is just a matter of creating the appropriate redirect in /etc/config/firewall.

Where I have hit a problem is that my router hosts two IPv4-only Wireguard VPN connections. The first ('sitetosite') is a site-to-site link to another OpenWRT router at a remote site that is behind CGNAT. This router listens on port 51820 for traffic from that remote site to initiate the tunnel. The other ('mobiles') supports remote connection by two roaming devices (a so called "roadwarrior" configuration) and listens on port 51821 for those roaming devices to make contact. As I cannot reliably know the IP addresses of any of the peer devices at configuration time, this router needs to act as a pseudo-server for all the tunnels.

Currently both Wireguard tunnels are within the 'lan' firewall zone, and use 'wan' in the 'wan' firewall zone to communicate with the internet, and both work correctly. For example, I can connect one of the road warrior devices and I have working local DNS resolution, and access to everything in the 'lan' firewall zone.

I now need to move these Wireguard tunnels from the 'wan' interface to the 'aaisp' interface, and I'm struggling. I had naively thought I would be able to just tweak the profiles of the peers to point them to the 'aaisp' IPv4 address, but so far I have completely failed to get it working. I can see the initial handshaking & keepalive packets turning up on 'aaisp' but I never see WG responding to them, and the tunnels never come up.

Thanks for reading this far! Any assistance would be greatly appreciated!

/etc/config/dhcp

config dnsmasq
	option domainneeded '1'
	option localise_queries '1'
	option rebind_protection '1'
	option rebind_localhost '1'
	option local '/site1.lan/'
	option domain 'site1.lan'
	option expandhosts '1'
	option cachesize '1000'
	option authoritative '1'
	option readethers '1'
	option leasefile '/tmp/dhcp.leases'
	option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
	option localservice '0'
	list rebind_domain 'site1.lan'
	list rebind_domain 'site2.lan'
	list rebind_domain 'ik40.home'		# Alcatel NDIS modem web interface
	option dnssec '1'
	option ednspacket_max '1232'
	list server '/*.site2.lan/192.168.43.1'
	list address '/ik40.home/192.168.1.1'

config dhcp 'lan'
	option interface 'lan'
	option start '100'
	option limit '150'
	option leasetime '5m'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option prefix_filter '2001:4XXX:XXa2::/64'
	option preferred_lifetime '5m'
	list domain 'site1.lan'
	list domain 'site2.lan'
	list ntp 'uk.pool.ntp.org'

config dhcp 'lan_server'
	option interface 'lan_server'
	option start '100'
	option limit '150'
	option leasetime '5m'
	option dhcpv4 'server'
	option dhcpv6 'server'
	option ra 'server'
	list ra_flags 'managed-config'
	list ra_flags 'other-config'
	option prefix_filter '2001:8XXX:XXee:4d01::/64'
	option preferred_lifetime '5m'
	list domain 'site1.lan'
	list domain 'site2.lan'
	list ntp 'uk.pool.ntp.org'

config dhcp 'wan'
	option interface 'wan'
	option ignore '1'

config odhcpd 'odhcpd'
	option maindhcp '0'
	option leasefile '/tmp/hosts/odhcpd'
	option leasetrigger '/usr/sbin/odhcpd-update'
	option loglevel '4'

/etc/config/network (with redactions)

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fdb4::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'lan2'
	list ports 'lan3'
	list ports 'lan4'
	list ports 'lan5'

config device
	option name 'lan1'
	option macaddr '94:83:c4:a2:e8:df'

config device
	option name 'lan2'
	option macaddr '94:83:c4:a2:e8:df'

config device
	option name 'lan3'
	option macaddr '94:83:c4:a2:e8:df'

config device
	option name 'lan4'
	option macaddr '94:83:c4:a2:e8:df'

config device
	option name 'lan5'
	option macaddr '94:83:c4:a2:e8:df'

config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'lan2:t'
	list ports 'lan3:u*'

config bridge-vlan
	option device 'br-lan'
	option vlan '10'
	list ports 'lan2:t'
	list ports 'lan4:u*'

config interface 'lan_server'
	option macaddr '02:11:22:33:44:0A'  # Unique MAC for VLAN 10
	option proto 'static'
	option device 'br-lan.10'
	list ipaddr '192.168.251.1/24'
	list ip6class 'local'
	list ip6class 'aaisp6'
	list dns_search 'site1.lan'
	list dns_search 'site2.lan'
	option ip6assign '64'
	option delegate '0'
	option ip6hint '1'
	option ip6ifaceid '::1'

config interface 'lan'
	option macaddr '02:11:22:33:44:01'  # Unique MAC for VLAN 1
	option proto 'static'
	option device 'br-lan.1'
	option ipaddr '192.168.252.1/22'
	list ip6class 'local'
	list ip6class 'wan6'
	list dns_search 'site1.lan'
	list dns_search 'site2.lan'
	option ip6assign '64'
	option delegate '0'
	option ip6hint '0'
	option ip6ifaceid '::1'

config device
	option name 'eth1'
	option macaddr '94:83:c4:a2:e8:dd'

config interface 'wan'
	option device 'eth1'
	option proto 'pppoe'
	option username ' '
	option password ' '
	option defaultroute '1'		# Ensure this is an IPv4 provider with a default route
	option ipv6 '0'			# Plusnet doesnt have IPv6, so use Hurricane Electric, below
	option peerdns '0'
	option metric '128'		# Set higher priority than aaisp
	list dns '9.9.9.9'
	list dns '149.112.112.112'
	list dns '2620:fe::fe'
	list dns '2620:fe::9'
	list dns_search 'site1.lan'
	list dns_search 'site2.lan'

config interface 'wan6'
	option proto '6in4'
	option metric '128'		# Priority doesnt seem to matter much with IPv6
	option ip6class 'wan6'		# ?????
	option peeraddr '216.66.80.26'
	option ip6addr '2001:4XXX:XX08:141::2/64'
	list ip6prefix '2001:4XXX:XXa2::/48'
	option defaultroute '1'		# Ensure this is an IPv6 provider with a default route
	option sourcefilter '0'		# Turn off source filtering on default routes

config interface 'aaisp'
	option proto 'l2tp'
	option server 'l2tp.aa.net.uk'
	option username ' '
	option password ' '
	option peerdns '0'
	option metric '512'		# Set lower priority than wan
	option checkup_interval '10'
	option defaultroute '1'		# Ensure we have a default route
	option ipv6 '1'
	option delegate '0'

config interface 'aaisp6'
	option proto 'dhcpv6'
	option reqaddress 'try'
	option reqprefix 'auto'
	option sourcefilter '0'		# Turn off source filtering on default routes
	option defaultroute '1'
	option ip6ifaceid '::1'
	option peerdns '0'
	option metric '512'		# Priority doesnt seem to matter much with IPv6
	option ip6class 'aaisp6'
	option device '@aaisp'
	option norelease '0'
	option sourcefilter '0'

config interface 'site2site'
	option proto 'wireguard'
	option private_key ' '
	option listen_port '51820'
	list addresses '192.168.100.1/30'

config wireguard_site2site 'wgclient'
	option persistent_keepalive '20'
	option public_key ' '
	option description 'Site 2'
	option route_allowed_ips '1'
	list allowed_ips '192.168.100.2/32'
	list allowed_ips '192.168.43.0/24'
	list allowed_ips '192.168.1.1/32'

config interface 'ModemWeb'
	option proto 'static'
	option device 'eth1'
	list ipaddr '192.168.155.2/24'

config interface 'mobiles'
	option proto 'wireguard'
	option private_key ' '
	option listen_port '51821'
	list addresses '192.168.101.1/24'

config wireguard_mobiles
	option description 'Phone'
	option public_key ' '
	option private_key ' '
	option preshared_key ' '
	option persistent_keepalive '20'
	list allowed_ips '192.168.101.2/32'

config wireguard_mobiles
	option description 'iPad'
	option public_key ' '
	option private_key ' '
	option preshared_key ' '
	option persistent_keepalive '20'
	list allowed_ips '192.168.101.3/32'

/etc/config/firewall

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'
	list network 'lan_server'
	list network 'site2site'
	list network 'mobiles'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'ModemWeb'
	list network 'wan'
	list network 'wan6'

config forwarding
	option src 'lan'
	option dest 'wan'

config zone
	option name 'overlay'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	list network 'aaisp'
	list network 'aaisp6'

config forwarding
	option src 'lan'
	option dest 'overlay'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'overlay'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-Ping'
	option src 'overlay'
	option proto 'icmp'
	option family 'ipv4'
	option target 'ACCEPT'
	list icmp_type 'echo-request'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'overlay'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'overlay'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'overlay'
	option proto 'icmp'
	option family 'ipv6'
	option target 'ACCEPT'
	list src_ip 'fe80::/10'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'overlay'
	option proto 'icmp'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'
	list icmp_type 'bad-header'
	list icmp_type 'destination-unreachable'
	list icmp_type 'echo-reply'
	list icmp_type 'echo-request'
	list icmp_type 'neighbour-advertisement'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'packet-too-big'
	list icmp_type 'router-advertisement'
	list icmp_type 'router-solicitation'
	list icmp_type 'time-exceeded'
	list icmp_type 'unknown-header-type'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'overlay'
	option dest '*'
	option proto 'icmp'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'
	list icmp_type 'bad-header'
	list icmp_type 'destination-unreachable'
	list icmp_type 'echo-reply'
	list icmp_type 'echo-request'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'unknown-header-type'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Wireguard-site2site-inbound-wan'
	list proto 'udp'
	option dest_port '51820'
	option target 'ACCEPT'
	option src 'wan'

config rule
	option name 'Wireguard-mobiles-inbound-wan'
	list proto 'udp'
	option src 'wan'
	option dest_port '51821'
	option target 'ACCEPT'

config include 'pbr'
	option fw4_compatible '1'
	option type 'script'
	option path '/usr/share/pbr/firewall.include'

/etc/config/pbr

config pbr 'config'
	option enabled '1'
	option verbosity '2'
	option strict_enforcement '1'
	option resolver_set 'dnsmasq.nftset'
	list resolver_instance '*'
	option ipv6_enabled '1'
	option boot_timeout '30'
	option rule_create_option 'add'
	option procd_reload_delay '1'
	option webui_show_ignore_target '0'
	option nft_rule_counter '1'
	option nft_set_auto_merge '1'
	option nft_set_counter '1'
	option nft_set_flags_interval '1'
	option nft_set_flags_timeout '0'
	option nft_set_policy 'performance'
	list webui_supported_protocol 'all'
	list webui_supported_protocol 'tcp'
	list webui_supported_protocol 'udp'
	list webui_supported_protocol 'tcp udp'
	list webui_supported_protocol 'icmp'

config policy
	option name 'lan_server via aaisp'
	option src_addr '192.168.251.0/24'
	option interface 'aaisp'
	option dest_addr '!192.168.0.0/16'

config policy
	option name 'Default IPv4 via wan'
	option src_addr '192.168.0.0/16'
	option interface 'wan'
	option dest_addr '!192.168.0.0/16'

ip route show

default via 195.166.130.254 dev pppoe-wan proto static metric 128 
default via 81.187.81.187 dev l2tp-aaisp proto static metric 512 
81.187.81.187 dev l2tp-aaisp proto kernel scope link src 81.187.73.113 
192.168.1.1 dev site2site proto static scope link 
192.168.43.0/24 dev site2site proto static scope link 
192.168.100.0/30 dev site2site proto kernel scope link src 192.168.100.1 
192.168.100.2 dev site2site proto static scope link 
192.168.101.0/24 dev mobiles proto kernel scope link src 192.168.101.1 
192.168.155.0/24 dev eth1 proto kernel scope link src 192.168.155.2 
192.168.251.0/24 dev br-lan.10 proto kernel scope link src 192.168.251.1 
192.168.252.0/22 dev br-lan.1 proto kernel scope link src 192.168.252.1 
194.4.172.12 via 195.166.130.254 dev pppoe-wan proto static metric 128 
195.166.130.254 dev pppoe-wan proto kernel scope link src 212.159.18.7 
216.66.80.26 via 195.166.130.254 dev pppoe-wan proto static metric 128

wg show

interface: site2site
  public key: fu0rw+gsAvgowg4m1YoIITwW583U8lZMIlveCbPyNXk=
  private key: (hidden)
  listening port: 51820

peer: LNZuCW8v7BhsGn9mwR14GfqSxaRcHbDa+i1w3kVFLHs=
  endpoint: 148.252.140.158:7318
  allowed ips: 192.168.100.2/32, 192.168.43.0/24, 192.168.1.1/32
  latest handshake: 23 seconds ago
  transfer: 488.99 KiB received, 482.80 KiB sent
  persistent keepalive: every 20 seconds

interface: mobiles
  public key: 1C3XojMKWiNqDxlgiE7eIXQu9aUsxek404LKH8fTvDc=
  private key: (hidden)
  listening port: 51821

peer: B63rtbkk7tvtpI3/AfQSpyK02KO/DmgRKnPux/8cEQY=
  preshared key: (hidden)
  endpoint: 82.132.232.25:29401
  allowed ips: 192.168.101.2/32
  latest handshake: 1 minute, 6 seconds ago
  transfer: 500.91 KiB received, 3.91 MiB sent
  persistent keepalive: every 20 seconds

peer: JMB7RTop4Mn1zaKnuLvMXEKiRGUdjqinJ1xFSaZmFAg=
  preshared key: (hidden)
  allowed ips: 192.168.101.3/32
  persistent keepalive: every 20 seconds

Edit: Apologies, I had the name of the firewall zone wrong in a couple of places. Now fixed.

egc · May 16, 2025, 7:33am

I must confess I did not study your post in detail so apologies as the following might be rubbish but your WG traffic seems to enter via an interface which is not the default (l2tp-aaisp)
So you have to use PBR to also route WG traffic out via the same interface.

PBR (on recent 1.1.8 versions) actually routes WG "server" traffic automatically back via the WAN interface (you can see that with ip rule show) first step is to disable that by adding the WG interfaces to the Ignored interfaces list, this will disable the automatic route back via the WAN.
see: https://docs.openwrt.melmac.net/pbr/1.1.8/#WireGuardServerUseCases

Next step is to manually route the sport of both WG interface back via the interface it comes in (l2tp-aaisp), a quick test is to add the sport of e.g. your mobile server, 51821 to pbr table for l2tp-aaisp e.g. pbr_l2tp-aaisp
So just add for testing ip rule add sport 51821 table pbr_l2tp-aaisp

See for some inspiration: https://github.com/egc112/OpenWRT-egc-add-on/tree/main/pbr-via-wan

krazeh · May 16, 2025, 10:57am

Out of curiosity, what service(s) are you running that need you to be able to set the rDNS record?

rmappleby · May 16, 2025, 2:27pm

My own mailserver. Checking that email is being delivered from a server with a valid reverse DNS lookup greatly reduces spam, and can significantly improve performance as messages from non-rDNS IP addresses can be trivially rejected by the receiving mail server, rather than requiring more complex filtering later. So without proper rDNS much of my outbound mail would be indeliverable.

krazeh · May 16, 2025, 2:38pm

That makes sense. I'm in a similar situation. But would it not be simpler all round to just use the l2tp tunnel for the mail server and have everything else use your main ISP?

The business light offering from AAISP would probably be sufficient for just email (as it's only SMTP traffic that will need to go over it). It's also cheap enough that you'd be able to get a static IPv4 from toob without paying anymore overall than your current plan.

rmappleby · May 16, 2025, 4:29pm

That's an interesting way to skin the cat! I like your lateral thinking!

My other reason for going this way was a certain concern about how financially secure the altnet providers are here in the UK. Toob seem more secure than most, but if they were to run into problems, I'd like to be in a position where I can swap suppler easily, without reconfiguring everything again - especially the mailserver!

Splitting the configuration like this means I can swap a new provider in almost trivially, but of course, the compromise is that I'm putting more eggs in the AAISP basket.

However, I know several people who have AAISP as their ISP, and I use them myself for my VoIP service, and they really are the techies ISP. There is something incredibly refreshing to be chatting to someone who is notionally in their "sales" team, who clearly knows the technology, and how they have implemented it, inside out.

And that has not been my experience when trying to talk to the people at Toob, sadly.

krazeh · May 16, 2025, 4:48pm

I appreciate those concerns. I had similar when moving to an alt net provider (brsk), but they have been surprisingly helpful whenever I've needed to contact their support team. Had an issue with spamhaus (My IP was on the dynamic IP block list despite being static) when I first moved over to them and within a day or so of reporting it they had registered with spamhaus, proven they owned the IP blocks in question, and corrected the data. They also happily registered rDNS for my IPv4 address as well as the IPv6 address assigned to my mail server.

I guess I'd just be mindful that it kinda goes both ways. For example, you could end up at a new provider who does support rDNS and then have to decide whether to keep paying AAISP for something that was unnecessary or reconfigure things anyway.

Obviously at the end of the day it's up to you which way you go, but I generally subscribe to the idea that the least complex method to achieve a given outcome (especially in IT) is probably the best one. I'd rather take simple now than over-complicated just in case of something that might happen in the future.

(And apologies for slightly derailing the thread. Interested tho to see what you end up with, even if its just doing email over AAISP. I never got further than it being a theoretical idea for my setup).

rmappleby · May 16, 2025, 4:55pm

@egc I think you have hit the nail on the head.

The current routing environment with PBR as I'd configured it:

Routing rules

# ip rule show
0:	from all lookup local
29996:	from all sport 51821 lookup pbr_wan
29997:	from all sport 51820 lookup pbr_wan
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_aaisp
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
32766:	from all lookup main
32767:	from all lookup default

# ip route show table pbr_wan
default via 195.166.130.254 dev pppoe-wan 
192.168.1.1 dev site2site proto static scope link 
192.168.43.0/24 dev site2site proto static scope link 
192.168.100.0/30 dev site2site proto kernel scope link src 192.168.100.1 
192.168.100.2 dev site2site proto static scope link 
192.168.101.0/24 dev mobiles proto kernel scope link src 192.168.101.1 
192.168.155.0/24 dev eth1 proto kernel scope link src 192.168.155.2 
192.168.251.0/24 dev br-lan.10 proto kernel scope link src 192.168.251.1 
192.168.252.0/22 dev br-lan.1 proto kernel scope link src 192.168.252.1 

# ip route show table pbr_aaisp
default via 81.187.81.187 dev l2tp-aaisp

So those two sport rules will cause the packet to go out on the wan interface. Which is fine if that's the interface my inbound WG packets are entering by. But switch the inbound interface to 'aaisp' and the outbound reply packets still go to 'wan' interface, and are probably dropped.

So if I put mobiles into the Ignored interfaces list, it should remove 29996: from all sport 51821 lookup pbr_wan from the rules, but still let the reply packets run out of 'wan' because it's higher priority then 'aaisp':

Routing rules

# ip rule show
0:	from all lookup local
29997:	from all sport 51820 lookup pbr_wan
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_aaisp
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
32766:	from all lookup main
32767:	from all lookup default
# ip r s table pbr_wan
default via 195.166.130.254 dev pppoe-wan 
192.168.1.1 dev site2site proto static scope link 
192.168.43.0/24 dev site2site proto static scope link 
192.168.100.0/30 dev site2site proto kernel scope link src 192.168.100.1 
192.168.100.2 dev site2site proto static scope link 
192.168.101.0/24 dev mobiles proto kernel scope link src 192.168.101.1 
192.168.155.0/24 dev eth1 proto kernel scope link src 192.168.155.2 
192.168.251.0/24 dev br-lan.10 proto kernel scope link src 192.168.251.1 
192.168.252.0/22 dev br-lan.1 proto kernel scope link src 192.168.252.1 
# ip r s table pbr_aaisp
default via 81.187.81.187 dev l2tp-aaisp 
192.168.1.1 dev site2site proto static scope link 
192.168.43.0/24 dev site2site proto static scope link 
192.168.100.0/30 dev site2site proto kernel scope link src 192.168.100.1 
192.168.100.2 dev site2site proto static scope link 
192.168.101.0/24 dev mobiles proto kernel scope link src 192.168.101.1 
192.168.155.0/24 dev eth1 proto kernel scope link src 192.168.155.2 
192.168.251.0/24 dev br-lan.10 proto kernel scope link src 192.168.251.1 
192.168.252.0/22 dev br-lan.1 proto kernel scope link src 192.168.252.1

Which is exactly what we see. The changes to the pbr_aaisp routing table are more extensive that I expected, but they mirror the pbr_wan table, apart from the default route, which makes sense.

But this still lets the return traffic to the mobiles WG tunnel attempt to follow the default in the main table, which is to the wan interface. So we do need your extra rule. Which made me wonder, what if I put such a rule in PBR to make anything from src_port 51821 go to aaisp?

So I added:

config policy
        option name 'WireGuard mobile control'
        option proto 'udp'
        option interface 'aaisp'
        option src_port '51821'

to /etc/config/pbr, and issued service mbr reload but I seem to be seeing an error with pbr:

# service pbr reload
Using wan interface (on_start): wan [✓]
Found wan gateway (on_start): 195.166.130.254 [✓]
Using wan6 interface (on_start): wan6 [✓]
Found wan6 gateway (on_start): 2001:4XXX:XX08:141::2/64 [✓]
Setting up routing for 'wan/pppoe-wan/195.166.130.254/2001:4XXX:XX08:141::2/64' [✓]
Setting up routing for 'aaisp/l2tp-aaisp/81.187.81.187/2001:8XXX:XX11:1111:0:ffff:51bb:4971/128' [✓]
Routing 'WireGuard mobile control' via aaisp [✓]
Routing 'lan_server via aaisp' via aaisp /etc/rc.common: eval: line 182: is_domain_negative: not found
/etc/rc.common: eval: line 182: is_ipv4_negative: not found
/etc/rc.common: eval: line 182: is_ipv6_negative: not found
[✓]
Routing 'Default IPv4 via wan' via wan /etc/rc.common: eval: line 182: is_domain_negative: not found
/etc/rc.common: eval: line 182: is_ipv4_negative: not found
/etc/rc.common: eval: line 182: is_ipv6_negative: not found
[✓]
Installing fw4 nft file [✓]
Setting interface trigger for wan [✓]
Setting interface trigger for aaisp [✓]

pbr 1.1.8-r10 monitoring interfaces: wan aaisp 
pbr 1.1.8-r10 (fw4 nft file mode) started with gateways:
wan/pppoe-wan/195.166.130.254/2001:4XXX:XX08:141::2/64 [✓]
aaisp/l2tp-aaisp/81.187.81.187/2001:8XXX:XX11:1111:0:ffff:51bb:4971/128

Checking the actual routing, and I can see that no rule has been inserted for my new configuration stanza in /etc/config/pbr:

# ip rule show
0:	from all lookup local
29997:	from all sport 51820 lookup pbr_wan
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_aaisp
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
32766:	from all lookup main
32767:	from all lookup default

But if I manually add the rule, as you recommended ip rule add sport 51821 table pbr_aaisp, and then check the routing rules and routes, I see the expected configuration.

Routing rules and tables

# ip rule show
0:	from all lookup local
29996:	from all sport 51821 lookup pbr_aaisp
29997:	from all sport 51820 lookup pbr_wan
29998:	from all fwmark 0x20000/0xff0000 lookup pbr_aaisp
30000:	from all fwmark 0x10000/0xff0000 lookup pbr_wan
32766:	from all lookup main
32767:	from all lookup default

# ip route show table pbr_aaisp
default via 81.187.81.187 dev l2tp-aaisp 
192.168.1.1 dev site2site proto static scope link 
192.168.43.0/24 dev site2site proto static scope link 
192.168.100.0/30 dev site2site proto kernel scope link src 192.168.100.1 
192.168.100.2 dev site2site proto static scope link 
192.168.101.0/24 dev mobiles proto kernel scope link src 192.168.101.1 
192.168.155.0/24 dev eth1 proto kernel scope link src 192.168.155.2 
192.168.251.0/24 dev br-lan.10 proto kernel scope link src 192.168.251.1 
192.168.252.0/22 dev br-lan.1 proto kernel scope link src 192.168.252.1 

# ip route show table pbr_wan
default via 195.166.130.254 dev pppoe-wan 
192.168.1.1 dev site2site proto static scope link 
192.168.43.0/24 dev site2site proto static scope link 
192.168.100.0/30 dev site2site proto kernel scope link src 192.168.100.1 
192.168.100.2 dev site2site proto static scope link 
192.168.101.0/24 dev mobiles proto kernel scope link src 192.168.101.1 
192.168.155.0/24 dev eth1 proto kernel scope link src 192.168.155.2 
192.168.251.0/24 dev br-lan.10 proto kernel scope link src 192.168.251.1 
192.168.252.0/22 dev br-lan.1 proto kernel scope link src 192.168.252.1

To test this, I switched my firewall Traffic Rule from 'wan' to 'overlay', altered the roadwarrior client configuraton to point to the overlay IP address, started WG on the roaming device, and of course it works perfectly! Thank you for the help!

One question that remains: why couldn't I set that rule within pbr itself? It is clearly capable of creating such rules, as the rules we had to disable initially are identical to what I need apart from the interface name, so is this expected behaviour for pbr, or should I raise an issue?

I also looked at https://github.com/egc112/OpenWRT-egc-add-on/tree/main/pbr-via-wan where I see you propose other ways to get the rule automatically installed, and I note you mention for adding rules into /etc/config/network, that "Unfortunately this does not yet work for source port (sport), the PR for that is pending". And indeed, when I check the firewall rules in my 24.10.0 install, there is no way to enter port based rules. Do you have any further information on progress of that PR, as it would seem like the ideal way forward if pbr doesnt intend to support this itself.

rmappleby · May 16, 2025, 5:06pm

This is very true. But another nice thing about the L2TP tunnel to AAISP is that there is no long term commitment (its PAYG a month at a time), so I can leave at any time with no penalty. I am slightly concerned at the complexity involved in getting it set up on OpenWRT, but in practice that's been more because of my lack of networking skills than any real difficulty in the actual approach.

I've learned a lot while doing it, which has its benefits too!

egc · May 16, 2025, 6:11pm

Unfortunately the maintainer of that package is very busy elsewhere so a lot of interesting PR's for netifd are still pending

Anyway very glad we could solve this

system · May 26, 2025, 6:11pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.