Wireguard PBR routing issue

Hello,

I'm facing an issue with a configuration of wireguard : I use 2 routers with a LAN behind each router. Both are connected (as 'client' peers) to a server (used as default gateway for outbound traffic)
I use PBR to let local router NTP traffic going through wan interface (to address issue with NTP sync).
The configuration works fine (I can communicate from LAN to LAN devices), but i can't reach routers from remote LAN. I've figured out that the router receive the packets, but respond on wan interface...

Here is the configuration of one of the router :

cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdd3:8237:xxxx::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '10.17.3.254'
        option netmask '255.255.255.0'
        option ip6assign '60'

config device
        option name 'wan'
        option macaddr '26:f5:a2:xx:xx:xx'

config interface 'wan'
        option device 'wan'
        option proto 'dhcp'

config interface 'wan6'
        option device 'wan'
        option proto 'dhcpv6'

config interface 'wg'
        option proto 'wireguard'
        option private_key 'xxx'
        list addresses '172.20.8.11/24'

config wireguard_wg
        option public_key 'xxx'
        list allowed_ips '0.0.0.0/0'
        option route_allowed_ips '0'
        option persistent_keepalive '25'
        option endpoint_host 'xxx'
        option endpoint_port '51820'
cat /etc/config/pbr
config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'none'
        option ipv6_enabled '0'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'

config policy
        option name 'LAN traffic through wg'
        option interface 'wg'
        option src_addr '10.17.3.0/24'

config policy
        option name 'NTP traffic through wan'
        option src_addr '127.0.0.1/8'
        option dest_port '123'
        option proto 'udp'
        option interface 'wan'
ip route list table pbr_wg
default via 172.20.8.11 dev wg
10.17.3.0/24 dev br-lan proto kernel scope link src 10.17.3.254

ip route list table pbr_wan
default via 192.168.18.254 dev wan
10.17.3.0/24 dev br-lan proto kernel scope link src 10.17.3.254

ip rule list
0:      from all lookup local
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_wg
32766:  from all lookup main
32767:  from all lookup default

ip route list table all
default via 192.168.18.254 dev wan table pbr_wan
10.17.3.0/24 dev br-lan table pbr_wan proto kernel scope link src 10.17.3.254
default via 172.20.8.11 dev wg table pbr_wg
10.17.3.0/24 dev br-lan table pbr_wg proto kernel scope link src 10.17.3.254
default via 192.168.18.254 dev wan proto static src 192.168.18.24
10.17.3.0/24 dev br-lan proto kernel scope link src 10.17.3.254
172.20.8.0/24 dev wg proto kernel scope link src 172.20.8.11
192.168.18.0/24 dev wan proto kernel scope link src 192.168.18.24
local 10.17.3.254 dev br-lan table local proto kernel scope host src 10.17.3.254
broadcast 10.17.3.255 dev br-lan table local proto kernel scope link src 10.17.3.254
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.20.8.11 dev wg table local proto kernel scope host src 172.20.8.11
broadcast 172.20.8.255 dev wg table local proto kernel scope link src 172.20.8.11
local 192.168.18.24 dev wan table local proto kernel scope host src 192.168.18.24
broadcast 192.168.18.255 dev wan table local proto kernel scope link src 192.168.18.24
service pbr status
============================================================
pbr - environment
pbr 1.1.1-7 running on OpenWrt 23.05.0. WAN (IPv4): wan/wan/192.168.18.254.
============================================================
Dnsmasq version 2.89  Copyright (c) 2000-2022 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP no-DHCPv6 no-Lua TFTP no-conntrack no-ipset no-nftset no-auth no-cryptohash no-DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward { # handle 35
        }
        chain pbr_input { # handle 36
        }
        chain pbr_output { # handle 37
        }
        chain pbr_prerouting { # handle 38
                ip saddr @pbr_wg_4_src_ip_cfg066ff5 goto pbr_mark_0x020000 comment "LAN traffic through wg" # handle 451
        }
        chain pbr_postrouting { # handle 39
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 { # handle 444
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 445
                return # handle 446
        }
        chain pbr_mark_0x020000 { # handle 447
                counter packets 13640 bytes 1155392 meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 448
                return # handle 449
        }
============================================================
pbr nft sets
        set pbr_wg_4_src_ip_cfg066ff5 { # handle 450
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "LAN traffic through wg"
                elements = { 10.17.3.0/24 counter packets 13640 bytes 1155392 }
        }
============================================================
IPv4 table 256 route: default via 192.168.18.254 dev wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 172.20.8.11 dev wg
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_wg

I tried to set route_allowed_ips to 1, but in this case the NTP traffic goes through the wireguard interface too.

I guess there is an issue with table route priority but i can't see how to solve that.

Any help is welcomed :slight_smile:

It would be best to illustrate the desired topology including the hosts involved in routing and the subnets for each network segment, as well as persistent firewall and runtime WireGuard configs.

Note that you can safely remove the NTP policy since your default gateway in the main table is WAN and PBR app doesn't support policies for outbound UPD anyway.

I agree with @vgaetera it is not really clear so we could use some more information.

However taking a wild guess I think the rule below should be added to the pbr routing tables

@psherman, we have another issue with Wireguard. I think, PBR does not work with UDP, so router sends replies via wan interface.

Routing Wireguard traffic may require setting net.ipv4.conf.wg0.rp_filter = 2 in /etc/sysctl.conf. Please refer to issue #41 for more details.

Add also mark in output chain in PBR.

It depends on the specific PBR implementation and whether the traffic is transit or not.
As long as routing rules don't require traffic marking, it can work even for outbound UDP.

1 Like

Thanks for your answers, here is a diagram of both LANs :

I removed the NTP PBR policy and it's not changing the behaviour. Te be honest i don't even know how both LANs achieve to reach each other :grin: e.g a traceroute from 1 NAS to the 2d one works fine :

traceroute 10.17.3.253
traceroute to 10.17.3.253 (10.17.3.253), 30 hops max, 60 byte packets
 1  10.107.5.254 (10.107.5.254)  0.285 ms  0.253 ms  0.237 ms
 2  172.20.8.254 (172.20.8.254)  12.384 ms  12.371 ms  12.416 ms
 3  * * *
 4  10.17.3.253 (10.17.3.253)  28.251 ms  28.245 ms  28.233 ms

But from the router when i try to reach other lan, it send the packets to wan :

root@10-17-3-254:~# traceroute 10.107.5.253
traceroute to 10.107.5.253 (10.107.5.253), 30 hops max, 46 byte packets
 1  192.168.18.254 (192.168.18.254)  0.626 ms  0.614 ms  0.509 ms
 2  *  *  *

I tried to set net.ipv4.conf.wg.rp_filter = 2 and it didn't change default routing

1 Like

Add mark to OUTPUT chain in PBR, and give full routing:
ip route show table all

I've marked the chain output in prb config for local traffic :

config policy
        option name 'LAN traffic through wg'
        option interface 'wg'
        option src_addr '10.17.3.0/24'

config policy
        option name 'Local traffic through wg'
        option interface 'wg'
        option src_addr '127.0.0.1/8'
        option dest_addr '10.107.5.0/24'
        option chain 'output'

config policy
        option name 'NTP traffic through wan'
        option src_addr '127.0.0.1/8'
        option dest_port '123'
        option proto 'udp'
        option interface 'wan'
        option chain 'output'

Here is the routing tables status (i've remove ipv6 routes for more visibility) :

default via 192.168.18.254 dev wan table pbr_wan
10.17.3.0/24 dev br-lan table pbr_wan proto kernel scope link src 10.17.3.254
default via 172.20.8.11 dev wg table pbr_wg
10.17.3.0/24 dev br-lan table pbr_wg proto kernel scope link src 10.17.3.254
default via 192.168.18.254 dev wan proto static src 192.168.18.24
10.17.3.0/24 dev br-lan proto kernel scope link src 10.17.3.254
172.20.8.0/24 dev wg proto kernel scope link src 172.20.8.11
192.168.18.0/24 dev wan proto kernel scope link src 192.168.18.24
local 10.17.3.254 dev br-lan table local proto kernel scope host src 10.17.3.254
broadcast 10.17.3.255 dev br-lan table local proto kernel scope link src 10.17.3.254
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.20.8.11 dev wg table local proto kernel scope host src 172.20.8.11
broadcast 172.20.8.255 dev wg table local proto kernel scope link src 172.20.8.11
local 192.168.18.24 dev wan table local proto kernel scope host src 192.168.18.24
broadcast 192.168.18.255 dev wan table local proto kernel scope link src 192.168.18.24

Local traffic still use wan interface prior to pbr_wg default route...

Try with deleting source address:

Edit, normally you do the routing with the allowed IPs but in your case route_allowed ips is turned off so have to do it otherwise

OK, where is route to 10.107.5.X network?

@egc I tried to remove src_addr and it dindn't change the routing to wan
@ulmwind : That's the point. I saw that if i remove the policy

config policy
        option name 'LAN traffic through wg'
        option interface 'wg'
        option src_addr '10.17.3.0/24'

And let route_allowed_ips to 0 i lost the connectivity between 2 LANs

But if i set route_allowed_ips to 1 all traffic is sent through wireguard tunnel, regardless if the precedent policy is set.

So from what i understand, when setting allowed_ips to 0.0.0.0/0 and route_allowed_ips to 0, wireguard accept all destinations (including 10.107.5.0/24 that's why it works despite route is not in the routing table) and the pbr is processed. So the issue is in the local output pbr which seems not to work...

1 Like

Try the following, just delete the two policy rules above and just make a static route, routing 10.107.5.0/24 (which I assume is the LAN address of the other side) via the WG interface.
This way there should always be a route to the other side via the WG interface

You are correct about route_allowed_ips :slight_smile:

Yes this does the trick, finally I let only this policy :

config policy
        option name 'LAN traffic through wg'
        option interface 'wg'
        option src_addr '10.17.3.0/24'

Configured wireguard like this

        list allowed_ips '0.0.0.0/0'
        option route_allowed_ips '0'

And added static route for the remote LAN :

config route '10_107_5_0_24'
        option interface 'wg'
        option target '10.107.5.0/24'
        option gateway '172.20.8.254'

Initially I hoped that i could avoid adding a static route/gateway and use only pbr, but it is working fine...

Thanks for your help

1 Like

It is 'dumb' approach. The question, why PBR does not add special table containing route to 10.107.5.X network.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.