PBR local packet routing issue

Recently i added pbr to my working site to site wireguard config. Router is standard x86 on both sides, with latest openwrt stable. Access works as expected from site A local subnet(s) to site B remote subnet(s). Firewall is set up properly. This config was working perfectly when not using pbr for the wireguard routes, but letting wireguard create its own routes. With pbr however, my local router A cannot access remote site B subnets, even though i have a pbr rule set up for "output". Interestingly, ping works if i specify the output interface/ip in the ping command.

root@HAWAII:~# ping 10.11.11.1
PING 10.11.11.1 (10.11.11.1): 56 data bytes
^C
--- 10.11.11.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

If i specify the interface, or the ip:

root@HAWAII:~# ping -I wg0 10.11.11.1
PING 10.11.11.1 (10.11.11.1): 56 data bytes
64 bytes from 10.11.11.1: seq=0 ttl=64 time=3.849 ms
64 bytes from 10.11.11.1: seq=1 ttl=64 time=4.424 ms
^C
--- 10.11.11.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.849/4.136/4.424 ms
root@HAWAII:~# ping -I 10.244.245.1 10.11.11.1
PING 10.11.11.1 (10.11.11.1) from 10.244.245.1: 56 data bytes
64 bytes from 10.11.11.1: seq=0 ttl=64 time=3.555 ms
64 bytes from 10.11.11.1: seq=1 ttl=64 time=4.980 ms
64 bytes from 10.11.11.1: seq=2 ttl=64 time=3.821 ms
^C
--- 10.11.11.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 3.555/4.118/4.980 ms

pbr config:

root@HAWAII:~# cat /etc/config/pbr

config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config policy
        option name 'wg0 output'
        option dest_addr '10.244.245.0/24 10.11.11.0/24 10.20.20.0/24 172.20.20.0/24 10.13.13.0/24 172.31.222.0/24'
        option chain 'output'
        option interface 'wg0'

config policy
        option name 'wg0 forward'
        option dest_addr '10.244.245.0/24 10.11.11.0/24 10.20.20.0/24 172.20.20.0/24 10.13.13.0/24 172.31.222.0/24'
        option interface 'wg0'

pbr status:

root@HAWAII:~# service pbr status
============================================================
pbr - environment
pbr 1.1.1-7 running on OpenWrt 23.05.3. WAN (IPv4): wan/pppoe-wan/xxx.xxx.238.115.
============================================================
Dnsmasq version 2.90  Copyright (c) 2000-2024 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth cryptohash DNSSEC no-ID loop-detect inotify dumpfile
============================================================
pbr chains - policies
        chain pbr_forward { # handle 4472
        }
        chain pbr_input { # handle 4473
        }
        chain pbr_output { # handle 4474
                ip daddr @pbr_wg0_4_dst_ip_cfg046ff5 goto pbr_mark_0x020000 comment "wg0 output" # handle 6601
        }
        chain pbr_prerouting { # handle 4475
                ip daddr @pbr_wg0_4_dst_ip_cfg056ff5 goto pbr_mark_0x020000 comment "wg0 forward" # handle 6603
        }
        chain pbr_postrouting { # handle 4476
        }
============================================================
pbr chains - marking
        chain pbr_mark_0x010000 { # handle 6591
                counter packets 0 bytes 0 meta mark set meta mark & 0xff01ffff | 0x00010000 # handle 6592
                return # handle 6593
        }
        chain pbr_mark_0x020000 { # handle 6594
                counter packets 421 bytes 38180 meta mark set meta mark & 0xff02ffff | 0x00020000 # handle 6595
                return # handle 6596
        }
        chain pbr_mark_0x030000 { # handle 6597
                counter packets 0 bytes 0 meta mark set meta mark & 0xff03ffff | 0x00030000 # handle 6598
                return # handle 6599
        }
============================================================
pbr nft sets
        set pbr_wg0_4_dst_ip_cfg046ff5 { # handle 6600
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "wg0 output"
                elements = { 10.11.11.0/24 counter packets 8 bytes 672, 10.13.13.0/24 counter packets 0 bytes 0,
                             10.20.20.0/24 counter packets 0 bytes 0, 10.244.245.0/24 counter packets 0 bytes 0,
                             172.20.20.0/24 counter packets 0 bytes 0, 172.31.222.0/24 counter packets 0 bytes 0 }
        }
        set pbr_wg0_4_dst_ip_cfg056ff5 { # handle 6602
                type ipv4_addr
                flags interval
                counter
                auto-merge
                comment "wg0 forward"
                elements = { 10.11.11.0/24 counter packets 408 bytes 37088, 10.13.13.0/24 counter packets 0 bytes 0,
                             10.20.20.0/24 counter packets 0 bytes 0, 10.244.245.0/24 counter packets 5 bytes 420,
                             172.20.20.0/24 counter packets 0 bytes 0, 172.31.222.0/24 counter packets 0 bytes 0 }
        }
============================================================
IPv4 table 256 route: default via xxx.xxx.238.115 dev pppoe-wan
IPv4 table 256 rule(s):
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
IPv4 table 257 route: default via 10.244.245.1 dev wg0
IPv4 table 257 rule(s):
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_wg0
IPv4 table 258 route: default via 172.20.21.1 dev eth0
IPv4 table 258 rule(s):
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_wan_modem

edit: After some more digging, i found the source of the issue:
When the local packet is generated, the source ip of that packet will be my WAN ip address, since i do not have a specific route for the destination subnet, so the default route will be used as source. After that, pbr properly routes that packet to the wireguard interface, but the source address is still my WAN ip, hence the other side will drop the packet. A quick dirty snat rule makes this work, but my wan ip is dynamic. Are there any way to insert some snat rule into pbr?

custom user script/file?

I'm ashamed, but i'm totally clueless about nftables and fw4, while understanding the basic and old concepts of netfilter.
How and where should i add a simple rule of: attached to output nat table (not postrouting), out interface "wg0" snat to ip x.x.x.x ?

sorry, I can help you with "how to insert a rule into pbr", but not "what rule do I need to insert".

You can use Luci to do SNAT (aka MASQUERADE):

https://openwrt.org/docs/guide-user/firewall/fw3_configurations/fw3_nat#selective_nat

But it is possible your setup is to blame, usually snat should not be necessary.

In a regular site-to-site setup you only have routes to each other sites and have internet via the own router, snat/masquerade should not be necessary as yoiou know the subnet of the other side and can make a return route.

Do you want some LAN clients to have internet access via the other side?

Do you only have these two routers connected or are there other peers involved?

Please exactly describe your wishes and please connect to your OpenWRT device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/firewall
ip route show
ip route show table all
ip rule show
wg show

Thank you for your reply.

I do understand i can use luci to do snat (masquerade), but as i said before, fw4 in openwrt explicitly adds snat rules to the postrouting hook, as can be seen in the fw4 source code. I explicitly want to add an snat rule to the output hook, since i do not want to snat forwarded packets, i want to snat only those packets that come locally from the openwrt device. Looking at the netfilter infographic you can hopefully understand the difference.

PBR is to blame (more specifically attaching a pbr rule to the output chain), but i need it for some specific purposes. I had a working setup for years before adding pbr.

That is true, and that exactly how it works, and used to work for me with wireguard with the option route_allowed_ips="1" . However, when PBR is used, these routes are not added to the main routing table, but PBR modifies them on the fly. I do not do nat/masquerade between my own subnets/vlans.

My goal is to have one device (mac address) have all his traffic routed through a wireguard tunnel. While other clients get split routing, default route to the internet, and wireguard tunnel to the other remote internal subnets.

I have a lot of "peers", multiple sites with remote subnets,. some single remote device with a single ip, etc...

I'll try to condense it as much as i can. This is my kernel routing table, when not using route allowed ips for wireguard:

root@HAWAII:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         lo1.esr0-gyor.n 0.0.0.0         UG    0      0        0 pppoe-wan
10.10.10.0      *               255.255.255.0   U     0      0        0 br-lan.5
10.12.12.0      *               255.255.255.0   U     0      0        0 br-lan.1
10.12.99.0      *               255.255.255.0   U     0      0        0 br-lan.99
10.244.245.0    *               255.255.255.0   U     0      0        0 wg0
x.x.185.46     lo1.esr0-gyor.n 255.255.255.255 UGH   0      0        0 pppoe-wan
x.x.238.115 *               255.255.255.255 UH    0      0        0 pppoe-wan
172.20.21.0     *               255.255.255.0   U     0      0        0 eth0

This is my /etc/config/pbr

root@HAWAII:~# cat /etc/config/pbr

config pbr 'config'
        option enabled '1'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config policy
        option name 'wg0 output'
        option dest_addr '10.244.245.0/24 10.11.11.0/24 10.20.20.0/24 172.20.20.0/24 10.13.13.0/24 172.31.222.0/24'
        option chain 'output'
        option interface 'wg0'

config policy
        option name 'wg0 forward'
        option dest_addr '10.244.245.0/24 10.11.11.0/24 10.20.20.0/24 172.20.20.0/24 10.13.13.0/24 172.31.222.0/24'
        option interface 'wg0'

Notice, the 'wg0 output' policy. All those subnets are reachable through my wireguard tunnel. Almost same as the 'wg0 forward' policy.
Now, this is what happens, when i ping a remote subnet ip (lets say 10.11.11.1) from the openwrt router: ping command does a lookup of my kernel routing table, does not find a kernel route to 10.11.11.1, so it expects that the packet will go through the default route, hence it will generate the icmp packet with the source ip address of my wan interface. As it goes through netfilter, it will reach the pbr rules that will modify the routing table for that packet to this:

root@HAWAII:~# ip route show table 257
default via 10.244.245.1 dev wg0
10.10.10.0/24 dev br-lan.5 proto kernel scope link src 10.10.10.1
10.12.12.0/24 dev br-lan.1 proto kernel scope link src 10.12.12.1
10.12.99.0/24 dev br-lan.99 proto kernel scope link src 10.12.99.1

Then this packet goes out through my wireguard interface with the source ip of my wan. And obviously, it will get dropped by the receiving side. The way to make this work is to rewrtite the source ip of this packet from the routers wan ip to routers openwrt interface ip.
I specifically do not mention the forwarding pbr rule, as that works as expected, i only have problems with the packets that come directly from the router.

I found something similar here.
According to this, i should be able to make a custom rule in /etc/nftables.d/10-custom-filter-chains.nft.
Something like this, but this does not work (and i have no clue why):

chain user_pre_output_nat {
    type nat hook output priority -1; policy accept;
    oifname "wg0" snat ip to 10.244.245.1
}
root@HAWAII:~# ubus call system board
{
        "kernel": "5.15.150",
        "hostname": "HAWAII",
        "system": "Intel(R) N100",
        "model": "Default string Default string",
        "board_name": "default-string-default-string",
        "rootfs_type": "ext4",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.3",
                "revision": "r23809-234f1a2efa",
                "target": "x86/64",
                "description": "OpenWrt 23.05.3 r23809-234f1a2efa"
        }
}
root@HAWAII:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix '*::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1'
        list ports 'eth2'
        list ports 'eth3'

config interface 'lan'
        option device 'br-lan.1'
        option proto 'static'
        option ipaddr '10.12.12.1'
        option netmask '255.255.255.0'
        option ip6assign '64'
        option ip6hint '0'
        list ip6class 'wan_6'

config interface 'wan'
        option device 'eth0'
        option proto 'pppoe'
        option username '*'
        option password '* '
        option ipv6 'auto'
        option mtu '1492'
        option peerdns '0'

config bridge-vlan
        option device 'br-lan'
        option vlan '1'
        list ports 'eth1:u*'
        list ports 'eth2:u*'
        list ports 'eth3:u*'

config bridge-vlan
        option device 'br-lan'
        option vlan '5'
        list ports 'eth1:t'

config interface 'guest'
        option proto 'static'
        option device 'br-lan.5'
        option ipaddr '10.10.10.1'
        option netmask '255.255.255.0'
        option ip6assign '64'
        option ip6hint '1'
        list ip6class 'wan_6'

config interface 'wg0'
        option proto 'wireguard'
        option private_key '*'
        option listen_port '51820'
        list addresses '10.244.245.1/24'
        option defaultroute '0'
        option delegate '0'
        option mtu '1400'

config device
        option name 'eth0'
        option ipv6 '0'

config device
        option name 'wg0'
        option ipv6 '0'

config wireguard_wg0
        option description '*'
        option public_key '*'
        list allowed_ips '10.244.245.4'

config wireguard_wg0
        option description '*'
        option public_key '*'
        list allowed_ips '10.244.245.5'

config wireguard_wg0
        option description '*'
        option public_key '*'
        list allowed_ips '10.244.245.6'
        list allowed_ips '172.31.222.0/24'

config wireguard_wg0
        option description '*'
        option public_key '*'
        list allowed_ips '*'

config wireguard_wg0
        option description '*'
        option public_key '*'
        option persistent_keepalive '25'
        option endpoint_host '*'
        option endpoint_port '51820'
        list allowed_ips '10.11.11.0/24'
        list allowed_ips '10.20.20.0/24'
        list allowed_ips '172.20.20.0/24'
        list allowed_ips '10.244.245.2'

config wireguard_wg0
        option description '*'
        option public_key '*'
        list allowed_ips '10.244.245.7'
        list allowed_ips '10.13.13.0/24'
        option persistent_keepalive '0'

config interface 'wan_modem'
        option proto 'dhcp'
        option device 'eth0'
        option defaultroute '0'
        option peerdns '0'

config interface 'iot'
        option proto 'static'
        option device 'br-lan.99'
        option ipaddr '10.12.99.1'
        option netmask '255.255.255.0'
        option ip6assign '64'
        option ip6hint '9'
        list ip6class 'wan_6'

config bridge-vlan
        option device 'br-lan'
        option vlan '99'
        list ports 'eth1:t'
root@HAWAII:~# cat /etc/config/firewall

config defaults
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'DROP'
        option synflood_protect '1'
        option drop_invalid '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'DROP'
        list network 'lan'

config zone
        option name 'wan'
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'DROP'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan_modem'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option family 'ipv4'
        option target 'ACCEPT'
        list icmp_type 'echo-request'
        option limit '1000/second'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Block-IPv6-mac-list'
        option src '*'
        option target 'DROP'
        option family 'ipv6'
        list src_mac '*'
        list src_mac '*'
        list proto 'all'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'guest'
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'DROP'
        list network 'guest'

config forwarding
        option src 'guest'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'guest'

config zone
        option name 'wireguard'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'DROP'
        option mtu_fix '1'
        option family 'ipv4'
        list network 'wg0'

config forwarding
        option src 'wireguard'
        option dest 'lan'

config forwarding
        option src 'wireguard'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'wireguard'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'DNS4 redirect lan'
        option src 'lan'
        option src_dport '53'
        option dest_ip '10.12.12.1'
        option family 'ipv4'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'DNS4 redirect guest'
        option src 'guest'
        option src_dport '53'
        option dest_ip '10.10.10.1'
        option family 'ipv4'

config redirect
        option dest 'iot'
        option target 'DNAT'
        option name 'DNS4 redirect iot'
        option family 'ipv4'
        option src 'iot'
        option src_dport '53'
        option dest_ip '10.12.99.1'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'DNS6 redirect lan'
        option src 'lan'
        option src_dport '53'
        option dest_ip '*'
        option family 'ipv6'

config redirect
        option dest 'guest'
        option target 'DNAT'
        option name 'DNS6 redirect guest'
        option src 'guest'
        option src_dport '53'
        option dest_ip '*'
        option family 'ipv6'

config redirect
        option dest 'iot'
        option target 'DNAT'
        option name 'DNS6 redirect iot'
        option family 'ipv6'
        option src 'iot'
        option src_dport '53'
        option dest_ip '*'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name '*'
        option family 'ipv4'
        list proto 'tcp'
        option src 'wan'
        option src_dport '1111'
        option dest_ip '10.12.12.16'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name '**'
        option family 'ipv4'
        option src 'wan'
        option src_dport '13318'
        option dest_ip '10.12.12.13'
        option enabled '0'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name '***'
        option family 'ipv4'
        option src 'wan'
        option src_dport '2222'
        option dest_ip '10.12.12.13'
        option enabled '0'

config rule
        option name 'V6 *** '
        option family 'ipv6'
        option src 'wan'
        option dest 'lan'
        list dest_ip '****'
        option dest_port '51414'
        option target 'ACCEPT'

config rule
        option name 'V6 ****'
        option family 'ipv6'
        option src 'wan'
        option dest 'lan'
        list dest_ip '****'
        option dest_port '8***'
        option target 'ACCEPT'

config rule
        option name 'V6 *****'
        option family 'ipv6'
        option src 'wan'
        option dest 'lan'
        list dest_ip '::*******/::ffff:ffff:ffff:ffff'
        option dest_port '****'
        option target 'ACCEPT'

config rule
        option name 'Allow wireguard'
        option src 'wan'
        option dest_port '51820'
        option target 'ACCEPT'
        list proto 'udp'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'DNAT4 *****'
        option family 'ipv4'
        option src 'wan'
        option src_dport '****'
        option dest_ip '10.12.12.13'
        option enabled '0'

config rule
        option name 'Allow guest dhcp4'
        list proto 'udp'
        option src 'guest'
        option dest_port '67'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow iot dhcp4'
        option family 'ipv4'
        list proto 'udp'
        option src 'iot'
        option dest_port '67'
        option target 'ACCEPT'

config rule
        option name 'Allow guest dhcp6'
        list proto 'udp'
        option src 'guest'
        option dest_port '547'
        option target 'ACCEPT'

config rule
        option name 'Allow iot dhcp6'
        list proto 'udp'
        option src 'iot'
        option dest_port '547'
        option target 'ACCEPT'

config rule
        option name 'Allow guest dns'
        option src 'guest'
        option dest_port '53'
        option target 'ACCEPT'

config rule
        option name 'Allow iot dns'
        option src 'iot'
        option dest_port '53'
        option target 'ACCEPT'

config rule
        option name 'Allow guest ICMP'
        list proto 'icmp'
        option src 'guest'
        option target 'ACCEPT'

config include 'miniupnpd'
        option type 'script'
        option path '/usr/share/miniupnpd/firewall.include'

config zone
        option name 'iot'
        option input 'DROP'
        option output 'ACCEPT'
        option forward 'DROP'
        list network 'iot'

config forwarding
        option src 'iot'
        option dest 'wan'

config forwarding
        option src 'lan'
        option dest 'iot'

config forwarding
        option src 'wireguard'
        option dest 'iot'

config rule
        option name 'Allow iot ICMP'
        list proto 'icmp'
        option src 'iot'
        option target 'ACCEPT'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/pbr.firewall.include'
root@HAWAII:~# ip route show
default via ****** dev pppoe-wan proto static
10.10.10.0/24 dev br-lan.5 proto kernel scope link src 10.10.10.1
10.12.12.0/24 dev br-lan.1 proto kernel scope link src 10.12.12.1
10.12.99.0/24 dev br-lan.99 proto kernel scope link src 10.12.99.1
10.244.245.0/24 dev wg0 proto kernel scope link src 10.244.245.1
******* via ****** dev pppoe-wan proto static
******* dev pppoe-wan proto kernel scope link src ******
172.20.21.0/24 dev eth0 proto kernel scope link src 172.20.21.224
root@HAWAII:~# ip route show table all
default via ***** dev pppoe-wan table pbr_wan
10.10.10.0/24 dev br-lan.5 table pbr_wan proto kernel scope link src 10.10.10.1
10.12.12.0/24 dev br-lan.1 table pbr_wan proto kernel scope link src 10.12.12.1
10.12.99.0/24 dev br-lan.99 table pbr_wan proto kernel scope link src 10.12.99.1
default via 10.244.245.1 dev wg0 table pbr_wg0
10.10.10.0/24 dev br-lan.5 table pbr_wg0 proto kernel scope link src 10.10.10.1
10.12.12.0/24 dev br-lan.1 table pbr_wg0 proto kernel scope link src 10.12.12.1
10.12.99.0/24 dev br-lan.99 table pbr_wg0 proto kernel scope link src 10.12.99.1
default via 172.20.21.1 dev eth0 table pbr_wan_modem
10.10.10.0/24 dev br-lan.5 table pbr_wan_modem proto kernel scope link src 10.10.10.1
10.12.12.0/24 dev br-lan.1 table pbr_wan_modem proto kernel scope link src 10.12.12.1
10.12.99.0/24 dev br-lan.99 table pbr_wan_modem proto kernel scope link src 10.12.99.1
default via ***** dev pppoe-wan proto static
10.10.10.0/24 dev br-lan.5 proto kernel scope link src 10.10.10.1
10.12.12.0/24 dev br-lan.1 proto kernel scope link src 10.12.12.1
10.12.99.0/24 dev br-lan.99 proto kernel scope link src 10.12.99.1
10.244.245.0/24 dev wg0 proto kernel scope link src 10.244.245.1
****** via ******* dev pppoe-wan proto static
****** dev pppoe-wan proto kernel scope link src *****
172.20.21.0/24 dev eth0 proto kernel scope link src 172.20.21.224
local 10.10.10.1 dev br-lan.5 table local proto kernel scope host src 10.10.10.1
broadcast 10.10.10.255 dev br-lan.5 table local proto kernel scope link src 10.10.10.1
local 10.12.12.1 dev br-lan.1 table local proto kernel scope host src 10.12.12.1
broadcast 10.12.12.255 dev br-lan.1 table local proto kernel scope link src 10.12.12.1
local 10.12.99.1 dev br-lan.99 table local proto kernel scope host src 10.12.99.1
broadcast 10.12.99.255 dev br-lan.99 table local proto kernel scope link src 10.12.99.1
local 10.244.245.1 dev wg0 table local proto kernel scope host src 10.244.245.1
broadcast 10.244.245.255 dev wg0 table local proto kernel scope link src 10.244.245.1
local 84.0.228.18 dev pppoe-wan table local proto kernel scope host src 84.0.228.18
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
local 172.20.21.224 dev eth0 table local proto kernel scope host src 172.20.21.224
broadcast 172.20.21.255 dev eth0 table local proto kernel scope link src 172.20.21.224
default from 2001:4c4e:*****::/64 via fe80::***** dev pppoe-wan proto static metric 512 pref medium
default from 2001:4c4e:*****::/56 via fe80::***** dev pppoe-wan proto static metric 512 pref medium
unreachable 2001:4c4e:*****::/64 dev lo proto static metric 2147483647 pref medium
2001:4c4e:***** via fe80::***** dev pppoe-wan proto static metric 512 pref medium
2001:4c4e:***::/64 dev br-lan.1 proto static metric 1024 pref medium
2001:4c4e:****::/64 dev br-lan.5 proto static metric 1024 pref medium
2001:4c4e:****::/64 dev br-lan.99 proto static metric 1024 pref medium
unreachable 2001:4c4e:*****::/56 dev lo proto static metric 2147483647 pref medium
unreachable *****::/48 dev lo proto static metric 2147483647 pref medium
fe80::****c dev pppoe-wan proto kernel metric 256 pref medium
fe80::**** dev pppoe-wan proto kernel metric 256 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev br-lan.1 proto kernel metric 256 pref medium
fe80::/64 dev br-lan.5 proto kernel metric 256 pref medium
fe80::/64 dev br-lan.99 proto kernel metric 256 pref medium
fe80::/64 dev ifb4pppoe-wan proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast 2001:4c4e:*****:: dev pppoe-wan table local proto kernel metric 0 pref medium
local 2001:4c4e:***** dev pppoe-wan table local proto kernel metric 0 pref medium
anycast 2001:4c4e:****:: dev br-lan.1 table local proto kernel metric 0 pref medium
local 2001:4c4e***::1 dev br-lan.1 table local proto kernel metric 0 pref medium
etc etc........ ipv6 stuff
root@HAWAII:~# ip rule show
0:      from all lookup local
30000:  from all fwmark 0x10000/0xff0000 lookup pbr_wan
30001:  from all fwmark 0x20000/0xff0000 lookup pbr_wg0
30002:  from all fwmark 0x30000/0xff0000 lookup pbr_wan_modem
32766:  from all lookup main
32767:  from all lookup default
root@HAWAII:~# wg show
interface: wg0
  public key: *********
  private key: (hidden)
  listening port: 51820

peer: ***************
  endpoint: *********:47416
  allowed ips: 10.244.245.7/32, 10.13.13.0/24
  latest handshake: 1 minute, 13 seconds ago
  transfer: 593.94 KiB received, 1.39 MiB sent

peer: ************
  endpoint: [2001:4c4e:**********]:51820
  allowed ips: 10.244.244.0/24, 10.11.11.0/24, 10.20.20.0/24, 172.20.20.0/24, 10.244.245.2/32
  latest handshake: 1 minute, 14 seconds ago
  transfer: 143.72 MiB received, 23.74 MiB sent
  persistent keepalive: every 25 seconds

peer: ***********
  allowed ips: 10.244.245.5/32

peer: ***********
  allowed ips: 10.244.245.3/32

peer: *************
  allowed ips: 10.244.245.4/32

peer:**************
  allowed ips: 10.244.245.6/32, 172.31.222.0/24

Thanks!

This needs some further study but before looking into that, why are you not enabling route allowed IPs on all peers (option route_allowed_ips '1')?
That should get you all the routes in the routing table.

Yes. I want some clients to route all traffic through the wg0 tunnel, and some clients to do split tunnelling, the reason i'm doing it with pbr. Hence if i want any client to be able to route all traffic thorugh that, i must use 0.0.0.0/0 in the allowed ip-s setting of the wg interface. But if i also set option route_allowed_ips '1' it will add a default route to my routing table, and i don't want that.

Edit: the pasted config does not reflect that, since i did a rollback yesterday.

The solution for that is to disable default routing: on the WG interface > Advanced Setting : untick: "Use default gateway"
or set option defaultroute '0'

Alternatively set a metric of 10 on the WG interface

Apparently, that does work.

option route_allowed_ips '1'
        list allowed_ips '0.0.0.0/0'
        list allowed_ips '10.244.245.2'
        list allowed_ips '10.11.11.0/24'
        list allowed_ips '10.20.20.0/24'
        list allowed_ips '172.20.20.0/24'

makes it have all the entries in the local routing table.
But it won't make a route for 0.0.0.0 as option defaultroute '0' is set.
It nicely circumvents the issue at hand, and the pbr rules for specific mac addresses work perfectly.

Thank you!

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.