Pbr weird behaviour LAN vs mobile network

Hi everyone,

TL;DR: If I route everything via VPN interface for a specific client, it works while I'm connected to wifi/ethernet. But it fails when I'm on mobile network.

I'm currently running OpenWRT on MT6000 (snapshot) and I have network a setup like this:

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd3f:f6bf:3a19::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        list ports 'lan4'
        list ports 'lan5'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth1'
        option proto 'dhcpv6'
        option reqaddress 'try'
        option reqprefix 'auto'
        option ip6assign '57'

config interface 'trint'
        option proto 'wireguard'
        option private_key 'redacted'
        list addresses '10.16.13.5'

config wireguard_trint
        option description 'trint'
        option public_key 'redacted'
        list allowed_ips '0.0.0.0/0'
        option persistent_keepalive '25'
        option endpoint_host '<redacted>'

So I have the interface trint, which is connected to endpoint_host all the time. And it works as expected (does not matter if I'm on LAN of connected via VPN to home)

And here's my current working pbr setup:

config pbr 'config'
        option verbosity '2'
        option strict_enforcement '1'
        option resolver_set 'dnsmasq.nftset'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        option webui_show_ignore_target '0'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        option enabled '1'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config policy
        option name 'github'
        option src_addr '192.168.1.0/24'
        option dest_addr 'github.com'
        option interface 'trint'

config policy
        option name 'bitbucket'
        option src_addr '192.168.1.0/24'
        option dest_addr 'bitbucket.org'
        option interface 'trint'

config policy
        option name 'steam'
        option src_addr '192.168.1.0/24'
        option dest_addr 'store.steampowered.com'
        option interface 'trint'

Recently, I've setup a new wireguard server on my raspberry pi, which runs on docker and uses macvlan as networking setup, so I assigned the IP 192.168.1.51 to container and it works as expected. I've defined port forwarding as well. So I can connect to VPN via my mobile devices. Here's my firewall config:

config defaults
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list network 'wan6'
        list network 'trint'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config redirect
        option target 'DNAT'
        option name 'Wireguard'
        option src 'wan'
        option src_dport '51820'
        option dest_ip '192.168.1.118'
        option dest 'lan'
        option family 'ipv4'
        option dest_port '51820'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'wireguardvlan'
        option src 'wan'
        option src_dport '51821'
        option dest_ip '192.168.1.52'
        option dest_port '51820'

config include 'pbr'
        option fw4_compatible '1'
        option type 'script'
        option path '/usr/share/pbr/firewall.include'

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'raspivpn'
        option src 'wan'
        option src_dport '51822'
        option dest_ip '192.168.1.51'
        option dest_port '51820'

Here's my problem begins. If I set the following rule on PBR:

config policy
        option name 'rulethemall'
        option src_addr '192.168.1.51'
        option interface 'trint'

It works as expected and I can see my traffic is routed through VPN. But strangely this is only working from LAN. If I connect to my vpn from mobile network, I cannot access anywhere.

Does anyone have an idea what I could be missing? Here's my client vpn configs:

Server config:

[Interface]
Address = 10.151.13.1
ListenPort = 51820
PrivateKey = redacted
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth+ -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth+ -j MASQUERADE

[Peer]

PublicKey = redacted
PresharedKey = redacted
AllowedIPs = 10.151.13.2/32

Client config

[Interface]
Address = 10.151.13.2
PrivateKey = redacted
ListenPort = 51820
DNS = 192.168.1.3

[Peer]
PublicKey = redacted
PresharedKey = redacted
Endpoint = redacted:51822
AllowedIPs = 0.0.0.0/0

I feel it's failing due to AllowedIPs (and routing at all) but like I mentioned, everything works great when I'm on LAN. It just fails when I switch to mobile network.

I think this might be your problem:
Your WireGuard server gets its traffic via your WAN (via the port forwards) so you cannot send all traffic from this server out via your WG client because traffic to your mobile client has to go out again via the WAN.

The PBR package has an include file (/usr/share/pbr/pbr.user.wg_server_and_client) which works by routing the WG servers source port back via the WAN to solve this, however your WG server is on your LAN so it does not apply to your situation but have a look at that file it might give you an idea how to solve this

Thanks for the guidance, I did a look and unfortunately, the setup I'm trying to do is getting overcomplicated :smiley:

When I changed the allowed IPs from 0.0.0.0/0 to split tunnel, I could at least keep connectivity alive and managed to use my DNS server while connected.

But again if I set a rule to forward all traffic via pbr, it acts the same. So the problem might not related to pbr at all.

I think I need to set some routes somewhere but I don't know how :D. I'll keep searching for solutions.

I think if you route all traffic via the tunnel the traffic from your mobile which connects from outside to your VPN server will not be able to be routed back via the WAN as this traffic is also routed via the tunnel.

If my theory is valid then the following script might work, see:

1 Like

OMG it works! Thank you :tada:

Great to hear it is working, glad my theory and solution were valid :slight_smile:

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.