Policy-Based-Routing (pbr) package discussion

d3adc0d3 · October 30, 2022, 3:23am

Makes sense, thanks for clearing that up. I had mistakenly assumed that these 2 interfaces were some unidirectional implementation of setting up wireguard with openwrt which confused me as it's all peers on wireguard anyway and that doesn't really make sense for a VPN.

Can you also explain the openwrt firewall bit about the wireguard client setup more. Or point me to a detailed resource on openwrt firewall zones/forwarding with wireguard.

Am trying to figure out how to selectively route traffic incoming on a specific port from VPN to a specific host on LAN, and redirect any response out of said LAN host through vpn to return to the client. Without the policy/VPN affecting any other network traffic from said host on other ports. I'll admit my knowledge in this space is a little bit lacking but I believe this "should" be possible.

The only concern I have is if the service I am trying to host on said LAN host receives a connection request, how does it appear to the LAN host's application: from the VPN IP, or from the actual client IP? If it is from the VPN IP then it seems as though it would be impossible for the VPN to forward to the right client once the VPN recieves it. Alternatively if it is from the client IP, then the application on the host will be able to correctly address the response but does that mean PBR will intercept this outgoing response packet and route it through the VPN? which would be invisible from the LAN host's perspective.

Please feel free to leave some of the above unanswered as I understand it isn't entirely relevant to the thread but I'm just trying to understand if it is possible and how it would work. Thanks

d3adc0d3 · October 30, 2022, 3:56am

To try and represent the mess of text above more concisely I'll just create an ASCII representation of what I'm desiring.

Request:
ClientIP:123 -> VPN IP:123
OpenWRT (sees said traffic is on port 123 of VPN interface and forwards it to LAN host?) -> LAN IP:123
Response:
LAN IP:123 -> ClientIP:123
OpenWRT (filters outgoing packet based on port to be sent over VPN interface vs WAN)

Once I get this working I'd be happy to provide screenshots from luci to aid in others setting up wireguard with pbr.

ddfddf23 · October 30, 2022, 2:25pm

Hello guys, after installing and enabling getting service error "failed to set up any gateway!" in luci.
PBR Config:

config pbr 'config'
        option verbosity '2'
        option strict_enforcement '1'
        option ipv6_enabled '0'
        list ignored_interface 'vpnserver'
        list ignored_interface 'wgserver'
        option boot_timeout '30'
        option rule_create_option 'add'
        option procd_reload_delay '1'
        list webui_supported_protocol 'all'
        list webui_supported_protocol 'tcp'
        list webui_supported_protocol 'udp'
        list webui_supported_protocol 'tcp udp'
        list webui_supported_protocol 'icmp'
        option resolver_set 'dnsmasq.ipset'
        option enabled '1'

config include
        option path '/usr/share/pbr/pbr.user.aws'
        option enabled '0'

config include
        option path '/usr/share/pbr/pbr.user.netflix'
        option enabled '0'

config policy
        option name 'Plex/Emby Local Server'
        option interface 'wan'
        option src_port '8096 8920 32400'
        option enabled '0'

config policy
        option name 'Plex/Emby Remote Servers'
        option interface 'wan'
        option dest_addr 'plex.tv my.plexapp.com emby.media app.emby.media tv.emby.media'
        option enabled '0'

config policy
        option name 'WireGuard Server'
        option interface 'wan'
        option src_port '51820'
        option chain 'output'
        option proto 'udp'
        option enabled '0'

And luci screenshot:

ray308 · October 30, 2022, 2:38pm

Check if you have the following packages installed;

ipset resolveip ip-full kmod-ipt-ipset iptables

To look if you have them installed, do:

opkg list-installed | grep 'ipset\|ip-full\|dnsmasq\|resolveip'`

Should show you something like;

root@Router:~# opkg list-installed | grep 'ipset\|ip-full\|dnsmasq\|resolveip'
dnsmasq-full - 2.86-14
ip-full - 5.15.0-3
ipset - 7.15-2
kmod-ipt-ipset - 5.10.146-1
libipset13 - 7.15-2
resolveip - 2
root@Router:~#

If not, do

opkg update; opkg install ipset resolveip ip-full kmod-ipt-ipset iptables

ddfddf23 · October 30, 2022, 3:08pm

Fixed it by renaming my wan interface to cable (can be called by another name). Btw thank you for advice!

spm1001 · October 31, 2022, 2:03pm

@stangri I’m trying to use PBR to selectively route a named device to one of two WANs but I'm finding strict enforcement is not honoured when the device has an ipv6 address and there is a v6 on only one WAN.

Setup
OpenWRT 22.03.2 x86, luci-app-pbr 0.9.9-9
Two WANs - wan/eth0/pppoe-wan and van/eth1. wan is both ipv4 and ipv6, while van has network.van.ipv6='0'.

Partial output of ip address

3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 80:61:5f:0e:68:33 brd ff:ff:ff:ff:ff:ff
    inet 86.16.59.XXX/22 brd 86.16.59.QQQ scope global eth1
       valid_lft forever preferred_lft forever
11: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN group default qlen 3
    link/ppp 
    inet 81.187.168.XXX peer 81.187.81.YYY/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever
    inet6 2001:8b0:1111:1111:0:ffff:51bb:a8f9/128 scope global dynamic noprefixroute 
       valid_lft 5495sec preferred_lft 1895sec

Which shows no ipv6 address on van/eth1

However luci-app-pbr status shows an ipv6 gateway on van - which is a repeat of those on wan.

wan/pppoe-wan/81.187.81.YYY/2001:8b0:1111:1111:0:ffff:51bb:yyyy/128 fe80::998f:ead0:7d5e:32fc ✓
van/eth1/86.16.56.ZZZ/2001:8b0:1111:1111:0:ffff:51bb:yyyy/128 fe80::998f:ead0:7d5e:32fc

Issue
When I create a named device rule, forcing a device to use van, it works correctly when I set the device to be link-local ipv6, but when it has a globally routable address, then traffic goes over wan even when strict enforcement is enabled.

Any thoughts on what I should try next?

gamma-577 · October 31, 2022, 3:44pm

Hello @stangri !

I have a BTHH5A running OpenWRT 22.03.2. firewall4 is default for this firmware version. However pbr tries to install firewall4 and there is a conflict. I tried --nodeps option and pbr was installed successfully. ( Before that I had to remove vpn-policy-routing )

During the post-install pbr showed :

:~# opkg install pbr --nodeps
Installing pbr (0.9.9-9) to root...
Downloading https://repo.openwrt.melmac.net/pbr_0.9.9-9_all.ipk
Configuring pbr.
//usr/lib/opkg/info/pbr.postinst: /usr/lib/opkg/info/pbr.postinst-pkg: line 5: fw4: not found
Installing rc.d symlink for pbr... OK
Migrating vpn-policy-routing config file.
uci: Entry not found
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Activating traffic killswitch [✓]
Error: No such file or directory
list table inet fw4
                ^^^
Error: No such file or directory
list table inet fw4
                ^^^
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them
Setting up routing for 'wan/dsl0.101/266.755.99.100' [✓]
Setting up routing for 'tun0if/tun0/10.73.8.96' [✓]
Setting up routing for 'WG0IF/10.23.0.76' [✓]
Routing 'SkyQ' via wan [✓]
Routing 'wifi' via tun0if [✓]
Routing 'eth-lan-2-wg' via WG0IF [✓]
Deactivating traffic killswitch [✓]
pbr 0.9.9-9 monitoring interfaces: wan tun0if WG0IF
pbr 0.9.9-9 (iptables) started with gateways:
wan/dsl0.101/266.755.99.100 [✓]
tun0if/tun0/10.73.8.96
WG0IF/10.23.0.76

My firewall package is firewall - 2022-02-17-4cd7d4f3-3

Having said that, the pbr is routing all 3 traffics ( OpenVPN, WireGuard and WAN ) exactly as expected.

Thank you for a robust app. Do I need to update it to the next version when available?

-Gamma

stangri · October 31, 2022, 4:50pm

Have you upgraded from OpenWrt 21.02 (or an earlier version) while keeping settings?

That's older than 22.03 and with the clean 22.03 installation I don't have a firewall package, I have firewall4:

SG-135 in ~ # opkg list-installed | grep firewall
firewall4 - 2022-10-18-7ae5e14b-1
luci-app-firewall - git-22.089.67563-7e3c1b4

The error message will be fixed in 0.9.9-10. However, if you're on 22.03, I don't understand how you would not have the fw4 table.

If you're not using dnsmasq.ipset, you may want to set the resolver_set option to none to force service to nft mode. However, given the issues you've reported, I don't really understand what and how is running on your system.

d3adc0d3 · October 31, 2022, 4:54pm

@stangri

Did you see my question earlier regarding a potential use case of PBR. Was curious to know if it was possible to configure PBR in such a manor.

stangri · October 31, 2022, 5:04pm

I inferred you mean wireguard when you said VPN from your previous posts, however next time please be more specific. With the wireguard (based on UDP) it's not possible. If you set up an OpenVPN server and switch its protocol to tcp, it might be possible with the combination of pbr and firewall configs. I would not be able to help you with those, but there are other users on the forum who might.

d3adc0d3 · October 31, 2022, 5:37pm

Thanks for the info. Guess I'll have to post elsewhere to find the folks who have experience with similar setups.

gamma-577 · October 31, 2022, 6:43pm

Yes. I did an inplace upgrade using attended sysupgrade. Then reset to factory default and then imported the config. I don't know when/where firewall4 is replaced with with old firewall package. As such 22.03.2 is shipped with firewall4.

Actually, my settings are carried from 19.07.#. Then the Network --> Interfaces --> Devices tab did not exist. I too do not have any idea of how the things work.

I think I should start with a blank slate. If something goes wrong, I have a working current config backed up anyway.

Thank you for guiding me

-Gamma

stangri · November 1, 2022, 1:00am

As far as I understand IPv6 and current implementation in 22.03, this is intentional, if you use the global address, there's no masquerading, so pbr policies cannot be applied.

pesa1234 · November 1, 2022, 10:09am

Dear, version: 0.9.9-9 using iptables.
Snapshot r21149-b99d377886 with kernel 5.15.76 dnsmasq 2.86 on WRT1900ACS works very well. THANK YOU

spm1001 · November 2, 2022, 11:35am

Thanks - my understanding of ipv6 is very hazy but keen to help get this working if I can!

I tried

uci set network.wan6.sourcefilter="0" # provide default route for all prefixes
uci set firewall.@zone[1].masq6='1' # enable masquerade for ipv6

Which gave partial success - the device now appears behind the router’s ipv6 confirming that the router is in the middle, and in some cases traffic is now going over the correct interface when there is a PBR rule to override the default but things like ipleaktest.com can still route via the ‘wrong’ interface.

Two things I’m wondering:

In the Luci app PBR reports v6 addresses on the v4 only interface. Not sure how it is discovering those but they aren’t correct so maybe that confusion is
Could go the whole hog and remove GUAs from the system making it like a classic v4 firewall/nat66 setup - which is a shame for the other devices in the network that I want to use the default route, but will give it a go.

stangri · November 2, 2022, 8:24pm

It may be. This needs to be investigated further, I'll try to reach out in the coming days/week with some commands to run on your router so I could figure out why pbr things one of your interfaces has an IPv6 address from another interface.

Fiouz · November 2, 2022, 10:39pm

Context
Built OpenWrt 22.03 SNAPSHOT, using source.openwrt.melmac.net commit d1db25c4cc9efadf10966e6e1f4bdb04f110bc74, with pbr-iptables instead of pbr.

The reason for using the -iptables variant is to avoid the build system from pulling nftables into the firmware (waiting for dnsmasq with nftset support before switching to fw4).

/usr/lib/opkg/status

Package: pbr-iptables
Version: 0.9.9-11

Package: luci-app-pbr
Version: 0.9.9-11

Issue

luci-app-pbr does not display the status:

Screenshot

Overall, pbr seems working, but it seems I have to manually reload the service whenever I make a change.

Quick analysis

Browser request

[{"jsonrpc":"2.0","id":6,"method":"call","params":["71019d030b9778aa89884d7e1c00fe25","luci.pbr","getInitStatus",{}]},{"jsonrpc":"2.0","id":7,"method":"call","params":["71019d030b9778aa89884d7e1c00fe25","session","access",{"scope":"uci","object":"pbr","function":"write"}]}]

Server response

[{"jsonrpc":"2.0","id":6,"result":[0,{"pbr":{"enabled":true,"running":true,"running_iptables":true,"running_nft":false,"version":"","gateways":"","warnings":"","errors":""}}]},{"jsonrpc":"2.0","id":7,"result":[0,{"access":true}]}]

Notice version is empty, which seems to make the frontend part unhappy.

A quick guess is that the code does not expect the pbr-iptables variant at https://github.com/stangri/source.openwrt.melmac.net/blob/d1db25c4cc9efadf10966e6e1f4bdb04f110bc74/luci-app-pbr/root/usr/libexec/rpcd/luci.pbr#L18-L24

stangri · November 2, 2022, 11:07pm

Outstanding report @Fiouz , could you please test if it's fixed in https://github.com/stangri/source.openwrt.melmac.net/commit/a2c91a28cfdd7ff0768eee30fb2caf055a278dc3 or by installing luci-app-pbr 0.9.9-13.

d3adc0d3 · November 3, 2022, 12:30am

Hi again,

I was able to successfully use iptables DNAT/SNAT rules on my vpn server to route specific traffic to my local machine through the vpn, running on said local machine.

I now have said vpn client running in openwrt but don't understand exactly how to use openwrt/pbr/nf rules to route the traffic incoming on the vpn interface to the local machine on lan, similarly I don't know how to create rules to route the outgoing traffic from the local machine from lan to vpn for said port.

Here is my current config (though I'm sure it's not right).

The VPN (wireguard) "client" IP in this case is 10.7.0.2 and the local machine LAN IP is 192.168.2.218, the vpn interface is named client3, I believe the vpn gateway is 10.7.0.1?

Any guidance or tips as to how to "debug" or inspect/log incoming packets to openwrt on client3 interface and their path would be much appreciated as I'm pretty lost. Thanks!

stangri · November 3, 2022, 2:16am

Sorry, I'm not quite clear on what the use case is.

Do you want to offer a server on your LAN client via VPN tunnel?