[Solved] rPI VPN + USG = no VPN traffic

burntfish · January 9, 2023, 6:20am

Hello there!

Hoping for some guidance / help to resolve an issue I've been grappling with for the last few weeks.

My home network is a Wireless ISP, connected to a Unifi Secure Gateway (USG), then to a Unifi Switch that provides both wired and wireless connectivity (the latter through Unifi APs). I have a secure private network, and an open WiFi guest network.

I have added a Raspberry PI 4 B between the USG and Switch, and the config for the ExpressVPN is working with no errors, and I can ping / traceroute on the vpn/tun0 interface. However all traffic regardless of using --pull-filter ignore redirect-gateway, or route-noexec, etc... all traffic passes between the LAN and WAN on the rPI and never goes over the VPN.

The outcome I'm hoping to achieve is the following:

Push all LAN traffic through the VPN; then once working...
Implement Split Tunnelling (perhaps using PBR / VPN-Bypass) so that only certain devices use the VPN, e.g. TV/Netflix, access my remote server from my home server, have guest-clients not use the VPN, and then selectively add private devices to use the VPN in the future as required.

The network looks like this:

Please note that I don't wish to use the rPI for WiFi as it just can't cover the distances required. Also note that I have set the firewall back to something that appears to work with a LAN-to-LAN config (including a VPN that gets not traffic!).

Having followed MANY(!) threads here and on Reddit, I'm stuck. I feel this is a routing and firewall issue but having tried lots of different variations, I simply can't get it to what I want. All help greatly appreciated.

Configs as per @trendy output, as follows:

root@OpenWrt:~# ubus call system board; \
> uci export network; uci export firewall; \
> head -n -0 /etc/firewall.user; \
> ip -4 addr ; ip -4 ro li tab all ; ip -4 ru
{
        "kernel": "5.10.146",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 4 Model B Rev 1.4",
        "board_name": "raspberrypi,4-model-b",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.2",
                "revision": "r19803-9a599fee93",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 22.03.2 r19803-9a599fee93"
        }
}
package network

config interface 'loopback'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option device 'lo'

config globals 'globals'
        option ula_prefix 'fd98:b5b9:9ffb::/48'

config interface 'lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option gateway '192.168.1.1'
        list dns '8.8.8.8'
        option delegate '0'
        option device 'br-lan'
        option ipaddr '192.168.1.4'

config interface 'wan'
        option device 'eth1'
        option proto 'dhcp'

config device
        option name 'br-lan'
        option type 'bridge'
        option ipv6 '0'
        option acceptlocal '1'
        list ports 'eth0'
        list ports 'eth1'

config device
        option name 'eth0'
        option ipv6 '0'
        option acceptlocal '1'

config device
        option name 'eth1'
        option acceptlocal '1'
        option ipv6 '0'

config device
        option name 'tun0'
        option acceptlocal '1'
        option ipv6 '0'

config interface 'vpn'
        option proto 'static'
        option device 'tun0'

package firewall

config defaults
        option output 'ACCEPT'
        option synflood_protect '1'
        option input 'ACCEPT'
        option forward 'REJECT'

config zone 'vpn'
        option name 'vpn'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list device 'tun0'
        option masq '1'

config zone 'lan'
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        list network 'lan'
        option forward 'ACCEPT'

config zone 'wan'
        option name 'wan'
        option output 'ACCEPT'
        option mtu_fix '1'
        option input 'REJECT'
        option forward 'REJECT'
        option masq '1'
        list network 'wan'
        list network 'vpn'

config forwarding
        option src 'lan'
        option dest 'wan'

config include
        option path '/etc/firewall.user'

config forwarding
        option src 'vpn'
        option dest 'lan'

config forwarding
        option src 'lan'
        option dest 'vpn'

head: /etc/firewall.user: No such file or directory
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group defaul                                                                                  t qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP gro                                                                                  up default qlen 1000
    inet 192.168.1.4/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
7: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state                                                                                   UNKNOWN group default qlen 500
    inet 10.159.0.194 peer 10.159.0.193/32 scope global tun0
       valid_lft forever preferred_lft forever
0.0.0.0/1 via 10.159.0.193 dev tun0
default via 192.168.1.1 dev br-lan proto static
10.159.0.1 via 10.159.0.193 dev tun0
10.159.0.193 dev tun0 proto kernel scope link src 10.159.0.194
45.x.x.x via 192.168.1.1 dev br-lan [ExpressVPN IP]
128.0.0.0/1 via 10.159.0.193 dev tun0
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.4
local 10.159.0.194 dev tun0 table local proto kernel scope host src 10.159.0.194                                                                                  
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0                                                                                  .1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168                                                                                  .1.4
local 192.168.1.4 dev br-lan table local proto kernel scope host src 192.168.1.4                                                                                  
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.1                                                                                  68.1.4
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

###OpenVPN config '#' are some attempts to get it to work...

dev tun
fast-io
persist-key
persist-tun
nobind
remote australia-sydney-ca-version-2.expressnetw.com 1195

remote-random
pull
#pull-filter ignore redirect-gateway
#pull-filter ignore dhcp-option
#pull-filter ignore route
#route-noexec
#route-nopull
#route 0.0.0.0 192.0.0.0 net_gateway
#route 10.0.0.0 192.0.0.0 net_gateway
#route 128.0.0.0 192.0.0.0 net_gateway
#route 192.0.0.0 192.0.0.0 net_gateway
comp-lzo no
tls-client
verify-x509-name Server name-prefix
remote-cert-tls server
key-direction 1
route-method exe
route-delay 2
tun-mtu 1500
fragment 1300
mssfix 1200
verb 3
data-ciphers AES-256-CBC
keysize 256
auth SHA512
sndbuf 524288
rcvbuf 524288
auth-user-pass "/etc/openvpn/eVPN.auth"
auth-nocache

psherman · January 9, 2023, 7:52am

First, you need to be aware that what you are attempting to do will, by default, break your ability to connect to the USG via your Unifi Network Application (from a cloudkey or computer on your lan). Your USG will effectively appear offline based on this configuration since UNA will not be able to reach it. So you'll need to play some games to get that working again.

Anyway, your lan and wan cannot have the same subnet. One of them has to change... and this is also going to make your Unifi network rather unhappy.

I think that there is actually a better way to handle this, and that would be to create another network on the USG and set the DHCP options in the USG such that it advertises a non-USG gateway --- the Pi that is running the VPN. You could even do this with the main LAN. This way, you don't disturb the existing Unifi network, but you can tell the devices on the network to use the Pi as the gateway instead of the USG.

There are many problems with your current config, though... I'm going to make some suggestions based on my idea that you'll just set the DHCP advertised gateway to point to the Pi... In this case, you can actually remove eth1 entirely (the USB connected Ethernet adapter).

remove the device 'eth1' line from below

make the br-lan as follows:

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

delete all of this:

burntfish:

config device
        option name 'eth0'
        option ipv6 '0'
        option acceptlocal '1'

config device
        option name 'eth1'
        option acceptlocal '1'
        option ipv6 '0'

config device
        option name 'tun0'
        option acceptlocal '1'
        option ipv6 '0'

config interface 'vpn'
        option proto 'static'
        option device 'tun0'

Change the VPN zone to this:

config zone 'vpn'
        option name 'vpn'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list device 'tun0'
        option masq '1'

Remove the vpn network from the wan zone

burntfish:

config zone 'wan'
        option name 'wan'
        option output 'ACCEPT'
        option mtu_fix '1'
        option input 'REJECT'
        option forward 'REJECT'
        option masq '1'
        list network 'wan'
        list network 'vpn'

remove this

We'll have to do some more work and it's possible we'll change a few things in the strategy, but this should at least clean up a bunch of the things that are currently wrong.. post the configs when you're done with my above suggestions.

burntfish · January 9, 2023, 8:51pm

Thanks for your help @psherman. Your assessment of what I want to achieve and the problem I have is spot on. The USG was not happy...

To test that I understand, are you proposing that rather than have all traffic passing through the Pi, the clients that will use the VPN will instead point to the Pi as their gateway to go through the VPN? Will this require changes to the clients IP address settings?

All changes you suggested have been made, the WAN is down showing "Error: Network device is not present", LAN and VPN up, and I can traceroute on tun0. Outputs as follows:

root@OpenWrt:~# ubus call system board; \
> uci export network; uci export firewall; \
> head -n -0 /etc/firewall.user; \
> ip -4 addr ; ip -4 ro li tab all ; ip -4 ru
{
        "kernel": "5.10.146",
        "hostname": "OpenWrt",
        "system": "ARMv8 Processor rev 3",
        "model": "Raspberry Pi 4 Model B Rev 1.4",
        "board_name": "raspberrypi,4-model-b",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "22.03.2",
                "revision": "r19803-9a599fee93",
                "target": "bcm27xx/bcm2711",
                "description": "OpenWrt 22.03.2 r19803-9a599fee93"
        }
}
package network

config interface 'loopback'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option device 'lo'

config globals 'globals'
        option ula_prefix 'fd98:b5b9:9ffb::/48'

config interface 'lan'
        option proto 'static'
        option netmask '255.255.255.0'
        option gateway '192.168.1.1'
        list dns '8.8.8.8'
        option delegate '0'
        option device 'br-lan'
        option ipaddr '192.168.1.4'

config interface 'wan'
        option proto 'dhcp'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth0'

package firewall

config defaults
        option output 'ACCEPT'
        option synflood_protect '1'
        option input 'ACCEPT'
        option forward 'REJECT'

config zone 'vpn'
        option name 'vpn'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list device 'tun0'
        option masq '1'

config zone 'lan'
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        list network 'lan'
        option forward 'ACCEPT'

config zone 'wan'
        option name 'wan'
        option output 'ACCEPT'
        option mtu_fix '1'
        option input 'REJECT'
        option forward 'REJECT'
        option masq '1'
        list network 'wan'

config forwarding
        option src 'lan'
        option dest 'wan'

config include
        option path '/etc/firewall.user'

config forwarding
        option src 'lan'
        option dest 'vpn'

head: /etc/firewall.user: No such file or directory
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 192.168.1.4/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
default via 192.168.1.1 dev br-lan proto static
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.4
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev br-lan table local proto kernel scope link src 192.168.1.4
local 192.168.1.4 dev br-lan table local proto kernel scope host src 192.168.1.4
broadcast 192.168.1.255 dev br-lan table local proto kernel scope link src 192.168.1.4
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Many thanks!

psherman · January 9, 2023, 9:19pm

this is not a problem and can be ignored. Or you can remove the wan interface entirely, if you want.

precisely. So the pi will have one physical connection to the network (connected to the switch). The USG will also connect to the switch, so it's really like the Pi is just any other device on the network.

Not the lan addresses, but the gateway, yes. Go into UNA and modify your LAN network such that the gateway that is specified by the DHCP server is 192.168.1.4 (it will be 'automatic' right now) If you need help finding that setting, lmk and I'll give you the breadcrumbs.

Then, force a client to renew the DHCP lease and it should get a new router/gateway address. From there, run a trace route and see if it goes through the tunnel.

burntfish · January 9, 2023, 10:00pm

Marvelous... well that works well! Thank you!

UNA DHCP gateway changed to 192.168.1.4, and now my external IP is the ExpressVPN. I've also run DNS / WebRTC leak test and both passed.

Is the next stage to use VPN-Bypass to specify which clients use the VPN?

psherman · January 9, 2023, 10:02pm

You're welcome! Glad it is working!

You can do this in two ways:

setup Policy Based Routing/VPN bypass on the Pi with OpenWrt
setup two independent networks on your Unifi system so that one uses the VPN and the other does not

(or combine both methods)

Meanwhile, since the main issue appears to be resolved now...

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.

burntfish · January 9, 2023, 10:28pm

Thanks again. I was just going in circles.

system · January 19, 2023, 10:28pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.