Thank you. Confirmed working.
Really appreciate your sticking with me through this. 
@stangri I think I found a boundary case bug with the new code.
If the conf-dir is in a location that gets backed up for firmware updates (which is exactly why I changed mine), after the upgrade the pbr file is re-placed back in the conf-dir before pbr starts.
At this point I believe that pbr adds the rules to it (again) resulting in duplicate entries.
It shouldn't, that file should be deleted and re-created empty on pbr start.
On a side-note, I've improved the code allowing targeting of a specific dnsmasq instance (which custom confdirs are a part of) so if you could test 1.1.3-2 on your setup I'd appreciate it.
Yeah I haven't been able to reproduce the duplication either.
Installed 1.1.3-2 and it appears to be working well.
[EDIT]
@stangri is it intentional that the pbr dnsmasq file gets created under both conf-dir as well as /var/dnsmasq.d/? I'm seeing that behavior. Stop removes both, start creates both.
[EDIT 2]
I don't know what caused it but it definitely just duplicated entries again.
For some testing of how can I get away from conf-dir this time I removed conf-dir completly and restarted the router. That might line up with why the sysupgrade also caused duplication because obviously that requires a restart.
Jumping on a meeting now so can't reboot for a while to do further checks.
nftset=/whatismyipaddress.com/4#inet#fw4#pbr_pbr_vpn_4_dst_ip_cfg056ff5 # websites
nftset=/whatismyipaddress.com/4#inet#fw4#pbr_pbr_vpn_4_dst_ip_cfg056ff5 # websites
[EDIT 3]
I can reproduce it without a reboot (it's not reboot related - I need to dig into that further and will share something in a different post). Steps to reproduce:
service pbr stop <-- All good. Any pbr files are removed
service pbr start <-- All good. No duplicates
service pbr restart <-- Duplicates created
Doing another restart does not cause a 3rd entry however.
1 Like
Yes it is, the code was simpler if I allowed the file in /var/dnsmasq.d/ as well.
Thank you for providing this, I'll have a look into it on a weekend most likely.
1 Like
As a data point, I generally use auc to bake in the packages. However, in my recent testing I've hit this error all 2-3 times I've upgraded pbr via normal Luci update package process.
"Command failed: ubus call service delete { "name": "pbr" } (Not found)"
Unfortunately the only detail I can provide beyond my config (which is in prior posts) is that the steps I take are as follows:
- Update pbr (gives error which in the end is non-blocking)
- Update luci-app-pbr --- no error of course.
I'll keep my eyes on the logs next time as well.
I believe it's because there's no true daemon for pbr. I don't know how to overcome this. The procd_set_param command /bin/true used to work at some point, but doesn't anymore. @jow -- do you happen to have a suggestion on how to fix this?
OpenWrt 22.03.3 r20028-43d71ad93e
pbr version: 1.1.1-7 using nft
Can anyone help me figure out why upon my Internet Gateway coming back online, sometimes traffic goes out outside my OpenVPN tunnel for a brief moment but long enough for my email to sync ?
I don't quite understand the pbr entries in the Syslog log as it's "Activating" and "Deactivating traffic killswitch" ?
config pbr 'config'
option verbosity '2'
option strict_enforcement '1'
option resolver_set 'none'
option ipv6_enabled '0'
list ignored_interface 'vpnserver'
list ignored_interface 'wgserver'
option boot_timeout '30'
option rule_create_option 'add'
option procd_reload_delay '1'
option webui_show_ignore_target '0'
list webui_supported_protocol 'all'
list webui_supported_protocol 'tcp'
list webui_supported_protocol 'udp'
list webui_supported_protocol 'tcp udp'
list webui_supported_protocol 'icmp'
option enabled '1'
SYSLOG
Sat Sep 16 07:32:39 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:32:40 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:32:40 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:32:41 2023 user.notice pbr: Setting up routing for 'VPN/tun0/0.0.0.0' [✓]
Sat Sep 16 07:32:41 2023 user.notice pbr: Setting up routing for 'WG2Droplet/0.0.0.0' [✓]
Sat Sep 16 07:32:41 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:32:41 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:32:41 2023 user.notice pbr: Reloading pbr WG2Droplet interface routing due to ifup-failed of WG2Droplet ()
Sat Sep 16 07:32:42 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:32:43 2023 daemon.notice netifd: WG2Droplet (13659): Try again: `www.spiramentum.ca:51820'. Trying again in 1.00 seconds...
Sat Sep 16 07:32:43 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:32:44 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:32:44 2023 user.notice pbr: Setting up routing for 'VPN/tun0/0.0.0.0' [✓]
Sat Sep 16 07:32:44 2023 daemon.notice netifd: Interface 'WG2Droplet' is now up
Sat Sep 16 07:32:44 2023 daemon.notice netifd: Network device 'WG2Droplet' link is up
Sat Sep 16 07:32:45 2023 user.notice pbr: Setting up routing for 'WG2Droplet/0.0.0.0' [✓]
Sat Sep 16 07:32:45 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:32:45 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:32:45 2023 user.notice firewall: Reloading firewall due to ifup of WG2Droplet (WG2Droplet)
Sat Sep 16 07:32:46 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net.WG2Droplet/main.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net.WG2Droplet/received.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net.WG2Droplet/sent.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.WG2Droplet/main.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.WG2Droplet/received.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.WG2Droplet/sent.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.WG2Droplet/multicast.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/main.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/RcvbufErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/SndbufErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/InErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/NoPorts.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/InCsumErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.udperrors/IgnoredMulti.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/main.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/InErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/OutErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/InCsumErrors.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/InDestUnreachs.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/InPktTooBigs.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/InTimeExcds.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/InParmProblems.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/OutDestUnreachs.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/OutPktTooBigs.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/OutTimeExcds.db.
Sat Sep 16 07:32:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv6.icmperrors/OutParmProblems.db.
Sat Sep 16 07:32:47 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:32:47 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:32:47 2023 user.notice pbr: Setting up routing for 'VPN/tun0/0.0.0.0' [✓]
Sat Sep 16 07:32:48 2023 user.notice pbr: Setting up routing for 'WG2Droplet/10.0.0.1' [✓]
Sat Sep 16 07:32:48 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:32:48 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:32:48 2023 user.notice pbr: Reloading pbr due to firewall action: includes
Sat Sep 16 07:32:49 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:32:50 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:32:51 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:32:51 2023 user.notice pbr: Setting up routing for 'VPN/tun0/0.0.0.0' [✓]
Sat Sep 16 07:32:51 2023 user.notice pbr: Setting up routing for 'WG2Droplet/10.0.0.1' [✓]
Sat Sep 16 07:32:52 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:32:52 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:32:52 2023 user.notice root: starting ntpclient
Sat Sep 16 07:32:52 2023 user.notice nlbwmon: Reloading nlbwmon due to ifup of WG2Droplet (WG2Droplet)
Sat Sep 16 07:32:52 2023 user.notice pbr: Reloading pbr WG2Droplet interface routing due to ifup of WG2Droplet (WG2Droplet)
Sat Sep 16 07:32:53 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:32:55 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:32:55 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:32:56 2023 user.notice pbr: Setting up routing for 'VPN/tun0/0.0.0.0' [✓]
Sat Sep 16 07:32:56 2023 user.notice pbr: Setting up routing for 'WG2Droplet/10.0.0.1' [✓]
Sat Sep 16 07:32:56 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:32:56 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:33:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_drops.br_lan/main.db.
Sat Sep 16 07:33:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_drops.br_lan/inbound.db.
Sat Sep 16 07:33:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_drops.br_lan/outbound.db.
Sat Sep 16 07:33:44 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 07:33:44 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcpsock/main.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcpsock/CurrEstab.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcppackets/main.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcppackets/InSegs.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcppackets/OutSegs.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcpopens/main.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcpopens/ActiveOpens.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcpopens/PassiveOpens.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcphandshake/main.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcphandshake/EstabResets.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcphandshake/OutRsts.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcphandshake/AttemptFails.db.
Sat Sep 16 07:33:46 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.tcphandshake/TCPSynRetrans.db.
Sat Sep 16 07:33:59 2023 daemon.warn openvpn(LA2)[5560]: WARNING: --ns-cert-type is DEPRECATED. Use --remote-cert-tls instead.
Sat Sep 16 07:33:59 2023 daemon.warn openvpn(LA2)[5560]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sat Sep 16 07:33:59 2023 daemon.notice openvpn(LA2)[5560]: Outgoing Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Sat Sep 16 07:33:59 2023 daemon.notice openvpn(LA2)[5560]: Incoming Control Channel Authentication: Using 512 bit message hash 'SHA512' for HMAC authentication
Sat Sep 16 07:33:59 2023 daemon.notice openvpn(LA2)[5560]: TCP/UDP: Preserving recently used remote address: [AF_INET]23.230.125.225:1195
Sat Sep 16 07:33:59 2023 daemon.notice openvpn(LA2)[5560]: Socket Buffers: R=[212992->1048576] S=[212992->1048576]
Sat Sep 16 07:33:59 2023 daemon.notice openvpn(LA2)[5560]: UDP link local: (not bound)
Sat Sep 16 07:33:59 2023 daemon.notice openvpn(LA2)[5560]: UDP link remote: [AF_INET]23.230.125.225:1195
Sat Sep 16 07:34:00 2023 daemon.notice openvpn(LA2)[5560]: TLS: Initial packet from [AF_INET]23.230.125.225:1195, sid=db2d6c77 e3623927
Sat Sep 16 07:34:00 2023 daemon.notice openvpn(LA2)[5560]: VERIFY OK: depth=1, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
Sat Sep 16 07:34:00 2023 daemon.notice openvpn(LA2)[5560]: VERIFY OK: nsCertType=SERVER
Sat Sep 16 07:34:00 2023 daemon.notice openvpn(LA2)[5560]: VERIFY X509NAME OK: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10939-0a, emailAddress=support@expressvpn.com
Sat Sep 16 07:34:00 2023 daemon.notice openvpn(LA2)[5560]: VERIFY OK: depth=0, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10939-0a, emailAddress=support@expressvpn.com
Sat Sep 16 07:34:01 2023 daemon.notice openvpn(LA2)[5560]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
Sat Sep 16 07:34:01 2023 daemon.notice openvpn(LA2)[5560]: [Server-10939-0a] Peer Connection Initiated with [AF_INET]23.230.125.225:1195
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: SENT CONTROL [Server-10939-0a]: 'PUSH_REQUEST' (status=1)
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 10.170.0.1,comp-lzo no,route 10.170.0.1,topology net30,ping 10,ping-restart 60,ifconfig 10.170.1.70 10.170.1.69,peer-id 105,cipher AES-256-GCM'
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: timers and/or timeouts modified
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: compression parms modified
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: --ifconfig/up options modified
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: route options modified
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: peer-id set
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: adjusting link_mtu to 1629
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: OPTIONS IMPORT: data channel crypto options modified
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: Data Channel: using negotiated cipher 'AES-256-GCM'
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: NCP: overriding user-set keysize with default
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: net_route_v4_best_gw query: dst 0.0.0.0
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: net_route_v4_best_gw result: via 192.168.250.1 dev eth0
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: TUN/TAP device tun0 opened
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: net_iface_mtu_set: mtu 1500 for tun0
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: net_iface_up: set tun0 up
Sat Sep 16 07:34:02 2023 daemon.notice netifd: Interface 'VPN' is enabled
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: net_addr_ptp_v4_add: 10.170.1.70 peer 10.170.1.69 dev tun0
Sat Sep 16 07:34:02 2023 daemon.notice netifd: Network device 'tun0' link is up
Sat Sep 16 07:34:02 2023 daemon.notice netifd: Interface 'VPN' has link connectivity
Sat Sep 16 07:34:02 2023 daemon.notice netifd: Interface 'VPN' is setting up now
Sat Sep 16 07:34:02 2023 daemon.notice openvpn(LA2)[5560]: /usr/libexec/openvpn-hotplug up LA2 tun0 1500 1557 10.170.1.70 10.170.1.69 init
Sat Sep 16 07:34:02 2023 daemon.info avahi-daemon[4446]: Joining mDNS multicast group on interface tun0.IPv6 with address fe80::fb5e:d7cf:84ba:a200.
Sat Sep 16 07:34:02 2023 daemon.info avahi-daemon[4446]: New relevant interface tun0.IPv6 for mDNS.
Sat Sep 16 07:34:02 2023 daemon.notice netifd: Interface 'VPN' is now up
Sat Sep 16 07:34:02 2023 daemon.info avahi-daemon[4446]: Registering new address record for fe80::fb5e:d7cf:84ba:a200 on tun0.*.
Sat Sep 16 07:34:02 2023 daemon.info avahi-daemon[4446]: Joining mDNS multicast group on interface tun0.IPv4 with address 10.170.1.70.
Sat Sep 16 07:34:02 2023 daemon.info avahi-daemon[4446]: New relevant interface tun0.IPv4 for mDNS.
Sat Sep 16 07:34:02 2023 daemon.info avahi-daemon[4446]: Registering new address record for 10.170.1.70 on tun0.IPv4.
Sat Sep 16 07:34:02 2023 user.notice firewall: Reloading firewall due to ifup of VPN (tun0)
Sat Sep 16 07:34:03 2023 user.notice pbr: Updated interface is an OpenVPN tunnel, restarting.
Sat Sep 16 07:34:03 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:34:04 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net.tun0/main.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net.tun0/received.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net.tun0/sent.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_operstate.tun0/main.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_operstate.tun0/state.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_carrier.tun0/main.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_carrier.tun0/carrier.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_mtu.tun0/main.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_mtu.tun0/mtu.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.tun0/main.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.tun0/received.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.tun0/sent.db.
Sat Sep 16 07:34:04 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/net_packets.tun0/multicast.db.
Sat Sep 16 07:34:04 2023 daemon.notice openvpn(LA2)[5560]: net_route_v4_add: 23.230.125.225/32 via 192.168.250.1 dev [NULL] table 0 metric -1
Sat Sep 16 07:34:04 2023 daemon.notice openvpn(LA2)[5560]: net_route_v4_add: 0.0.0.0/1 via 10.170.1.69 dev [NULL] table 0 metric -1
Sat Sep 16 07:34:04 2023 daemon.notice openvpn(LA2)[5560]: net_route_v4_add: 128.0.0.0/1 via 10.170.1.69 dev [NULL] table 0 metric -1
Sat Sep 16 07:34:04 2023 daemon.notice openvpn(LA2)[5560]: net_route_v4_add: 10.170.0.1/32 via 10.170.1.69 dev [NULL] table 0 metric -1
Sat Sep 16 07:34:04 2023 daemon.warn openvpn(LA2)[5560]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Sat Sep 16 07:34:04 2023 daemon.notice openvpn(LA2)[5560]: Initialization Sequence Completed
Sat Sep 16 07:34:04 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:34:05 2023 user.notice pbr: Setting up routing for 'VPN/tun0/10.170.1.70' [✓]
Sat Sep 16 07:34:05 2023 user.notice pbr: Setting up routing for 'WG2Droplet/10.0.0.1' [✓]
Sat Sep 16 07:34:05 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:34:05 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:34:06 2023 user.notice pbr: Reloading pbr due to firewall action: includes
Sat Sep 16 07:34:06 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:34:08 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:34:08 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:34:09 2023 user.notice pbr: Setting up routing for 'VPN/tun0/10.170.1.70' [✓]
Sat Sep 16 07:34:09 2023 user.notice pbr: Setting up routing for 'WG2Droplet/10.0.0.1' [✓]
Sat Sep 16 07:34:09 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:34:10 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:34:10 2023 user.notice nlbwmon: Reloading nlbwmon due to ifup of VPN (tun0)
Sat Sep 16 07:34:10 2023 user.notice pbr: Reloading pbr VPN interface routing due to ifup of VPN (tun0)
Sat Sep 16 07:34:11 2023 user.notice pbr: Updated interface is an OpenVPN tunnel, restarting.
Sat Sep 16 07:34:11 2023 user.notice pbr: Activating traffic killswitch [✓]
Sat Sep 16 07:34:12 2023 user.notice pbr: Setting up routing for 'wan/eth0/192.168.250.1' [✓]
Sat Sep 16 07:34:13 2023 user.notice pbr: Setting up routing for 'wwan/0.0.0.0' [✓]
Sat Sep 16 07:34:13 2023 user.notice pbr: Setting up routing for 'VPN/tun0/10.170.1.70' [✓]
Sat Sep 16 07:34:14 2023 user.notice pbr: Setting up routing for 'WG2Droplet/10.0.0.1' [✓]
Sat Sep 16 07:34:14 2023 user.notice pbr: Deactivating traffic killswitch [✓]
Sat Sep 16 07:34:14 2023 user.notice pbr: service monitoring interfaces: wan wwan VPN WG2Droplet
Sat Sep 16 07:35:00 2023 cron.err crond[3657]: USER root pid 23703 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 07:35:00 2023 cron.err crond[3657]: USER root pid 23704 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 07:38:44 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 07:38:44 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 07:40:00 2023 cron.err crond[3657]: USER root pid 24128 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 07:40:00 2023 cron.err crond[3657]: USER root pid 24129 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 07:41:26 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/cpu.cpu3_softirqs/HI.db.
Sat Sep 16 07:43:44 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 07:43:44 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 07:45:00 2023 cron.err crond[3657]: USER root pid 24496 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 07:45:00 2023 cron.err crond[3657]: USER root pid 24497 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 07:45:19 2023 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 192.168.8.40 88:9f:6f:e5:5b:44
Sat Sep 16 07:45:19 2023 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 192.168.8.40 88:9f:6f:e5:5b:44 SergioTab
Sat Sep 16 07:47:59 2023 user.info : luci: accepted login on / for root from 192.168.8.30
Sat Sep 16 07:48:00 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.sockstat_tcp_mem/main.db.
Sat Sep 16 07:48:00 2023 daemon.info netdata[8749]: Initializing file /var/cache/netdata/ipv4.sockstat_tcp_mem/mem.db.
Sat Sep 16 07:48:44 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 07:48:44 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 07:50:00 2023 cron.err crond[3657]: USER root pid 25031 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 07:50:00 2023 cron.err crond[3657]: USER root pid 25032 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 07:53:44 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 07:53:44 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 07:54:32 2023 authpriv.info dropbear[25565]: Child connection from 192.168.8.30:55492
Sat Sep 16 07:54:36 2023 authpriv.notice dropbear[25565]: Password auth succeeded for 'root' from 192.168.8.30:55492
Sat Sep 16 07:55:00 2023 cron.err crond[3657]: USER root pid 25634 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 07:55:00 2023 cron.err crond[3657]: USER root pid 25635 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 07:58:44 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 07:58:44 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 07:59:16 2023 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 192.168.8.30 00:23:24:71:17:73
Sat Sep 16 07:59:16 2023 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 192.168.8.30 00:23:24:71:17:73 m93p
Sat Sep 16 08:00:00 2023 cron.err crond[3657]: USER root pid 26011 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:00:00 2023 cron.err crond[3657]: USER root pid 26012 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:03:45 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:03:45 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 08:05:00 2023 cron.err crond[3657]: USER root pid 26417 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:05:00 2023 cron.err crond[3657]: USER root pid 26418 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:08:45 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:08:45 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 08:10:00 2023 cron.err crond[3657]: USER root pid 26815 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:10:00 2023 cron.err crond[3657]: USER root pid 26816 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:13:45 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:13:45 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 08:15:00 2023 cron.err crond[3657]: USER root pid 27181 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:15:00 2023 cron.err crond[3657]: USER root pid 27182 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:15:19 2023 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 192.168.8.40 88:9f:6f:e5:5b:44
Sat Sep 16 08:15:19 2023 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 192.168.8.40 88:9f:6f:e5:5b:44 SergioTab
Sat Sep 16 08:18:45 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:18:45 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 08:20:00 2023 cron.err crond[3657]: USER root pid 27581 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:20:00 2023 cron.err crond[3657]: USER root pid 27582 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:23:45 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:23:45 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 08:25:00 2023 cron.err crond[3657]: USER root pid 27947 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:25:00 2023 cron.err crond[3657]: USER root pid 27948 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:27:09 2023 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 192.168.8.30 00:23:24:71:17:73
Sat Sep 16 08:27:09 2023 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 192.168.8.30 00:23:24:71:17:73 m93p
Sat Sep 16 08:28:45 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:28:45 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
Sat Sep 16 08:30:00 2023 cron.err crond[3657]: USER root pid 28340 cmd /custom_commands/wg0_undo_route.sh
Sat Sep 16 08:30:00 2023 cron.err crond[3657]: USER root pid 28341 cmd /custom_commands/Restart_OpenVPN.sh
Sat Sep 16 08:31:40 2023 daemon.notice openvpn(LA2)[5560]: VERIFY OK: depth=1, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=ExpressVPN CA, emailAddress=support@expressvpn.com
Sat Sep 16 08:31:40 2023 daemon.notice openvpn(LA2)[5560]: VERIFY OK: nsCertType=SERVER
Sat Sep 16 08:31:40 2023 daemon.notice openvpn(LA2)[5560]: VERIFY X509NAME OK: C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10939-0a, emailAddress=support@expressvpn.com
Sat Sep 16 08:31:40 2023 daemon.notice openvpn(LA2)[5560]: VERIFY OK: depth=0, C=VG, ST=BVI, O=ExpressVPN, OU=ExpressVPN, CN=Server-10939-0a, emailAddress=support@expressvpn.com
Sat Sep 16 08:31:41 2023 daemon.notice openvpn(LA2)[5560]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Sat Sep 16 08:31:41 2023 daemon.notice openvpn(LA2)[5560]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Sat Sep 16 08:31:41 2023 daemon.notice openvpn(LA2)[5560]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
Sat Sep 16 08:33:46 2023 daemon.notice netifd: wan (1078): udhcpc: sending renew to server 192.168.250.1
Sat Sep 16 08:33:46 2023 daemon.notice netifd: wan (1078): udhcpc: lease of 192.168.250.12 obtained from 192.168.250.1, lease time 600
I'm not seeing any policies defined, so not sure what do you expect.
Oupss... Obviously, I need to do my homework first.
My goal is to have all traffic from my LAN going through OpenVPN and have a 'Kill swich' feature enable.
Thanks
You don't need pbr. You need to edit the following section of /etc/config/firewall:
config forwarding
option src 'lan'
option dest 'wan'
and instead of 'wan' in that you need to put your VPN interface. For further help, please create a separate thread, as this is completely unrelated to pbr.
1 Like
It's much clearer to me now...thank you again.
Ah, then I can use pbr to allow specific devices on my LAN to 'Bypass VPN'.
No, if you disallow traffic from LAN to WAN, the pbr can't make it magically flow to WAN (to bypass VPN).
Thank you again for the prompt reply.
Ok I understand, so I have LAN to WAN traffic re-enable.
The default gateway is OpenVPN, all Internet traffic goes through the VPN Tunnel
A PBR policy allows me to Bypass VPN for my specific LAN device.
All good...
But I struggle to implement the Internet Kill Switch when the VPN Tunnel is down with:
option strict_enforcement '1'
Another hint would be very much appreciated.
If there are no policies to enforce, the strict enforcement won't work.
OK, you have set me on the right track 
After going over the README very carefully again and setting a pbr policy to route all traffic to VPN Tunnel, I got the Kill-Switch working.
With this new understanding of how pbr is working, I will set a few more policies that I need in the next few days.
Thank you for your support and for your time.
1 Like
Great job!
Please do consider either starting a new thread or posting here what you did to help others with similar use cases.
I'd gladly welcome your contribution to the README either in a form of PR against docs repo or a little write-up mentioned above I could use to create a special killswitch section of the README.
1 Like
For the KillSwitch section:
First, read, understand, and apply the Basic OpenVPN Client Config from the README
I did struggled a bit here to get PBR working with my VPN provider .ovpn config file until adding dev 'ovpnc0' in the proper section of /etc/config/openvpn:
config openvpn 'My_VPN_Provider'
option dev 'ovpnc0'
option config '/etc/openvpn/MY_VPN_Provider.ovpn'
option enabled '1'
Next the essential PBR policy routing LAN (192.168.8.0/24) traffic to VPN_Tunnel:
/etc/config/pbr
config pbr 'config'
option strict_enforcement '1'
...
config policy
option name 'All_Local_Devices'
option interface 'vpn_client'
option src_addr '192.168.8.0/24'
This PBR routing policy combined with the strict_enforcement option is how to implement the Kill-Switch.
VPN_Tunnel_Bypass : The VPN_Tunnel can easily be bypass by adding a device-specific policy in PBR routing it directly to the WAN
config policy
option name 'Specific_LAN_Device'
option src_addr '192.168.8.40'
option interface 'wan'
option enabled '1'
p.s.:For reasons beyond my present level of routing understanding, this PBR / KillSwitch works when the VPN tunnel is used as a default gateway...otherwise something needs to be added somewhere...I am sure it's trivial
2 Likes
In addition to the service restart issue, I believe the most recent commit has caused an issue.
Specifically, I believe it is due to the "tree-wide: busybox dependencies" update, which has resulted in this error immediately upon SSHing into the box:
ash: fgrep: not found
In my case, I have the following (full / non-busybox) packages installed:
sed 4.9-1 ~73.86 KiB
gawk 5.2.1-2 ~279.91 KiB
grep 3.8-2 ~97.87 KiB
I think that this issue is likely happening because as part of the pbr install, it mentioned updating these utilities configuration, and I'm using the non-busybox version of these.
I have a standaone grep installed and I can't reproduce this. Do you install sed, gawk and grep manually or are you saying they all got installed because of the new pbr Makefile?
PS.
xg-135r3 in ~ # ls -la $(which grep)
lrwxrwxrwx 1 root root 21 Jun 26 04:20 /bin/grep -> /usr/libexec/grep-gnu
xg-135r3 in ~ # ls -la $(which fgrep)
lrwxrwxrwx 1 root root 22 Jun 26 04:20 /bin/fgrep -> /usr/libexec/fgrep-gnu