I'm using WG with PBR to connect to a remote server. During the first 4-5 minutes after restarting the tunnel or router, pings fail about 60% of the time with a "Destination port unreachable" error. After that 4-5 minutes pings are successful 99.9% of the time. Laptops and phones using WG client apps to connect to the same server though this router work immediately without error.
Is this expected behavior? If not, I'll post my configuration files.
Thanks.
No. Not expected at all.
Yes, please do.
Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
Remember to redact passwords, MAC addresses and any public IP addresses you may have:
ubus call system board
cat /etc/config/network
cat /etc/config/firewall
Thanks for the reply.
The two tunnels that are having this problem are "WG_EvilCorp" and "WG_WA_Pi_Client". "WG_SoB_Server" is only used for incoming connections from phones and PC's and seems to be OK. The other two tunnels may also have the problem, but aren't working right now due to an equipment failure at the remote end.
root@SoB_OpenWrt:/etc/config# **ubus call system board**
{
"kernel": "5.15.134",
"hostname": "SoB_OpenWrt",
"system": "Qualcomm Atheros QCA956X ver 1 rev 0",
"model": "TP-Link Archer A7 v5",
"board_name": "tplink,archer-a7-v5",
"rootfs_type": "squashfs",
"release": {
"distribution": "OpenWrt",
"version": "23.05.0",
"revision": "r23497-6637af95aa",
"target": "ath79/generic",
"description": "OpenWrt 23.05.0 r23497-6637af95aa"
}
}
root@SoB_OpenWrt:/etc/config#
The specific WG entries are indented to make it a bit more readable.
root@SoB_OpenWrt:/etc/config# **cat network**
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth0.1'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option ipaddr '10.1.0.1'
option netmask '255.255.255.0'
option ip6assign '60'
option delegate '0'
config device
option name 'eth0.2'
option macaddr 'xx:xx:xx:xx:xx:xx'
config interface 'wan'
option device 'eth0.2'
option proto 'dhcp'
option delegate '0'
config switch
option name 'switch0'
option reset '1'
option enable_vlan '1'
config switch_vlan
option device 'switch0'
option vlan '1'
option ports '2 3 4 5 0t'
config switch_vlan
option device 'switch0'
option vlan '2'
option ports '1 0t'
config interface 'WG_SoCal_Pi5'
option proto 'wireguard'
option private_key 'xxxxxxxxxx='
list addresses '10.6.0.9/24'
list dns '10.6.0.1'
config wireguard_WG_SoCal_Pi5
option description 'Imported peer configuration'
option public_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
option endpoint_host 'siteSoCal.anydns.org'
option endpoint_port 'xxxxx'
list allowed_ips '0.0.0.0/0'
config interface 'WG_EvilCorp'
option proto 'wireguard'
option private_key 'xxxxxxxxxx='
list addresses '10.237.131.6/24'
list dns '208.67.222.222'
list dns '208.67.220.220'
config wireguard_WG_EvilCorp
option description 'Imported peer configuration'
option public_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
list allowed_ips '0.0.0.0/0'
option endpoint_host 'xxx.xxx.xxx.xxx'
option endpoint_port 'xxxxx'
option persistent_keepalive '25'
config interface 'WG_SoB_Server'
option proto 'wireguard'
option private_key 'xxxxxxxxxx='
list addresses '10.4.0.1/24'
option listen_port 'xxxxx'
config wireguard_WG_SoB_Server
option public_key 'xxxxxxxxxx='
option private_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
option description 'User_1__Phone'
list allowed_ips '10.4.0.2/32'
option persistent_keepalive '25'
option route_allowed_ips '1'
config wireguard_WG_SoB_Server
option description 'Surface'
option public_key 'xxxxxxxxxx='
option private_key 'xxxxxxxxxx='
list allowed_ips '10.4.0.3/32'
option persistent_keepalive '25'
config wireguard_WG_SoB_Server
option description 'User_2__Phone'
option public_key 'xxxxxxxxxx='
option private_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
list allowed_ips '10.4.0.4/32'
option persistent_keepalive '25'
config wireguard_WG_SoB_Server
option description 'User_1__Desktop'
option public_key 'xxxxxxxxxx='
option private_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
list allowed_ips '10.4.0.5/32'
config interface 'WG_SoCal_Client'
option proto 'wireguard'
option private_key 'xxxxxxxxxx='
list addresses '10.17.0.2/32'
list dns '10.0.0.1'
config route
option interface 'lan'
option target '10.237.131.0/24'
option gateway '10.237.131.1'
config interface 'WG_WA_Pi_Client'
option proto 'wireguard'
option private_key 'xxxxxxxxxx='
list addresses '10.104.121.16/24'
list dns '1.1.1.1'
list dns '1.0.0.1'
option listen_port 'xxxxx'
config wireguard_WG_WA_Pi_Client
option description 'Imported peer configuration'
option public_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
list allowed_ips '0.0.0.0/0'
option endpoint_host 'siteWA.anydns.org'
option endpoint_port 'xxxxx'
option persistent_keepalive '25'
config wireguard_WG_SoCal_Client
option description 'Imported peer configuration'
option public_key 'xxxxxxxxxx='
option preshared_key 'xxxxxxxxxx='
option persistent_keepalive '25'
option endpoint_host 'siteSoCal.anydns.org'
option endpoint_port 'xxxxx'
list allowed_ips '0.0.0.0/0'
list allowed_ips '128.0.0.0/1'
option private_key 'xxxxxxxxx='
config wireguard_WG_SoB_Server
option description 'SoCal_Router'
option public_key 'xxxxxxxxxx='
option private_key 'xxxxxxxxxx='
option route_allowed_ips '1'
list allowed_ips '10.4.0.6/32'
option preshared_key 'xxxxxxxxxx='
option persistent_keepalive '25'
root@SoB_OpenWrt:/etc/config#
I've tested this with the WG interfaces in the WAN zone and a separate Wireguard zone. Same problem.
root@SoB_OpenWrt:/etc/config# cat firewall
config defaults
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'
config zone
option name 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
list network 'lan'
config zone
option name 'wan'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
list network 'wan'
list network 'WG_EvilCorp'
list network 'WG_SoCal_Pi5'
list network 'WG_SoCal_Client'
list network 'WG_WA_Pi_Client'
config forwarding
option src 'lan'
option dest 'wan'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config include 'pbr'
option fw4_compatible '1'
option type 'script'
option path '/usr/share/pbr/pbr.firewall.include'
config redirect
option dest 'Wireguard'
option target 'DNAT'
option name 'WG_SoB_Server'
list proto 'udp'
option src 'wan'
option src_dport 'xxxxx'
option dest_ip '10.1.0.1'
option dest_port 'xxxxx'
config zone
option name 'Wireguard'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
option mtu_fix '1'
list network 'WG_SoB_Server'
config forwarding
option src 'lan'
option dest 'Wireguard'
config forwarding
option src 'Wireguard'
option dest 'lan'
root@SoB_OpenWrt:/etc/config#
...I spent a few hours looking for the problem and broadened my search. Data outside the Wireguard tunnel is only used for some devices and upon closer look this doesn't look like this is just a Wireguard problem. Looking at the log the firewall and PBR restart again and again adding 6 minutes to the restart time before traffic starts flowing normally. The log is below.
Is there anyway to speed this process up, or configure it so the PBR and the firewall only start once?
...
Tue Aug 13 12:34:28 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:34:29 2024 daemon.info dnsmasq-dhcp[1]: DHCPDISCOVER(br-lan) xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:29 2024 daemon.info dnsmasq-dhcp[1]: DHCPOFFER(br-lan) 10.1.0.54 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:29 2024 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.1.0.54 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:29 2024 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.1.0.54 xx:xx:xx:xx:xx:xx Zigbee
Tue Aug 13 12:34:35 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:34:38 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:34:38 2024 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.1.0.220 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:38 2024 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.1.0.220 xx:xx:xx:xx:xx:xx FrontCam
Tue Aug 13 12:34:39 2024 user.notice firewall: Reloading firewall due to ifup of WG_SoB_Server (WG_SoB_Server)
Tue Aug 13 12:34:39 2024 daemon.info dnsmasq-dhcp[1]: DHCPDISCOVER(br-lan) xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:39 2024 daemon.info dnsmasq-dhcp[1]: DHCPOFFER(br-lan) 10.1.0.54 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:39 2024 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.1.0.54 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:34:39 2024 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.1.0.54 xx:xx:xx:xx:xx:xx Zigbee
Tue Aug 13 12:34:42 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:34:44 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:34:46 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:34:48 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:34:50 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:34:52 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:34:56 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:34:57 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:34:57 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:34:58 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:34:58 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:35:01 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:35:04 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:35:06 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:35:09 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:35:11 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:35:13 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:35:15 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:35:17 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:35:21 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:35:22 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:35:23 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:35:23 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:35:24 2024 user.notice pbr: Reloading pbr WG_SoB_Server interface routing due to ifup of WG_SoB_Server (WG_SoB_Server)
Tue Aug 13 12:35:27 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:35:30 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:35:32 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:35:34 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:35:36 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:35:38 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:35:39 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:35:41 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:35:45 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:35:47 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:35:47 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:35:48 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:35:51 2024 user.notice firewall: Reloading firewall due to ifup of WG_EvilCorp (WG_EvilCorp)
Tue Aug 13 12:35:54 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:35:57 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:36:00 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:36:02 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:36:04 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:36:05 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:36:07 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:36:09 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:36:11 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:36:15 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:36:16 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:36:17 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:36:17 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:36:18 2024 user.notice pbr: Reloading pbr WG_EvilCorp interface routing due to ifup of WG_EvilCorp (WG_EvilCorp)
Tue Aug 13 12:36:21 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:36:24 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:36:26 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:36:28 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:36:30 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:36:32 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:36:33 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:36:35 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:36:39 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:36:41 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:36:41 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:36:42 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:36:43 2024 user.notice firewall: Reloading firewall due to ifup of wan (eth0.2)
Tue Aug 13 12:36:45 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:36:48 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:36:51 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:36:53 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:36:55 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:36:57 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:36:58 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:37:00 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:37:02 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:37:06 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:37:09 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:37:09 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:37:10 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:37:10 2024 user.notice pbr: Reloading pbr wan interface routing due to ifup of wan (eth0.2)
Tue Aug 13 12:37:13 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:37:16 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:37:18 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:37:20 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:37:22 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:37:24 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:37:25 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:37:27 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:37:31 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:37:33 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:37:33 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:37:34 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:37:37 2024 user.notice firewall: Reloading firewall due to ifup of WG_WA_Pi_Client (WG_WA_Pi_Client)
Tue Aug 13 12:37:37 2024 user.notice ddns-scripts[1866]: duckdns: PID '1866' started at 2024-08-13 12:37
Tue Aug 13 12:37:41 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:37:45 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:37:48 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:37:49 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:37:51 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:37:53 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:37:55 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:37:57 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:37:59 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:38:03 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:38:04 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:38:04 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:38:05 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:38:06 2024 user.notice pbr: Reloading pbr WG_WA_Pi_Client interface routing due to ifup of WG_WA_Pi_Client (WG_WA_Pi_Client)
Tue Aug 13 12:38:12 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:38:16 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:38:19 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:38:21 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:38:23 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:38:25 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:38:27 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:38:28 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:38:33 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:38:34 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:38:34 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:38:35 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:38:37 2024 user.notice firewall: Reloading firewall due to ifup of WG_SoCal_Pi5 (WG_SoCal_Pi5)
Tue Aug 13 12:38:38 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:38:41 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:38:46 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:38:48 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:38:50 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:38:52 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:38:55 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:38:57 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:38:58 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:39:03 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:39:04 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:39:05 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:39:05 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:39:06 2024 user.notice pbr: Reloading pbr WG_SoCal_Pi5 interface routing due to ifup of WG_SoCal_Pi5 (WG_SoCal_Pi5)
Tue Aug 13 12:39:09 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:39:12 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:39:14 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:39:17 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:39:19 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:39:21 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:39:22 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:39:24 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:39:28 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:39:30 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:39:30 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:39:31 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:39:32 2024 user.notice firewall: Reloading firewall due to ifup of WG_SoCal_Client (WG_SoCal_Client)
Tue Aug 13 12:39:34 2024 user.notice pbr: Reloading pbr due to firewall action: includes
Tue Aug 13 12:39:37 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:39:39 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:39:41 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:39:43 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:39:45 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:39:48 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:39:51 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:39:52 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:39:57 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:39:58 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:39:58 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:39:59 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:40:00 2024 user.notice pbr: Reloading pbr WG_SoCal_Client interface routing due to ifup of WG_SoCal_Client (WG_SoCal_Client)
Tue Aug 13 12:40:03 2024 user.notice pbr: Activating traffic killswitch [✓]
Tue Aug 13 12:40:05 2024 daemon.info hostapd: phy1-ap0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: authenticated
Tue Aug 13 12:40:05 2024 user.notice pbr: Setting up routing for 'wan/eth0.2/192.168.2.254' [✓]
Tue Aug 13 12:40:06 2024 daemon.info hostapd: phy1-ap0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: associated (aid 3)
Tue Aug 13 12:40:06 2024 daemon.notice hostapd: phy1-ap0: AP-STA-CONNECTED xx:xx:xx:xx:xx:xx auth_alg=sae
Tue Aug 13 12:40:06 2024 daemon.info hostapd: phy1-ap0: STA xx:xx:xx:xx:xx:xx WPA: pairwise key handshake completed (RSN)
Tue Aug 13 12:40:06 2024 daemon.notice hostapd: phy1-ap0: EAPOL-4WAY-HS-COMPLETED xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:06 2024 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.1.0.74 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:06 2024 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.1.0.74 xx:xx:xx:xx:xx:xx User1_OP9
Tue Aug 13 12:40:07 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Pi5/10.6.0.9' [✓]
Tue Aug 13 12:40:09 2024 user.notice pbr: Setting up routing for 'WG_EvilCorp/10.237.131.6' [✓]
Tue Aug 13 12:40:11 2024 user.notice pbr: Setting up routing for 'WG_SoB_Server/10.4.0.1' [✓]
Tue Aug 13 12:40:13 2024 daemon.notice hostapd: phy1-ap0: AP-STA-DISCONNECTED xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:13 2024 daemon.info hostapd: phy1-ap0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: disassociated
Tue Aug 13 12:40:13 2024 user.notice pbr: Setting up routing for 'WG_SoCal_Client/10.17.0.2' [✓]
Tue Aug 13 12:40:14 2024 daemon.info hostapd: phy1-ap0: STA xx:xx:xx:xx:xx:xx IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
Tue Aug 13 12:40:15 2024 user.notice pbr: Setting up routing for 'WG_WA_Pi_Client/10.104.121.16' [✓]
Tue Aug 13 12:40:17 2024 user.notice pbr: Routing 'WG_EvilCorp' via WG_EvilCorp [✓]
Tue Aug 13 12:40:19 2024 daemon.info dnsmasq-dhcp[1]: DHCPDISCOVER(br-lan) xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:19 2024 daemon.info dnsmasq-dhcp[1]: DHCPOFFER(br-lan) 10.1.0.74 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:19 2024 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.1.0.74 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:19 2024 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.1.0.74 xx:xx:xx:xx:xx:xx User1_OP9
Tue Aug 13 12:40:21 2024 user.notice pbr: Routing 'WG_SoCal_Router' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:40:23 2024 user.notice pbr: Routing 'WG_WA_Pi' via WG_WA_Pi_Client [✓]
Tue Aug 13 12:40:23 2024 user.notice pbr: Deactivating traffic killswitch [✓]
Tue Aug 13 12:40:24 2024 user.notice pbr: service monitoring interfaces: wan WG_SoCal_Pi5 WG_SoCal_Client WG_EvilCorp WG_SoB_Server WG_WA_Pi_Client
Tue Aug 13 12:40:25 2024 daemon.info dnsmasq-dhcp[1]: DHCPDISCOVER(br-lan) xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:25 2024 daemon.info dnsmasq-dhcp[1]: DHCPOFFER(br-lan) 10.1.0.176 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:25 2024 daemon.info dnsmasq-dhcp[1]: DHCPREQUEST(br-lan) 10.1.0.176 xx:xx:xx:xx:xx:xx
Tue Aug 13 12:40:25 2024 daemon.info dnsmasq-dhcp[1]: DHCPACK(br-lan) 10.1.0.176 xx:xx:xx:xx:xx:xx Vacuum
...
Install the pbr 1.1.6-20 version:
https://dev.melmac.net/repo/
And report there: https://forum.openwrt.org/t/policy-based-routing-pbr-package-discussion/
Thanks. Will give that a try.
It seems to take a long time to bring up those 6 or 8 different interfaces. I wonder if the poor little single core MIPS CPU is just being clobbered.
Was using the same router and interfaces with DD-WRT and had no problem with long delays after rebooting, but maybe OpenWRT requires more processing power. Will restart later and see what happens to the CPU load.
Another thing to watch out for is if you have specified DNS servers that are run by a VPN provider and thus only reachable through certain VPN tunnels, you will get DNS failures and retries that can delay the bringing up of other VPN tunnels where the peer is specified by name rather than IP.
A common misconception is that list dns somehow binds attempts to reach that DNS server to a particular interface. It does not. Every list dns goes into the same single list which is consulted in a round-robin / failover fashion, and the standard routing table or PBR is used to determine the route. It can be simpler to only list one or two public DNS servers which are always reachable by regular WAN or any tunnel.
I ran into something similar while using dd-wrt when trying to use PiHole at the other end of a tunnel, but don't think that's the problem here. The tunnels connect fine (according to the Interface status) but PBR and the firewall repeatedly restart and interfere with traffic flow. If I'm reading the log correctly that's the cause of the long delays.
I'm pinging various IP addresses directly and intermittently get the "Destination Port Unreachable" error until the PBR/Firewall restarts complete.
Not an issue with CPU capacity. It loafs along at under 3% utilization during the entire time it takes for the system to stabilize.