Inconsistent connectivity – Wireguard client configuration

Help, I have a working wireguard client configuration but am struggling with reliability in the face of network restarts and loss of wan connectivity.

Goal:

  • By default, route traffic through warp tunnel
  • Use DoT (Stubby) for secure DNS to 1.1.1.1
  • Use vpn-policy-routing to send guest subnet and selected internal clients (set-top boxes) out wan
  • Handle wan coming up and down (flaky ISP here)

I have a working config that does not meet the last goal, and which has the following major problem:
Inconsistency of response to ‘service network restart’ and reboots. It’s a 50/50 proposition (or worse) the config delivers connectivity.
Sometimes, an ifup wan will get it back, at others it won’t.

To simplify debugging, I have disabled VPN-Policy-routing and only have stubby, sqm and wireguard (connecting to Cloudflare warp) enabled.

The logs look clean, with slight variations in order, but pretty much the same log entries for the interfaces and services as they come up.
The outputs of ‘ip address show; ip route show table all’ are identical (except sequence #s’) in either case.
Firewall outputs show the same rule sets.
The ‘wg’ command consistently shows the tunnel is up, but we just are not passing data.

I’ve been reading every thread on wireguard clients + openwrt, and even deployed a hotplug to /iface to ensure the wireguard interface is restarted/stopped in sync with the wan.

Here’s the hotplug code for /iface/50-warp:

#!/bin/sh
# skip if not enabled
if [ $(uci -q get network.warp.disabled) -eq 1 ] ; then exit 0; fi

if [ "$INTERFACE" = wan ] ; then
        [ "${ACTION}" = ifdown ] && ubus call network.interface.warp down
        if [ "${ACTION}" = "ifup" -o "${ACTION}" = "ifupdate" ]; then
                logger -t warp "performing an ifup on warp due to ${ACTION} of wan"
                ifup warp
        fi
        exit 0
fi

But even without that hotplug, the results are about the same, it seems to help, but clearly is not a fix.
It would seem there is a race condition somewhere, but I can’t spot it.

Anything in these configs and diagnostic outputs that would give us a clue as to what to try next?

Platform is a x86 (i5) with 4 intel NIC’s eth0 is wan, eth1 & eth2 are br-lan, eth2.3 & eth3 are br-guest.
OpenWRT is 19.07.8

Here is the output requested for troubleshooting wireguard, these are while in the non-working state:

pgrep -f -a wg; wg show; wg showconf warp
19545 wg-crypt-warp
interface: warp
  public key: 9Cfa37HmktBBSyDjjsrFIkcTW/yyzAodPHBnwNx34mc=
  private key: (hidden)
  listening port: 57218

peer: bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo=
  endpoint: 162.159.192.1:2408
  allowed ips: 0.0.0.0/0, ::/0
  latest handshake: 15 seconds ago
  transfer: 276 B received, 96.23 KiB sent
  persistent keepalive: every 24 seconds
[Interface]
ListenPort = 57218
PrivateKey = [redacted PrivateKey]

[Peer]
PublicKey = bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = 162.159.192.1:2408
PersistentKeepalive = 24
ip address show; ip route show table all
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether de:37:f5:24:bb:a1 brd ff:ff:ff:ff:ff:ff
3: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether 2a:af:39:06:a1:8e brd ff:ff:ff:ff:ff:ff
4: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc cake state UP group default qlen 1000
    link/ether 40:62:31:15:01:c4 brd ff:ff:ff:ff:ff:ff
    inet 75.117.160.x/22 brd 75.117.163.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::4262:31ff:fe15:1c4/64 scope link 
       valid_lft forever preferred_lft forever
5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP group default qlen 1000
    link/ether 40:62:31:15:01:c5 brd ff:ff:ff:ff:ff:ff
6: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master br-lan state DOWN group default qlen 1000
    link/ether 40:62:31:15:01:c6 brd ff:ff:ff:ff:ff:ff
7: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-guest state UP group default qlen 1000
    link/ether 40:62:31:15:01:c7 brd ff:ff:ff:ff:ff:ff
8: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
9: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
83: br-guest: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 40:62:31:15:01:c7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.1/24 brd 192.168.3.255 scope global br-guest
       valid_lft forever preferred_lft forever
    inet6 fe80::4262:31ff:fe15:1c7/64 scope link 
       valid_lft forever preferred_lft forever
84: eth2.3@eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue master br-guest state LOWERLAYERDOWN group default qlen 1000
    link/ether 40:62:31:15:01:c6 brd ff:ff:ff:ff:ff:ff
85: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 40:62:31:15:01:c5 brd ff:ff:ff:ff:ff:ff
    inet 192.168.7.1/24 brd 192.168.7.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 fda0:a0eb:3c56::1/60 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::4262:31ff:fe15:1c5/64 scope link 
       valid_lft forever preferred_lft forever
86: warp: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc cake state UNKNOWN group default qlen 1000
    link/none 
    inet 172.16.0.2/32 brd 255.255.255.255 scope global warp
       valid_lft forever preferred_lft forever
    inet6 fd01:5ca1:ab1e:8353:2283:e81c:x:x/128 scope global 
       valid_lft forever preferred_lft forever
89: ifb4eth0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc cake state UNKNOWN group default qlen 32
    link/ether 3e:9f:0d:ae:56:63 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::3c9f:dff:feae:5663/64 scope link 
       valid_lft forever preferred_lft forever
93: ifb4warp: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 32
    link/ether e6:d1:ec:84:8b:e9 brd ff:ff:ff:ff:ff:ff
default dev warp proto static scope link 
75.117.160.0/22 dev eth0 proto kernel scope link src 75.117.160.x 
162.159.192.1 via 75.117.160.1 dev eth0 proto static 
192.168.3.0/24 dev br-guest proto kernel scope link src 192.168.3.1 
192.168.7.0/24 dev br-lan proto kernel scope link src 192.168.7.1 
broadcast 75.117.160.0 dev eth0 table local proto kernel scope link src 75.117.160.x 
local 75.117.160.x dev eth0 table local proto kernel scope host src 75.117.160.x 
broadcast 75.117.163.255 dev eth0 table local proto kernel scope link src 75.117.160.x 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
local 172.16.0.2 dev warp table local proto kernel scope host src 172.16.0.2 
broadcast 192.168.3.0 dev br-guest table local proto kernel scope link src 192.168.3.1 
local 192.168.3.1 dev br-guest table local proto kernel scope host src 192.168.3.1 
broadcast 192.168.3.255 dev br-guest table local proto kernel scope link src 192.168.3.1 
broadcast 192.168.7.0 dev br-lan table local proto kernel scope link src 192.168.7.1 
local 192.168.7.1 dev br-lan table local proto kernel scope host src 192.168.7.1 
broadcast 192.168.7.255 dev br-lan table local proto kernel scope link src 192.168.7.1 
fd01:5ca1:ab1e:8353:2283:e81c:1490:2238 dev warp proto kernel metric 256 pref medium
fda0:a0eb:3c56::/64 dev br-lan proto static metric 1024 pref medium
unreachable fda0:a0eb:3c56::/48 dev lo proto static metric 2147483647 error 4294967183 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev br-guest proto kernel metric 256 pref medium
fe80::/64 dev ifb4eth0 proto kernel metric 256 pref medium
default dev warp proto static metric 1024 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fd01:5ca1:ab1e:8353:2283:e81c:1490:2238 dev warp table local proto kernel metric 0 pref medium
anycast fda0:a0eb:3c56:: dev br-lan table local proto kernel metric 0 pref medium
local fda0:a0eb:3c56::1 dev br-lan table local proto kernel metric 0 pref medium
anycast fe80:: dev ifb4eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev br-guest table local proto kernel metric 0 pref medium
anycast fe80:: dev br-lan table local proto kernel metric 0 pref medium
local fe80::3c9f:dff:feae:5663 dev ifb4eth0 table local proto kernel metric 0 pref medium
local fe80::4262:31ff:fe15:1c4 dev eth0 table local proto kernel metric 0 pref medium
local fe80::4262:31ff:fe15:1c5 dev br-lan table local proto kernel metric 0 pref medium
local fe80::4262:31ff:fe15:1c7 dev br-guest table local proto kernel metric 0 pref medium
ff00::/8 dev br-lan table local proto kernel metric 256 pref medium
ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
ff00::/8 dev br-guest table local proto kernel metric 256 pref medium
ff00::/8 dev ifb4eth0 table local proto kernel metric 256 pref medium
ff00::/8 dev warp table local proto kernel metric 256 pref medium
ip rule show; iptables-save -c
0:	from all lookup local 
32766:	from all lookup main 
32767:	from all lookup default 
# Generated by iptables-save v1.8.3 on Mon Dec 13 12:14:12 2021
*nat
:PREROUTING ACCEPT [334:131744]
:INPUT ACCEPT [23:1652]
:OUTPUT ACCEPT [43:2944]
:POSTROUTING ACCEPT [5:321]
:MINIUPNPD - [0:0]
:MINIUPNPD-POSTROUTING - [0:0]
:postrouting_guest_rule - [0:0]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_wan_rule - [0:0]
:postrouting_warpvpn_rule - [0:0]
:prerouting_guest_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_wan_rule - [0:0]
:prerouting_warpvpn_rule - [0:0]
:zone_guest_postrouting - [0:0]
:zone_guest_prerouting - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
:zone_warpvpn_postrouting - [0:0]
:zone_warpvpn_prerouting - [0:0]
[335:131806] -A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
[78:16359] -A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
[257:115447] -A PREROUTING -i eth0 -m comment --comment "!fw3" -j zone_wan_prerouting
[0:0] -A PREROUTING -i br-guest -m comment --comment "!fw3" -j zone_guest_prerouting
[0:0] -A PREROUTING -i warp -m comment --comment "!fw3" -j zone_warpvpn_prerouting
[198:11395] -A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
[0:0] -A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
[0:0] -A POSTROUTING -o eth0 -m comment --comment "!fw3" -j zone_wan_postrouting
[0:0] -A POSTROUTING -o br-guest -m comment --comment "!fw3" -j zone_guest_postrouting
[193:11074] -A POSTROUTING -o warp -m comment --comment "!fw3" -j zone_warpvpn_postrouting
[0:0] -A zone_guest_postrouting -m comment --comment "!fw3: Custom guest postrouting rule chain" -j postrouting_guest_rule
[0:0] -A zone_guest_prerouting -m comment --comment "!fw3: Custom guest prerouting rule chain" -j prerouting_guest_rule
[0:0] -A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
[78:16359] -A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
[0:0] -A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
[0:0] -A zone_wan_postrouting -j MINIUPNPD-POSTROUTING
[0:0] -A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
[257:115447] -A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
[257:115447] -A zone_wan_prerouting -j MINIUPNPD
[193:11074] -A zone_warpvpn_postrouting -m comment --comment "!fw3: Custom warpvpn postrouting rule chain" -j postrouting_warpvpn_rule
[193:11074] -A zone_warpvpn_postrouting -m comment --comment "!fw3" -j MASQUERADE
[0:0] -A zone_warpvpn_prerouting -m comment --comment "!fw3: Custom warpvpn prerouting rule chain" -j prerouting_warpvpn_rule
COMMIT
# Completed on Mon Dec 13 12:14:12 2021
# Generated by iptables-save v1.8.3 on Mon Dec 13 12:14:12 2021
*raw
:PREROUTING ACCEPT [1403:283799]
:OUTPUT ACCEPT [2245:342238]
:zone_guest_helper - [0:0]
:zone_lan_helper - [0:0]
[1061:159251] -A PREROUTING -i br-lan -m comment --comment "!fw3: lan CT helper assignment" -j zone_lan_helper
[0:0] -A PREROUTING -i br-guest -m comment --comment "!fw3: guest CT helper assignment" -j zone_guest_helper
[0:0] -A OUTPUT -p tcp -m tcp --dport 1723 -j CT --helper pptp
[0:0] -A zone_guest_helper -p udp -m comment --comment "!fw3: Amanda backup and archiving proto" -m udp --dport 10080 -j CT --helper amanda
[0:0] -A zone_guest_helper -p udp -m comment --comment "!fw3: RAS proto tracking" -m udp --dport 1719 -j CT --helper RAS
[0:0] -A zone_guest_helper -p tcp -m comment --comment "!fw3: Q.931 proto tracking" -m tcp --dport 1720 -j CT --helper Q.931
[0:0] -A zone_guest_helper -p tcp -m comment --comment "!fw3: IRC DCC connection tracking" -m tcp --dport 6667 -j CT --helper irc
[0:0] -A zone_guest_helper -p tcp -m comment --comment "!fw3: PPTP VPN connection tracking" -m tcp --dport 1723 -j CT --helper pptp
[0:0] -A zone_guest_helper -p tcp -m comment --comment "!fw3: SIP VoIP connection tracking" -m tcp --dport 5060 -j CT --helper sip
[0:0] -A zone_guest_helper -p udp -m comment --comment "!fw3: SIP VoIP connection tracking" -m udp --dport 5060 -j CT --helper sip
[0:0] -A zone_guest_helper -p udp -m comment --comment "!fw3: SNMP monitoring connection tracking" -m udp --dport 161 -j CT --helper snmp
[0:0] -A zone_guest_helper -p udp -m comment --comment "!fw3: TFTP connection tracking" -m udp --dport 69 -j CT --helper tftp
[0:0] -A zone_lan_helper -p udp -m comment --comment "!fw3: Amanda backup and archiving proto" -m udp --dport 10080 -j CT --helper amanda
[0:0] -A zone_lan_helper -p udp -m comment --comment "!fw3: RAS proto tracking" -m udp --dport 1719 -j CT --helper RAS
[0:0] -A zone_lan_helper -p tcp -m comment --comment "!fw3: Q.931 proto tracking" -m tcp --dport 1720 -j CT --helper Q.931
[0:0] -A zone_lan_helper -p tcp -m comment --comment "!fw3: IRC DCC connection tracking" -m tcp --dport 6667 -j CT --helper irc
[0:0] -A zone_lan_helper -p tcp -m comment --comment "!fw3: PPTP VPN connection tracking" -m tcp --dport 1723 -j CT --helper pptp
[0:0] -A zone_lan_helper -p tcp -m comment --comment "!fw3: SIP VoIP connection tracking" -m tcp --dport 5060 -j CT --helper sip
[0:0] -A zone_lan_helper -p udp -m comment --comment "!fw3: SIP VoIP connection tracking" -m udp --dport 5060 -j CT --helper sip
[0:0] -A zone_lan_helper -p udp -m comment --comment "!fw3: SNMP monitoring connection tracking" -m udp --dport 161 -j CT --helper snmp
[0:0] -A zone_lan_helper -p udp -m comment --comment "!fw3: TFTP connection tracking" -m udp --dport 69 -j CT --helper tftp
COMMIT
# Completed on Mon Dec 13 12:14:12 2021
# Generated by iptables-save v1.8.3 on Mon Dec 13 12:14:12 2021
*mangle
:PREROUTING ACCEPT [1413:284383]
:INPUT ACCEPT [767:163068]
:FORWARD ACCEPT [641:109429]
:OUTPUT ACCEPT [2253:345566]
:POSTROUTING ACCEPT [2771:448707]
[0:0] -A FORWARD -o br-guest -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone guest MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
[0:0] -A FORWARD -i br-guest -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone guest MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Mon Dec 13 12:14:12 2021
# Generated by iptables-save v1.8.3 on Mon Dec 13 12:14:12 2021
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:MINIUPNPD - [0:0]
:forwarding_guest_rule - [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_wan_rule - [0:0]
:forwarding_warpvpn_rule - [0:0]
:input_guest_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_wan_rule - [0:0]
:input_warpvpn_rule - [0:0]
:output_guest_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_wan_rule - [0:0]
:output_warpvpn_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_guest_dest_ACCEPT - [0:0]
:zone_guest_dest_REJECT - [0:0]
:zone_guest_forward - [0:0]
:zone_guest_input - [0:0]
:zone_guest_output - [0:0]
:zone_guest_src_REJECT - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
:zone_warpvpn_dest_ACCEPT - [0:0]
:zone_warpvpn_dest_REJECT - [0:0]
:zone_warpvpn_forward - [0:0]
:zone_warpvpn_input - [0:0]
:zone_warpvpn_output - [0:0]
:zone_warpvpn_src_REJECT - [0:0]
[20:1284] -A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
[750:161940] -A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
[301:23736] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[10:404] -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
[148:18491] -A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
[301:119713] -A INPUT -i eth0 -m comment --comment "!fw3" -j zone_wan_input
[0:0] -A INPUT -i br-guest -m comment --comment "!fw3" -j zone_guest_input
[0:0] -A INPUT -i warp -m comment --comment "!fw3" -j zone_warpvpn_input
[641:109429] -A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
[37:4418] -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[604:105011] -A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
[0:0] -A FORWARD -i eth0 -m comment --comment "!fw3" -j zone_wan_forward
[0:0] -A FORWARD -i br-guest -m comment --comment "!fw3" -j zone_guest_forward
[0:0] -A FORWARD -i warp -m comment --comment "!fw3" -j zone_warpvpn_forward
[0:0] -A FORWARD -m comment --comment "!fw3" -j reject
[20:1284] -A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
[2238:345822] -A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
[1663:307292] -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[3:96] -A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
[1:40] -A OUTPUT -o eth0 -m comment --comment "!fw3" -j zone_wan_output
[1:40] -A OUTPUT -o br-guest -m comment --comment "!fw3" -j zone_guest_output
[570:38354] -A OUTPUT -o warp -m comment --comment "!fw3" -j zone_warpvpn_output
[146:68431] -A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
[153:51114] -A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
[10:404] -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 200/sec --limit-burst 500 -m comment --comment "!fw3" -j RETURN
[0:0] -A syn_flood -m comment --comment "!fw3" -j DROP
[1:40] -A zone_guest_dest_ACCEPT -o br-guest -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_guest_dest_REJECT -o br-guest -m comment --comment "!fw3" -j reject
[0:0] -A zone_guest_forward -m comment --comment "!fw3: Custom guest forwarding rule chain" -j forwarding_guest_rule
[0:0] -A zone_guest_forward -m comment --comment "!fw3: Zone guest to wan forwarding policy" -j zone_wan_dest_ACCEPT
[0:0] -A zone_guest_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_guest_forward -m comment --comment "!fw3" -j zone_guest_dest_REJECT
[0:0] -A zone_guest_input -m comment --comment "!fw3: Custom guest input rule chain" -j input_guest_rule
[0:0] -A zone_guest_input -m comment --comment "!fw3: GuestDNS" -j ACCEPT
[0:0] -A zone_guest_input -p udp -m udp --dport 67:68 -m comment --comment "!fw3: GuestDHCP" -j ACCEPT
[0:0] -A zone_guest_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[0:0] -A zone_guest_input -m comment --comment "!fw3" -j zone_guest_src_REJECT
[1:40] -A zone_guest_output -m comment --comment "!fw3: Custom guest output rule chain" -j output_guest_rule
[1:40] -A zone_guest_output -m comment --comment "!fw3" -j zone_guest_dest_ACCEPT
[0:0] -A zone_guest_src_REJECT -i br-guest -m comment --comment "!fw3" -j reject
[3:96] -A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
[604:105011] -A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
[604:105011] -A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
[604:105011] -A zone_lan_forward -m comment --comment "!fw3: Zone lan to warpvpn forwarding policy" -j zone_warpvpn_dest_ACCEPT
[0:0] -A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
[148:18491] -A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
[0:0] -A zone_lan_input -s 169.254.0.0/16 -p tcp -m comment --comment "!fw3: BlockLinkLocal" -j DROP
[0:0] -A zone_lan_input -s 169.254.0.0/16 -p udp -m comment --comment "!fw3: BlockLinkLocal" -j DROP
[0:0] -A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[148:18491] -A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
[3:96] -A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
[3:96] -A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
[148:18491] -A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_wan_dest_ACCEPT -o eth0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[1:40] -A zone_wan_dest_ACCEPT -o eth0 -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_wan_dest_REJECT -o eth0 -m comment --comment "!fw3" -j reject
[0:0] -A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
[0:0] -A zone_wan_forward -p esp -m comment --comment "!fw3: Allow-IPSec-ESP" -j zone_lan_dest_ACCEPT
[0:0] -A zone_wan_forward -p udp -m udp --dport 500 -m comment --comment "!fw3: Allow-ISAKMP" -j zone_lan_dest_ACCEPT
[0:0] -A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_wan_forward -j MINIUPNPD
[0:0] -A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
[301:119713] -A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
[0:0] -A zone_wan_input -p udp -m udp --dport 68 -m comment --comment "!fw3: Allow-DHCP-Renew" -j ACCEPT
[2:168] -A zone_wan_input -p icmp -m icmp --icmp-type 8 -m comment --comment "!fw3: Allow-Ping" -j ACCEPT
[0:0] -A zone_wan_input -p igmp -m comment --comment "!fw3: Allow-IGMP" -j ACCEPT
[0:0] -A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[299:119545] -A zone_wan_input -j MINIUPNPD
[299:119545] -A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
[1:40] -A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
[1:40] -A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
[299:119545] -A zone_wan_src_REJECT -i eth0 -m comment --comment "!fw3" -j reject
[126:6384] -A zone_warpvpn_dest_ACCEPT -o warp -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[1048:136981] -A zone_warpvpn_dest_ACCEPT -o warp -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_warpvpn_dest_REJECT -o warp -m comment --comment "!fw3" -j reject
[0:0] -A zone_warpvpn_forward -m comment --comment "!fw3: Custom warpvpn forwarding rule chain" -j forwarding_warpvpn_rule
[0:0] -A zone_warpvpn_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_warpvpn_forward -m comment --comment "!fw3" -j zone_warpvpn_dest_REJECT
[0:0] -A zone_warpvpn_input -m comment --comment "!fw3: Custom warpvpn input rule chain" -j input_warpvpn_rule
[0:0] -A zone_warpvpn_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[0:0] -A zone_warpvpn_input -m comment --comment "!fw3" -j zone_warpvpn_src_REJECT
[570:38354] -A zone_warpvpn_output -m comment --comment "!fw3: Custom warpvpn output rule chain" -j output_warpvpn_rule
[570:38354] -A zone_warpvpn_output -m comment --comment "!fw3" -j zone_warpvpn_dest_ACCEPT
[0:0] -A zone_warpvpn_src_REJECT -i warp -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Mon Dec 13 12:14:12 2021

uci show network

uci show network
network.loopback=interface
network.loopback.ifname='lo'
network.loopback.proto='static'
network.loopback.ipaddr='127.0.0.1'
network.loopback.netmask='255.0.0.0'
network.globals=globals
network.globals.ula_prefix='fda0:a0eb:3c56::/48'
network.lan=interface
network.lan.type='bridge'
network.lan.proto='static'
network.lan.netmask='255.255.255.0'
network.lan.ip6assign='60'
network.lan.mtu='1500'
network.lan.ipaddr='192.168.7.1'
network.lan.igmp_snooping='1'
network.lan.ifname='eth1 eth2'
network.lan.dns='1.1.1.1'
network.wan=interface
network.wan.proto='dhcp'
network.wan.broadcast='1'
network.wan.ifname='eth0'
network.wan.hostname='OpenWrt'
network.wan.release='1'
network.wan.clientid='014062311501C4'
network.wan.peerdns='0'
network.wan.dns='1.1.1.1'
network.wan6=interface
network.wan6.ifname='eth0'
network.wan6.proto='dhcpv6'
network.guest=interface
network.guest.proto='static'
network.guest.ipaddr='192.168.3.1'
network.guest.netmask='255.255.255.0'
network.guest.type='bridge'
network.guest.ifname='eth3 eth2.3'
network.guest.dns='1.1.1.1'
network.warp=interface
network.warp.proto='wireguard'
network.warp.mtu='1280'
network.warp.force='1'
network.warp.dns='1.1.1.1'
network.warp.private_key='[redacted]'
network.warp.disabled='0'
network.warp.addresses='172.16.0.2/32' 'fd01:5ca1:ab1e:8353:2283:e81c:x:x/128'
network.@wireguard_warp[0]=wireguard_warp
network.@wireguard_warp[0].public_key='bmXOC+F1FxEMF9dyiK2H5/1SUtzH0JuVo51h2wPfgyo='
network.@wireguard_warp[0].description='cfwarp'
network.@wireguard_warp[0].route_allowed_ips='1'
network.@wireguard_warp[0].endpoint_host='engage.cloudflareclient.com'
network.@wireguard_warp[0].endpoint_port='2408'
network.@wireguard_warp[0].persistent_keepalive='24'
network.@wireguard_warp[0].allowed_ips='0.0.0.0/0' '::/0'

uci show firewall

firewall.@defaults[0]=defaults
firewall.@defaults[0].syn_flood='1'
firewall.@defaults[0].input='ACCEPT'
firewall.@defaults[0].output='ACCEPT'
firewall.@defaults[0].forward='REJECT'
firewall.@defaults[0].synflood_rate='200/s'
firewall.@defaults[0].synflood_burst='500'
firewall.@defaults[0].tcp_ecn='1'
firewall.@zone[0]=zone
firewall.@zone[0].name='lan'
firewall.@zone[0].network='lan'
firewall.@zone[0].input='ACCEPT'
firewall.@zone[0].output='ACCEPT'
firewall.@zone[0].forward='ACCEPT'
firewall.@zone[1]=zone
firewall.@zone[1].name='wan'
firewall.@zone[1].network='wan' 'wan6'
firewall.@zone[1].input='REJECT'
firewall.@zone[1].output='ACCEPT'
firewall.@zone[1].forward='REJECT'
firewall.@zone[1].masq='1'
firewall.@forwarding[0]=forwarding
firewall.@forwarding[0].src='lan'
firewall.@forwarding[0].dest='wan'
firewall.@rule[0]=rule
firewall.@rule[0].name='Allow-DHCP-Renew'
firewall.@rule[0].src='wan'
firewall.@rule[0].proto='udp'
firewall.@rule[0].dest_port='68'
firewall.@rule[0].target='ACCEPT'
firewall.@rule[0].family='ipv4'
firewall.@rule[1]=rule
firewall.@rule[1].name='Allow-Ping'
firewall.@rule[1].src='wan'
firewall.@rule[1].proto='icmp'
firewall.@rule[1].icmp_type='echo-request'
firewall.@rule[1].family='ipv4'
firewall.@rule[1].target='ACCEPT'
firewall.@rule[2]=rule
firewall.@rule[2].name='Allow-IGMP'
firewall.@rule[2].src='wan'
firewall.@rule[2].proto='igmp'
firewall.@rule[2].family='ipv4'
firewall.@rule[2].target='ACCEPT'
firewall.@rule[3]=rule
firewall.@rule[3].name='Allow-DHCPv6'
firewall.@rule[3].src='wan'
firewall.@rule[3].proto='udp'
firewall.@rule[3].src_ip='fc00::/6'
firewall.@rule[3].dest_ip='fc00::/6'
firewall.@rule[3].dest_port='546'
firewall.@rule[3].family='ipv6'
firewall.@rule[3].target='ACCEPT'
firewall.@rule[4]=rule
firewall.@rule[4].name='Allow-MLD'
firewall.@rule[4].src='wan'
firewall.@rule[4].proto='icmp'
firewall.@rule[4].src_ip='fe80::/10'
firewall.@rule[4].icmp_type='130/0' '131/0' '132/0' '143/0'
firewall.@rule[4].family='ipv6'
firewall.@rule[4].target='ACCEPT'
firewall.@rule[5]=rule
firewall.@rule[5].name='Allow-ICMPv6-Input'
firewall.@rule[5].src='wan'
firewall.@rule[5].proto='icmp'
firewall.@rule[5].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type' 'router-solicitation' 'neighbour-solicitation' 'router-advertisement' 'neighbour-advertisement'
firewall.@rule[5].limit='1000/sec'
firewall.@rule[5].family='ipv6'
firewall.@rule[5].target='ACCEPT'
firewall.@rule[6]=rule
firewall.@rule[6].name='Allow-ICMPv6-Forward'
firewall.@rule[6].src='wan'
firewall.@rule[6].dest='*'
firewall.@rule[6].proto='icmp'
firewall.@rule[6].icmp_type='echo-request' 'echo-reply' 'destination-unreachable' 'packet-too-big' 'time-exceeded' 'bad-header' 'unknown-header-type'
firewall.@rule[6].limit='1000/sec'
firewall.@rule[6].family='ipv6'
firewall.@rule[6].target='ACCEPT'
firewall.@rule[7]=rule
firewall.@rule[7].name='Allow-IPSec-ESP'
firewall.@rule[7].src='wan'
firewall.@rule[7].dest='lan'
firewall.@rule[7].proto='esp'
firewall.@rule[7].target='ACCEPT'
firewall.@rule[8]=rule
firewall.@rule[8].name='Allow-ISAKMP'
firewall.@rule[8].src='wan'
firewall.@rule[8].dest='lan'
firewall.@rule[8].dest_port='500'
firewall.@rule[8].proto='udp'
firewall.@rule[8].target='ACCEPT'
firewall.@include[0]=include
firewall.@include[0].path='/etc/firewall.user'
firewall.@zone[2]=zone
firewall.@zone[2].name='guest'
firewall.@zone[2].forward='REJECT'
firewall.@zone[2].output='ACCEPT'
firewall.@zone[2].network='guest'
firewall.@zone[2].input='REJECT'
firewall.@zone[2].mtu_fix='1'
firewall.@forwarding[1]=forwarding
firewall.@forwarding[1].dest='wan'
firewall.@forwarding[1].src='guest'
firewall.@rule[9]=rule
firewall.@rule[9].target='ACCEPT'
firewall.@rule[9].src='guest'
firewall.@rule[9].name='GuestDNS'
firewall.@rule[9].proto='all'
firewall.@rule[9].dest_port='53'
firewall.@rule[10]=rule
firewall.@rule[10].enabled='1'
firewall.@rule[10].target='ACCEPT'
firewall.@rule[10].src='guest'
firewall.@rule[10].name='GuestDHCP'
firewall.@rule[10].proto='udp'
firewall.@rule[10].dest_port='67-68'
firewall.@rule[11]=rule
firewall.@rule[11].name='BlockLinkLocal'
firewall.@rule[11].proto='tcp udp'
firewall.@rule[11].src_ip='169.254.0.0/16'
firewall.@rule[11].target='DROP'
firewall.@rule[11].src='lan'
firewall.@zone[3]=zone
firewall.@zone[3].network='warp'
firewall.@zone[3].forward='REJECT'
firewall.@zone[3].name='warpvpn'
firewall.@zone[3].output='ACCEPT'
firewall.@zone[3].input='REJECT'
firewall.@zone[3].masq='1'
firewall.@forwarding[2]=forwarding
firewall.@forwarding[2].dest='warpvpn'
firewall.@forwarding[2].src='lan'
firewall.miniupnpd=include
firewall.miniupnpd.type='script'
firewall.miniupnpd.path='/usr/share/miniupnpd/firewall.include'
firewall.miniupnpd.family='any'
firewall.miniupnpd.reload='1'

note: while there is config for miniupnpd, the package is NOT enabled.

https://openwrt.org/docs/guide-user/services/vpn/wireguard/extras#dynamic_connection

Thanks @vgaetera, but that still does not seem to work reliably. in 6 or so tries, it fails after this sequence:

After uci set network.wan.metric="1024"
Network restart worked
another service network restart, worked
Reboot, worked
Pull modem cable, worked
Wait 5 minutes, service network restart, no service

update routes listing when in a no-service state

ip route show table all
default dev warp proto static scope link 
default via 75.117.160.1 dev eth0 proto static src 75.117.160.x metric 1024 
75.117.160.0/22 dev eth0 proto static scope link metric 1024 
162.159.192.1 via 75.117.160.1 dev eth0 proto static metric 1024 
192.168.3.0/24 dev br-guest proto kernel scope link src 192.168.3.1 
192.168.7.0/24 dev br-lan proto kernel scope link src 192.168.7.1 
broadcast 75.117.160.0 dev eth0 table local proto kernel scope link src 75.117.160.x 
local 75.117.160.x dev eth0 table local proto kernel scope host src 75.117.160.x 
broadcast 75.117.163.255 dev eth0 table local proto kernel scope link src 75.117.160.x 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
local 172.16.0.2 dev warp table local proto kernel scope host src 172.16.0.2 
broadcast 192.168.3.0 dev br-guest table local proto kernel scope link src 192.168.3.1 
local 192.168.3.1 dev br-guest table local proto kernel scope host src 192.168.3.1 
broadcast 192.168.3.255 dev br-guest table local proto kernel scope link src 192.168.3.1 
broadcast 192.168.7.0 dev br-lan table local proto kernel scope link src 192.168.7.1 
local 192.168.7.1 dev br-lan table local proto kernel scope host src 192.168.7.1 
broadcast 192.168.7.255 dev br-lan table local proto kernel scope link src 192.168.7.1 
fd01:5ca1:ab1e:8353:2283:e81c:1490:2238 dev warp proto kernel metric 256 pref medium
fda0:a0eb:3c56::/64 dev br-lan proto static metric 1024 pref medium
unreachable fda0:a0eb:3c56::/48 dev lo proto static metric 2147483647 error 4294967183 pref medium
fe80::/64 dev br-lan proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev br-guest proto kernel metric 256 pref medium
fe80::/64 dev ifb4eth0 proto kernel metric 256 pref medium
default dev warp proto static metric 1024 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fd01:5ca1:ab1e:8353:2283:e81c:1490:2238 dev warp table local proto kernel metric 0 pref medium
anycast fda0:a0eb:3c56:: dev br-lan table local proto kernel metric 0 pref medium
local fda0:a0eb:3c56::1 dev br-lan table local proto kernel metric 0 pref medium
anycast fe80:: dev ifb4eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium
anycast fe80:: dev br-guest table local proto kernel metric 0 pref medium
anycast fe80:: dev br-lan table local proto kernel metric 0 pref medium
local fe80::4262:31ff:fe15:1c4 dev eth0 table local proto kernel metric 0 pref medium
local fe80::4262:31ff:fe15:1c5 dev br-lan table local proto kernel metric 0 pref medium
local fe80::4262:31ff:fe15:1c7 dev br-guest table local proto kernel metric 0 pref medium
local fe80::bcee:5aff:fedd:1c56 dev ifb4eth0 table local proto kernel metric 0 pref medium
ff00::/8 dev br-lan table local proto kernel metric 256 pref medium
ff00::/8 dev eth0 table local proto kernel metric 256 pref medium
ff00::/8 dev br-guest table local proto kernel metric 256 pref medium
ff00::/8 dev ifb4eth0 table local proto kernel metric 256 pref medium
ff00::/8 dev warp table local proto kernel metric 256 pref medium

I was able to bring back connectivity with an ifup wan

Progress, but still need some help.

The metric might be an option for some, but for me, it made it even less reliable, so I removed it.

I think I found the root cause of the race condition: DNS Services are not available at times during network restarts or at boot time, so the configured endpoint is not resolvable.
This seems to cause wireguard to not come up cleanly when that happens.

But replacing this: network.@wireguard_warp[0].endpoint_host='engage.cloudflareclient.com'
with this: network.@wireguard_warp[0].endpoint_host='162.159.192.1'

I now get clean (and fast) results when restarting the network, when the modem cable is pulled, or after a reboot.

However, a hardcoded IP is not the best idea for the endpoint config, so now looking for an OpenWRT-native method of achieving what this thread talks about fixing this on desktop Linux: https://bbs.archlinux.org/viewtopic.php?id=266627

Basically, wait until the endpoint is resolved and reachable before allowing the WG tunnel to continue coming up.
So I'd welcome some input on how to achieve that in the most effective, natural way on OpenWRT.

Resolve the endpoint domain bypassing the VPN tunnel:
https://openwrt.org/docs/guide-user/base-system/dhcp_configuration#selective_dns_forwarding

Thanks, but that is not the problem, as the tunnel is not up when the wireguard protocol is trying to resolve the domain for the endpoint.

The race is with any DNS service being available at the point in time the WG tunnel is being instantiated.

The fix seems to be that we need to wait until we can confirm resolution (and access) to engage.cloudflareclient.com:2408 before the system even attempts to bring up wireguard.

As shown in the linked thread, on a Linux PC they add the means of waiting until that is confirmed using this pre-start function :

# /etc/systemd/system/wg-quick@.service.d/override.conf
[Service]
ExecStartPre=/bin/bash -c '{ exec {sleep}<> <(:); while ! : >/dev/udp/engage.cloudflareclient.com/2408; do read -t 1 -u $sleep; done } 2>/dev/null; exit 0'

The question remains, on OpenWRT, what is the best mechanism to achieve a similar delay until resolution and access are confirmed?

Since wireguard is a network protocol, not a userspace app, the solution needs to mesh well with the architecture of the network workflow.

As far as I know, OpenWrt doesn't provide this kind of delay method, unless you implement it with procd.
Race conditions related to netifd and dnsmasq are typically resolved with split routing and split DNS.

2 Likes

This seems to contradict with DNS explicitly set for interfaces in the /etc/config/network. But if I understood multiple posts correctly and the problem seems to be with the DNS resolution failing shortly after WAN restart, I'd experiment with removing reliance on stubby or trying to use dhcp.@dnsmasq[0].server='/engage.cloudflareclient.com/1.1.1.1' option to resolve engage.cloudflareclient.com without stubby.

1 Like