I've set up a route-based site-to-site tunnel between an OpenWrt router and an EdgeRouter X. I've finally gotten it to work, but with one major hitch: once the link is established, the firewall on the OpenWrt side must be restarted. (I use service firewall restart.) Otherwise, the OpenWrt router won't accept any packets over the tunnel from the EdgeRouter.
Both routers are connected directly to the internet and aren't behind NAT. In the logs posted later in this post, I'll be using 77.77.77.77 for the WAN IP of the OpenWrt router and 66.66.66.66 for the WAN IP of the EdgeRouter. 10.255.20.2/24 and 10.255.20.1/24 are the addresses of the VTIs of the OpenWrt router and the EdgeRouter, respectively.
The issue is one-way. Pinging a client PC running Wireshark on the EdgeRouter's LAN from the OpenWrt router results in Wireshark picking up the echo requests and the replies back. The dashboard of the EdgeRouter web UI also shows data flowing in both directions through interface vti0 while the ping is going. However, pinging anything on the OpenWrt LAN from the EdgeRouter LAN shows nothing but the requests, including the EdgeRouter only indicating packets being sent out interface vti0. This rules out anything on the EdgeRouter side as being the cause.
This wouldn't be too big of a deal if the firewall only needed to be restarted when the link is first established after booting OpenWrt, but sometimes IKE rekeying causes the tunnel to stop working in the same way, with OpenWrt not accepting packets from the tunnel and the firewall needing to be restarted to fix it.
Here's the output of logread on the OpenWrt router after running ipsec restart. (P01 is the name of the VPN connection.) I'm not sure what of this might not be relevant, so I've included everything after running the command:
Mon Jun 21 21:04:45 2021 daemon.info : 00[DMN] signal of type SIGINT received. Shutting down
Mon Jun 21 21:04:45 2021 daemon.info : 00[IKE] deleting IKE_SA P01[3] between 77.77.77.77[77.77.77.77]...66.66.66.66[66.66.66.66]
Mon Jun 21 21:04:45 2021 authpriv.info : 00[IKE] deleting IKE_SA P01[3] between 77.77.77.77[77.77.77.77]...66.66.66.66[66.66.66.66]
Mon Jun 21 21:04:45 2021 daemon.info : 00[IKE] sending DELETE for IKE_SA P01[3]
Mon Jun 21 21:04:45 2021 daemon.info : 00[ENC] generating INFORMATIONAL request 2 [ D ]
Mon Jun 21 21:04:45 2021 daemon.info : 00[NET] sending packet: from 77.77.77.77[4500] to 66.66.66.66[4500] (76 bytes)
Mon Jun 21 21:04:45 2021 daemon.info : 00[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:45 2021 daemon.info : 00[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:45 2021 daemon.info : 00[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:45 2021 daemon.info : 00[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:45 2021 authpriv.info ipsec_starter[29445]: charon stopped after 200 ms
Mon Jun 21 21:04:45 2021 authpriv.info ipsec_starter[29445]: ipsec starter stopped
Mon Jun 21 21:04:48 2021 authpriv.info ipsec_starter[30516]: Starting strongSwan 5.8.2 IPsec [starter]...
Mon Jun 21 21:04:48 2021 daemon.info : 00[DMN] Starting IKE charon daemon (strongSwan 5.8.2, Linux 4.14.221, mips)
Mon Jun 21 21:04:49 2021 daemon.info : 00[CFG] PKCS11 module '<name>' lacks library path
Mon Jun 21 21:04:53 2021 daemon.info : 00[LIB] curl SSL backend 'mbedTLS/2.16.10' not supported, https:// disabled
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] disabling load-tester plugin, not configured
Mon Jun 21 21:04:53 2021 daemon.info : 00[LIB] plugin 'load-tester': failed to load - load_tester_plugin_create returned NULL
Mon Jun 21 21:04:53 2021 daemon.info : 00[LIB] plugin 'uci' failed to load: Error relocating /usr/lib/ipsec/plugins/libstrongswan-uci.so: uci_lookup: symbol not found
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] attr-sql plugin: database URI not set
Mon Jun 21 21:04:53 2021 daemon.info : 00[NET] using forecast interface br-GUEST
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] joining forecast multicast groups: 224.0.0.1,224.0.0.22,224.0.0.251,224.0.0.252,239.255.255.250
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loading ca certificates from '/etc/ipsec.d/cacerts'
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loading aa certificates from '/etc/ipsec.d/aacerts'
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loading ocsp signer certificates from '/etc/ipsec.d/ocspcerts'
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loading attribute certificates from '/etc/ipsec.d/acerts'
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loading crls from '/etc/ipsec.d/crls'
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loading secrets from '/etc/ipsec.secrets'
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loaded IKE secret for 66.66.66.66 77.77.77.77
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] sql plugin: database URI not set
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] loaded 0 RADIUS server configurations
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] HA config misses local/remote address
Mon Jun 21 21:04:53 2021 daemon.info : 00[CFG] coupling file path unspecified
Mon Jun 21 21:04:53 2021 daemon.info : 00[LIB] loaded plugins: charon test-vectors ldap pkcs11 aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt af-alg fips-prf gmp curve25519 agent xcbc cmac hmac ctr ccm gcm curl mysql sqlite attr kernel-netlink resolve socket-default connmark forecast farp stroke vici smp updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap dhcp whitelist led duplicheck addrblock unity
Mon Jun 21 21:04:53 2021 daemon.info : 00[JOB] spawning 16 worker threads
Mon Jun 21 21:04:53 2021 authpriv.info ipsec_starter[30540]: charon (30541) started after 4640 ms
Mon Jun 21 21:04:53 2021 daemon.info : 06[CFG] received stroke: add connection 'P01'
Mon Jun 21 21:04:53 2021 daemon.info : 06[CFG] added configuration 'P01'
Mon Jun 21 21:04:53 2021 daemon.info : 08[CFG] received stroke: initiate 'P01'
Mon Jun 21 21:04:53 2021 daemon.info : 08[IKE] initiating IKE_SA P01[1] to 66.66.66.66
Mon Jun 21 21:04:53 2021 authpriv.info : 08[IKE] initiating IKE_SA P01[1] to 66.66.66.66
Mon Jun 21 21:04:53 2021 daemon.info : 08[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Mon Jun 21 21:04:53 2021 daemon.info : 08[NET] sending packet: from 77.77.77.77[500] to 66.66.66.66[500] (464 bytes)
Mon Jun 21 21:04:55 2021 daemon.info : 14[NET] received packet: from 66.66.66.66[500] to 77.77.77.77[500] (440 bytes)
Mon Jun 21 21:04:55 2021 daemon.info : 14[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(MULT_AUTH) ]
Mon Jun 21 21:04:55 2021 daemon.info : 14[CFG] selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
Mon Jun 21 21:04:55 2021 daemon.info : 14[IKE] authentication of '77.77.77.77' (myself) with pre-shared key
Mon Jun 21 21:04:55 2021 daemon.info : 14[IKE] establishing CHILD_SA P01{1}
Mon Jun 21 21:04:55 2021 authpriv.info : 14[IKE] establishing CHILD_SA P01{1}
Mon Jun 21 21:04:55 2021 daemon.info : 14[ENC] generating IKE_AUTH request 1 [ IDi N(INIT_CONTACT) IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(ADD_6_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Mon Jun 21 21:04:55 2021 daemon.info : 14[NET] sending packet: from 77.77.77.77[4500] to 66.66.66.66[4500] (364 bytes)
Mon Jun 21 21:04:55 2021 daemon.info : 07[NET] received packet: from 66.66.66.66[4500] to 77.77.77.77[4500] (252 bytes)
Mon Jun 21 21:04:55 2021 daemon.info : 07[ENC] parsed IKE_AUTH response 1 [ IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) ]
Mon Jun 21 21:04:55 2021 daemon.info : 07[IKE] authentication of '66.66.66.66' with pre-shared key successful
Mon Jun 21 21:04:55 2021 daemon.info : 07[IKE] IKE_SA P01[1] established between 77.77.77.77[77.77.77.77]...66.66.66.66[66.66.66.66]
Mon Jun 21 21:04:55 2021 authpriv.info : 07[IKE] IKE_SA P01[1] established between 77.77.77.77[77.77.77.77]...66.66.66.66[66.66.66.66]
Mon Jun 21 21:04:55 2021 daemon.info : 07[IKE] scheduling reauthentication in 9650s
Mon Jun 21 21:04:55 2021 daemon.info : 07[IKE] maximum IKE_SA lifetime 10250s
Mon Jun 21 21:04:55 2021 daemon.info : 07[CFG] selected proposal: ESP:AES_CBC_256/HMAC_SHA1_96/NO_EXT_SEQ
Mon Jun 21 21:04:55 2021 daemon.info : 07[IKE] CHILD_SA P01{1} established with SPIs cca59633_i ceaea8f0_o and TS 0.0.0.0/0 === 0.0.0.0/0
Mon Jun 21 21:04:55 2021 authpriv.info : 07[IKE] CHILD_SA P01{1} established with SPIs cca59633_i ceaea8f0_o and TS 0.0.0.0/0 === 0.0.0.0/0
Mon Jun 21 21:04:55 2021 daemon.info : 07[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:55 2021 daemon.info : 07[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:55 2021 daemon.info : 07[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:55 2021 daemon.info : 07[CHD] updown: RTNETLINK answers: File exists
Mon Jun 21 21:04:55 2021 daemon.info : 07[IKE] peer supports MOBIKE
Also, here's the output of ipsec statusall on a freshly-established link before restarting the firewall:
Status of IKE charon daemon (strongSwan 5.8.2, Linux 4.14.221, mips):
uptime: 5 seconds, since Jun 21 21:06:55 2021
worker threads: 10 of 16 idle, 6/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
loaded plugins: charon test-vectors ldap pkcs11 aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt af-alg fips-prf gmp curve25519 agent xcbc cmac hmac ctr ccm gcm curl mysql sqlite attr kernel-netlink resolve socket-default connmark forecast farp stroke vici smp updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap dhcp whitelist led duplicheck addrblock unity
Listening IP addresses:
10.21.0.1
10.20.0.1
[IPv6 address redacted]
[IPv6 address redacted]
77.77.77.77
[IPv6 address redacted]
10.255.20.2
Connections:
P01: 77.77.77.77...66.66.66.66 IKEv2
P01: local: [77.77.77.77] uses pre-shared key authentication
P01: remote: [66.66.66.66] uses pre-shared key authentication
P01: child: 0.0.0.0/0 === 0.0.0.0/0 TUNNEL
Security Associations (1 up, 0 connecting):
P01[1]: ESTABLISHED 3 seconds ago, 77.77.77.77[77.77.77.77]...66.66.66.66[66.66.66.66]
P01[1]: IKEv2 SPIs: 83dc49cc43212d25_i* f6da58189e5d831e_r, pre-shared key reauthentication in 2 hours
P01[1]: IKE proposal: AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
P01{1}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cdb42929_i cadd10d4_o
P01{1}: AES_CBC_256/HMAC_SHA1_96, 0 bytes_i, 0 bytes_o, rekeying in 40 minutes
P01{1}: 0.0.0.0/0 === 0.0.0.0/0
And now after restarting the firewall:
Status of IKE charon daemon (strongSwan 5.8.2, Linux 4.14.221, mips):
uptime: 43 seconds, since Jun 21 21:06:56 2021
worker threads: 10 of 16 idle, 6/0/0/0 working, job queue: 0/0/0/0, scheduled: 2
loaded plugins: charon test-vectors ldap pkcs11 aes des blowfish rc2 sha2 sha1 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt af-alg fips-prf gmp curve25519 agent xcbc cmac hmac ctr ccm gcm curl mysql sqlite attr kernel-netlink resolve socket-default connmark forecast farp stroke vici smp updown eap-identity eap-md5 eap-mschapv2 eap-radius eap-tls xauth-generic xauth-eap dhcp whitelist led duplicheck addrblock unity
Listening IP addresses:
10.21.0.1
10.20.0.1
[IPv6 address redacted]
[IPv6 address redacted]
77.77.77.77
[IPv6 address redacted]
10.255.20.2
Connections:
P01: 77.77.77.77...66.66.66.66 IKEv2
P01: local: [77.77.77.77] uses pre-shared key authentication
P01: remote: [66.66.66.66] uses pre-shared key authentication
P01: child: 0.0.0.0/0 === 0.0.0.0/0 TUNNEL
Security Associations (1 up, 0 connecting):
P01[1]: ESTABLISHED 41 seconds ago, 77.77.77.77[77.77.77.77]...66.66.66.66[66.66.66.66]
P01[1]: IKEv2 SPIs: 83dc49cc43212d25_i* f6da58189e5d831e_r, pre-shared key reauthentication in 2 hours
P01[1]: IKE proposal: AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
P01{1}: INSTALLED, TUNNEL, reqid 1, ESP SPIs: cdb42929_i cadd10d4_o
P01{1}: AES_CBC_256/HMAC_SHA1_96, 84 bytes_i (1 pkt, 6s ago), 84 bytes_o (1 pkt, 6s ago), rekeying in 39 minutes
P01{1}: 0.0.0.0/0 === 0.0.0.0/0
I don't notice anything meaningfully different in either result.
Is this just normal behavior, or do I likely have something configured wrong? I tried searching here and elsewhere, but found no one else with this same issue (though googling "openwrt ipsec stops working until firewall restarted" just turns up a ton of guides that tell you to restart the firewall after initially modifying it).