I'm losing my mind tonight, I've been staring at a screen for 7 hours trying to get this to work and I cannot fathom what is wrong.
The intent is to create a routed VPN for management of OpenWRT devices with OpenWISP. The OpenVPN server is running on a Debian 12 box, the clients are OpenWRT devices. The VPN connects ok, initializes the connection ok, the route tables populate ok, but no traffic will pass across the VPN. Ah-ha, you say, it must be a firewall, but no, firewall is disabled on OpenWRT and not configured on Debian. I'm at a loss.
As you can see, both sides of the tunnel (tun0) are sending traffic but the other side is not recieving it, despite an established VPN link. If I watch the local tun0 interface I can see the packets being sent to the correct vpn tun0 interface, but they don't appear on the other side at the tun0 interface on the other end?!?!??!?
root@GVH-AP-RR-2:/# tcpdump -i tun0
[ 8865.353500] device tun0 entered promiscuous mode
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tun0, link-type RAW (Raw IP), snapshot length 262144 bytes
21:00:02.876845 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2657, length 64
21:00:03.877040 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2658, length 64
21:00:04.877225 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2659, length 64
21:00:05.877394 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2660, length 64
21:00:06.877566 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2661, length 64
21:00:07.877739 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2662, length 64
21:00:08.877940 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2663, length 64
21:00:09.878135 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2664, length 64
21:00:10.878324 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2665, length 64
21:00:11.878498 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2666, length 64
21:00:12.878669 IP 10.8.0.2 > 10.8.0.1: ICMP echo request, id 4912, seq 2667, length 64
root@openwisp:/etc/openvpn# tcpdump -i tun0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tun0, link-type RAW (Raw IP), snapshot length 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
Please help!
Server OpenVPN Config
root@openwisp:/etc/openvpn# cat /etc/openvpn/server.conf
ca /etc/easyrsa/pki/ca.crt
cert /etc/easyrsa/pki/issued/server.crt
key /etc/easyrsa/pki/private/server.key # This file should be kept secret
dh /etc/easyrsa/pki/dh.pem
client-to-client
mute-replay-warnings
persist-key
persist-tun
tls-server
auth SHA512
cipher AES-256-GCM
dev tun0
dev-type tun
keepalive 10 120
mode server
port 11194
push "persist-key"
push "persist-tun"
push "topology subnet"
push "route 10.8.0.0 255.255.255.0"
server 10.8.0.0 255.255.255.0
status /var/log/openvpn-udp-status.log
topology subnet
verb 3
Client OpenVPN Config
root@GVH-AP-RR-2:/# cat /tmp/etc/openvpn-openwisp_vpn.conf
nobind
persist-key
persist-tun
pull
tls-client
auth SHA512
ca /etc/x509/ca-2-Easy-RSA_CA.pem
cert /etc/x509/client-98b5c0de6c764b9ab788aa83edf22e0b.pem
cipher AES-256-GCM
comp-lzo no
dev tun0
dev-type tun
fragment 0
key /etc/x509/key-98b5c0de6c764b9ab788aa83edf22e0b.pem
mssfix 1450
mute 0
proto udp
remote openwisp.abcd.net 11194
remote-cert-tls server
reneg-sec 3600
resolv-retry infinite
tls-timeout 2
verb 3
Client Connect Log (success)
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OpenVPN 2.5.8 arm-openwrt-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD]
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: library versions: OpenSSL 3.0.15 3 Sep 2024, LZO 2.10
Tue Oct 8 20:50:48 2024 daemon.warn openvpn(openwisp_vpn)[6304]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: TCP/UDP: Preserving recently used remote address: [AF_INET]192.168.1.77:11194
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: Socket Buffers: R=[196608->196608] S=[196608->196608]
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: UDP link local: (not bound)
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: UDP link remote: [AF_INET]192.168.1.77:11194
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: TLS: Initial packet from [AF_INET]192.168.1.77:11194, sid=429c1955 b5a5817e
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: VERIFY OK: depth=1, CN=Easy-RSA CA
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: VERIFY KU OK
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: Validating certificate extended key usage
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: VERIFY EKU OK
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: VERIFY OK: depth=0, CN=server
Tue Oct 8 20:50:48 2024 daemon.warn openvpn(openwisp_vpn)[6304]: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1550', remote='link-mtu 1549'
Tue Oct 8 20:50:48 2024 daemon.warn openvpn(openwisp_vpn)[6304]: WARNING: 'comp-lzo' is present in local config but missing in remote config, local='comp-lzo'
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, peer certificate: 2048 bit RSA, signature: RSA-SHA256
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: [server] Peer Connection Initiated with [AF_INET]192.168.1.77:11194
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: PUSH: Received control message: 'PUSH_REPLY,persist-key,persist-tun,topology subnet,route 10.8.0.0 255.255.255.0,route-gateway 10.8.0.1,topology subnet,ping 10,ping-restart 120,ifconfig 10.8.0.2 255.255.255.0,peer-id 0,cipher AES-256-GCM'
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: timers and/or timeouts modified
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: --persist options modified
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: --ifconfig/up options modified
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: route options modified
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: route-related options modified
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: peer-id set
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: adjusting link_mtu to 1625
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: OPTIONS IMPORT: data channel crypto options modified
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: net_route_v4_best_gw query: dst 0.0.0.0
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: net_route_v4_best_gw result: via 10.70.70.1 dev br-lan
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: TUN/TAP device tun0 opened
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: net_iface_mtu_set: mtu 1500 for tun0
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: net_iface_up: set tun0 up
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: net_addr_v4_add: 10.8.0.2/24 dev tun0
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: /usr/libexec/openvpn-hotplug up openwisp_vpn tun0 1500 1625 10.8.0.2 255.255.255.0 init
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: net_route_v4_add: 10.8.0.0/24 via 10.8.0.1 dev [NULL] table 0 metric -1
Tue Oct 8 20:50:48 2024 daemon.notice openvpn(openwisp_vpn)[6304]: Initialization Sequence Completed
Server Interface and Route Table
tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1500
inet 10.8.0.1 netmask 255.255.255.0 destination 10.8.0.1
inet6 fe80::a7fa:6f2f:272d:71b2 prefixlen 64 scopeid 0x20<link>
unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 18 bytes 1224 (1.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@openwisp:/etc/openvpn# ip route show
default via 192.168.1.1 dev ens3
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.1
192.168.1.0/24 dev ens3 proto kernel scope link src 192.168.1.77
Client Interface & Route Table
tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.8.0.2 P-t-P:10.8.0.2 Mask:255.255.255.0
inet6 addr: fe80::bb7e:d4a7:429b:385f/64 Scope:Link
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:231 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B) TX bytes:19372 (18.9 KiB)
default via 10.70.70.1 dev br-lan src 10.70.70.224
10.8.0.0/24 via 10.8.0.1 dev tun0
10.70.70.0/24 dev br-lan scope link src 10.70.70.224
Ping Server (10.8.0.1) From Client (fail)
root@GVH-AP-RR-2:/# ping 10.8.0.1
PING 10.8.0.1 (10.8.0.1): 56 data bytes
^C
--- 10.8.0.1 ping statistics ---
13 packets transmitted, 0 packets received, 100% packet loss
Ping Client (10.8.0.2) From Server (fail)
root@openwisp:/etc/openvpn# ping 10.8.0.2
PING 10.8.0.2 (10.8.0.2) 56(84) bytes of data.
^C
--- 10.8.0.2 ping statistics ---
15 packets transmitted, 0 received, 100% packet loss, time 14319ms