OWRT firewall block output UDP connection if number of received packets is greater than emitted ones

Hello !

I'm using OWRT on 3 Proxmox nodes, in VM.
On each Proxmox, the VMs use Wireguard to communicate. (The three nodes are not on the same place, so I need to encrypt the connection)

OWRT_1 is a VM on Proxmox 1, all traffic of Proxmox VMs go through there.
Node_1 is a VM on Proxmox 1.
OWRT_2 and Node_2 are on Proxmox 2.

On OWRT_1 I opened the Wireguard port via the forwarding rules, to Node_1. On OWRT_2, nothing to do, the VM on Node_2 initiate the connection to Node_1.

The Wireguard connection works very well, the OWRT_2 use a random output port to connect to the Node_1 in UDP, the VMs can ping each other.

BUT ! Sometimes, Node_1 send a big quantity of data to Node_2 via the Wireguard tunnel, and the wireguard connection is lost. After using conntrack on each OWRT, I can see the connection is lost when the received packets is greater than the emitted packets on OWRT_2.

From the OWRT_2, the connection is an output UDP connection, this is the conntrack line (more packets received than emitted, wireguard connection is lost on the VM) : udp 17 177 src=172.16.1.4 dst=xxx.xxx.xxx.xxx sport=51823 dport=43023 packets=32904 bytes=5445480 src=xxx.xxx.xxx.xxx dst=172.31.0.176 sport=43023 dport=51823 packets=35191 bytes=11946188 [ASSURED] mark=0 use=1

I've not found any documentation about this feature, how can I say to the firewall to not block the connection in this case ?

Thanks

Rather than crafting firewall rules, why not use a watchdog to monitor the state of your wg tunnel.

Check out the watchcat package.

Perhaps it requires tuning the conntrack settings:
https://github.com/openwrt/openwrt/blob/main/package/kernel/linux/files/sysctl-nf-conntrack.conf

Thanks for the quick reply !

The Wireguard tunnel is opened between VMs, not by the OWRT. When this situation happen, I need to close the wirguard connection on each VMs, find the conntrack connection with conntrack -L and close-it with conntrack -D. Then I can re-run the wg-quick up on the VMs.

If I forget to cut-off the Wireguard network, conntrack refuse to close the connection. If I restart the Wireguard connection on VMs without close the oppened connection on OWRT, the the same connection will be re-used and... nothing happend, still blocked :stuck_out_tongue:

Perhaps it requires tuning the conntrack settings:
https://github.com/openwrt/openwrt/blob/main/package/kernel/linux/files/sysctl-nf-conntrack.conf

Ow, thanks for the tip ! I will try to disable the "nf_conntrack_acct" option, to confirm or not that the number of packets is a problem.

You can also build VPN tunnels over IPv6, so it won't require NAT and connection tracking.
Then ingress transit VPN traffic can be allowed with explicit permissive firewall rules.
A static IPv6 prefix can be obtained from a free tunnel broker or cheap VPS provider.

I have many VMs on my Proxmox nodes, each VM have one or two Wireguard networks (not the same network on all VMs, for isolation). I can't buy an IPV6 for each VM.

The solution of disabling the nf_conntrack_acct to not count the packets seems to work, but I will continue the tests and write a new message about it. I will read some documentations about this feature, and the security risks of disabling-it :thinking:

An other solution, I guess, is to open a static port for each VM, each WG network, each directions of the tunnel. But this require to modify all my WG configurations on VMs, and so many ports to open :grimacing:

Okay, so... We can't disable the nf_conntrack_acct on OWRT 22 :frowning: Or I dont know how

# echo 0 > cat /proc/sys/net/netfilter/nf_conntrack_acct
# cat /proc/sys/net/netfilter/nf_conntrack_acct
1

After 6~7 hours, connection is blocked again... I need to stop the wireguard network on VMs, close the connection on OWRT, and re-start the wireguard network.

I will rework all my WG configuration, and forward a port on each direction for each node and network.

EDIT : My bad, wrong command for disabling the nf_conntrack_acct :sweat_smile:
I continue my tests with the good command (without the "cat")