Netfilter "Flow offload" / HW NAT

It seems that we could bypass wireguard interface, for example

iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -i pppoe-wan -j FLOWOFFLOAD

@lucize

Unfortunately didn't solve it for me. Disabling flow offload or inserting an ACCEPT rule in the FORWARD chain before it makes it work.

@nbd

Any comment on this? I'm running strongswan (IKEv2) on my router and use latest OpenWrt git including your 650-netfilter-add-xt_OFFLOAD-target.patch patch.

hmm...actually i have the problem with the firestick (but not the leaking connections) even if offloading is turned off. bouncing the firewall fixes it. i haven't had this problem with anything other apps/devices yet.

Yes, PPPoE can be a bottleneck, it's why the Qualcomm NSS cores include PPPoE acceleration.

Also for me, is that how it should work?

As far as I understand, this shouldn't happen. These connections should properly time-out like they do without flow offload. Because eventually, the conntrack table will fill completely and you won't be able to open any more new connections. I haven't actually tried that yet myself though. I turned off flowoffload when I noticed this bug.

hw flow offload on mt7621 also offloads PPPoE. I am seeing 97-98% idle CPU usage while fully loading my 500/500 mbit connection. I was simply trying to figure out how I can keep the PPPoE offloading enabled and do the shaping on a dummy ethernet interface. Haven't had time yet to try this myself, but @dlakelan has a very interesting post regarding this on his blog: http://models.street-artists.org/2017/12/11/inbound-qos-with-virtual-ethernet-and-policy-routing/

Amazingly cool stuff :smiley:

1 Like

Thanks for your information.

based on that i created a little firewall.user script to do that on every interface except wireguard, it will offload pppoe-wan, wan, wan6, vpn interfaces, everything, just copy to the firewall.user file. If hw=1 then it is hw offloaded if 0 then software...

#network flow offload enable on ALL interfaces except wireguards
hw=1
ifaces=$(uci show network |grep =interface |sed "s|=.*||")
for i in $ifaces;do
	ifname=$(uci get $i.ifname)
	if [ ! "$(uci get $i.proto|grep wireguard)" ];then
		[ $hw ]&&hwswitch="--hw" ||hwswitch=""
		iptables -I FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -i $ifname -j FLOWOFFLOAD $hwswitch
	fi
done
4 Likes

I can confirm my active connection count is much higher with flowoffload on than when it is off. Although for me, I consider it a small tradeoff for the kind of speeds I can get with it enabled.
Aside from this, it hasn't become so big that it's unmanageable: from 109 average to 460 average. I'm seeing 150 total and 70 average on the graphs. I'm also using unbound with Cloudflare's DNS-over-TLS which is known to keep a lot of connections active when using it so that could potentially be the reason as well.
Related question: I know I can increase the default 16384 shown in "active connections" because it is nothing but conntrack counts, right? A simple sysctl change should increase this number, but what are the side effects? higher RAM usage?

Yes, but is minimal.

Aren't active connections also a security risk in a sense? The firewall accepts packets for established and related connections. If connections linger around indefinitely, that might mean that even weeks later the firewall will accept these packets.

Also, eventually the conntrack table will be full and setting up new connections will be impossible.

Wireguard + Flow offload should work together after this patch hits the master branch: http://lists.infradead.org/pipermail/openwrt-devel/2018-May/012675.html :slight_smile:

2 Likes

Every session (connection) has a inactivity timeout on the conntrack table in firewall: if packets are being exchanged between session endpoints, timeout are reset to zero, but: firewall checks packets to see if they are valid for a certain session/connection (source, destination, ports, sequence number, etc) and if they are all valid, allows traffic.

Ok, so in that case it isn't a security risk. But no longer being able to initiate new connections definitely is an issue. Also, this is clearly a bug since I can see active connections to a PC that was shutdown 12 hours ago :slight_smile:

That is strange and should be looked into. I personally do not have this problem so I cannot vouch nor help with it. What I can confirm is that the amount of connections active I've been seeing is because of the new dnscrypt-proxy 2.0 (new because it's based on Go) that I run on a different PC keeps active connections to every single DNS in the list that properly responds to queries. I thought I configured it to be its own DNS without asking the router for every single DNS query, but I guess not.

Wireguard 20180531 could work together with FLOWOFFLOAD finally. But it still has some bugs. Some websites can not load, or load slowly. If FLOWOFFLOAD disabled, or not visiting these sites through wireguard, they will be loaded quickly. All Google sites, connecting through QUIC, are not affected. I tried to do some research by wireshark, and found many tcp spurious retransmission.
Sorry for little information, maybe someone else could do more investigation.

Is anyone testing this patch from ndb's staging tree?

kernel: avoid flow offload for connections with xfrm on the dst entry (should fix IPSec) https://git.openwrt.org/?p=openwrt/staging/nbd.git;a=commit;h=df87ab765855f98673e861a6d60b1ef8ff8979b5

Results?

I haven't seen such bugs. All websites that I visit load fine, both on LAN devices and on the devices that are connected via Wireguard. Do you have an example of websites that don't load properly?

@bouwew

Yes, didn't fix IPSec for me. See my comment above :slight_smile:

Thanks for your feedback.
In fact, almost all the websites, except Google's sites which use QUIC. Maybe it's a TCP thing.
Could you give me some advice about how to diagnose it? I could ping these sites, but loading slowly in web browser. I have tried to lower the MTU of the wireguard device, but nothing changed.

Are you using:

client -> Wireguard on router -> internet

Or

Client -> lan on router -> Wireguard on router -> VPN server -> internet