Netfilter "Flow offload" / HW NAT


Every session (connection) has a inactivity timeout on the conntrack table in firewall: if packets are being exchanged between session endpoints, timeout are reset to zero, but: firewall checks packets to see if they are valid for a certain session/connection (source, destination, ports, sequence number, etc) and if they are all valid, allows traffic.


Ok, so in that case it isn't a security risk. But no longer being able to initiate new connections definitely is an issue. Also, this is clearly a bug since I can see active connections to a PC that was shutdown 12 hours ago :slight_smile:


That is strange and should be looked into. I personally do not have this problem so I cannot vouch nor help with it. What I can confirm is that the amount of connections active I've been seeing is because of the new dnscrypt-proxy 2.0 (new because it's based on Go) that I run on a different PC keeps active connections to every single DNS in the list that properly responds to queries. I thought I configured it to be its own DNS without asking the router for every single DNS query, but I guess not.


Wireguard 20180531 could work together with FLOWOFFLOAD finally. But it still has some bugs. Some websites can not load, or load slowly. If FLOWOFFLOAD disabled, or not visiting these sites through wireguard, they will be loaded quickly. All Google sites, connecting through QUIC, are not affected. I tried to do some research by wireshark, and found many tcp spurious retransmission.
Sorry for little information, maybe someone else could do more investigation.


Is anyone testing this patch from ndb's staging tree?

kernel: avoid flow offload for connections with xfrm on the dst entry (should fix IPSec);a=commit;h=df87ab765855f98673e861a6d60b1ef8ff8979b5



I haven't seen such bugs. All websites that I visit load fine, both on LAN devices and on the devices that are connected via Wireguard. Do you have an example of websites that don't load properly?



Yes, didn't fix IPSec for me. See my comment above :slight_smile:


Thanks for your feedback.
In fact, almost all the websites, except Google's sites which use QUIC. Maybe it's a TCP thing.
Could you give me some advice about how to diagnose it? I could ping these sites, but loading slowly in web browser. I have tried to lower the MTU of the wireguard device, but nothing changed.


Are you using:

client -> Wireguard on router -> internet


Client -> lan on router -> Wireguard on router -> VPN server -> internet


computer β†’ wireguard on router β†’ remote wireguard peer β†’ internet


And I recorded some messages, maybe it's userful for devs.

[102331.323686] WARNING: CPU: 1 PID: 28597 at net/netfilter/core.c:336 nf_unregister_net_hook+0x64/0xc8
[102331.332870] Modules linked in: pppoe ppp_async b43 pppox ppp_generic nf_conntrack_ipv6 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_recent xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_LOG xt_HL xt_FLOWOFFLOAD xt_DSCP xt_CT xt_CLASSIFY wireguard slhc nf_reject_ipv4 nf_nat_redirect nf_nat_masquerade_ipv4 nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_log_ipv4 nf_flow_table_hw nf_flow_table nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack macvlan iptable_mangle iptable_filter ipt_ECN ip_tables crc_ccitt compat ledtrig_usbport xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport
[102331.403932]  ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables x_tables ip6_udp_tunnel udp_tunnel sit tunnel4 ip_tunnel leds_gpio xhci_plat_hcd xhci_pci xhci_hcd ohci_platform ohci_hcd phy_bcm_ns_usb3 phy_bcm_ns_usb2 ehci_platform ehci_hcd gpio_button_hotplug usbcore nls_base usb_common bcma_hcd
[102331.455267] CPU: 1 PID: 28597 Comm: kworker/1:0 Not tainted 4.14.44 #0
[102331.461880] Hardware name: BCM5301X
[102331.465504] Workqueue: events_power_efficient xt_flowoffload_hook_work [xt_FLOWOFFLOAD]
[102331.473587] Backtrace: 
[102331.476188] [<c0105dcc>] (dump_backtrace) from [<c01060c4>] (show_stack+0x18/0x1c)
[102331.483852]  r7:c041b70c r6:60000113 r5:00000000 r4:c071d510
[102331.489628] [<c01060ac>] (show_stack) from [<c04e4230>] (dump_stack+0x90/0xa4)
[102331.496949] [<c04e41a0>] (dump_stack) from [<c011374c>] (__warn+0xec/0x108)
[102331.503996]  r7:c041b70c r6:c05ce1cc r5:00000000 r4:00000000
[102331.509760] [<c0113660>] (__warn) from [<c0113820>] (warn_slowpath_null+0x28/0x30)
[102331.517429]  r9:00000000 r8:00000000 r7:00000100 r6:c0716980 r5:c6ab8b48 r4:c6bd0230
[102331.525264] [<c01137f8>] (warn_slowpath_null) from [<c041b70c>] (nf_unregister_net_hook+0x64/0xc8)
[102331.534317] [<c041b6a8>] (nf_unregister_net_hook) from [<bf302400>] (xt_flowoffload_hook_work+0x154/0x1ac [xt_FLOWOFFLOAD])
[102331.545522]  r7:00000100 r6:bf303200 r5:bf303200 r4:c6ab8b40
[102331.551304] [<bf3022ac>] (xt_flowoffload_hook_work [xt_FLOWOFFLOAD]) from [<c01296cc>] (process_one_work+0x248/0x3d4)
[102331.561984]  r7:c6de9d00 r6:c6de69c0 r5:c6bc4780 r4:bf3032a4
[102331.567730] [<c0129484>] (process_one_work) from [<c012a740>] (worker_thread+0x358/0x590)
[102331.575993]  r10:c0702d00 r9:ffffe000 r8:00000008 r7:c6de69d8 r6:c6bc4798 r5:c6de69c0
[102331.583904]  r4:c6bc4780
[102331.586565] [<c012a3e8>] (worker_thread) from [<c012f1ac>] (kthread+0x140/0x14c)
[102331.594063]  r10:c6bc4780 r9:c7b6badc r8:c69c1e88 r7:00000000 r6:c7b6ba00 r5:c675a000
[102331.601977]  r4:c7b6bac0
[102331.604622] [<c012f06c>] (kthread) from [<c0102868>] (ret_from_fork+0x14/0x2c)
[102331.611923]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c012f06c
[102331.619827]  r4:c7b6ba00
[102331.622543] ---[ end trace 040970fbd0e98a17 ]---


What is the output of:

opkg list-installed | grep -i wireguard


Both kmod-wireguard and wireguard-tools are latest version 20180531.
In addition, my environment is:
computer (ipv4) β†’ wireguard on router(dual stack wan) β†’ 4in6 wireguard tunnel β†’ remote wireguard peer (dual stack) β†’ Internet
Everything is ok with FLOWOFFLAD disabled.


Send the full description of the issue + the stack trace to Wireguard's mailing list. They are most helpful. From what I can see everything is fine.

The only thing left to try is to enable MSS clamping on the Wireguard interface but other than that I am out of ideas.


so this is already in the luci packages? does it select the proper modules? i have an archer c7 v2 that has hardware nat offloading will this be better than fast path? im currently using the original firmware because im quaranteed gigabit speeds due to hardware nat offloading.


It is already included in Luci, yes. And the modules will automatically be included in the build if it is supported. Software flow offloading (similar to fastpath) requires kernel 4.14 to function. Unfortunately, Archer c7 is still on kernel 4.9 and hence is not supported. They are working on it to bring it to kernel 4.14 in the future. Hw flow offloading is only supported on the mt7621 platform as it is a SOC specific implementation. It will be brought to other SOCs in the future if possible. Not sure whether the SOC in the Archer c7 is open enough to be supported in the future.


As i know it depends on the switch and not on the soc for c7...
And as i know qca8k dsa driver does support hw nat and the archer c7 devices had all an ar8327 or ar8337 switch.
So it should be supported in the near feature...

With software flow offloading on kernel 4.14 you will get close to gigabit ---> ~920 MBit/s
DonΒ΄t thik that with hw nat you will get a significant higher through put than that...


This just landed :slight_smile:


@nbd is it technically possible to to have hw flow offload, without bypassing the q-disc? The reason I am asking, is because I love the offloading with regards to the heavy CPU load I am seeing with a PPPoE connection. However, not being able to use SQM/QoS is a pity. I am not sure how intertwined the whole process is, or whether is is possible to offload some parts, but not all.

Personally, the best setup would probably be to use hw offloading for PPPoE, while running software based flow offload for stuff like NAT/routing so that SQM can still be used.


So you didn't tried that virtual ethernet method you linked?