Yes, it is, that's why I asked if the issue is still being looked into.
OFFLOAD work well with mwan3 for me , but it has problem with wireguard. some sites such as inoreader.com load very slowly, and RTSP TCP connection is broken, UDP connection is not affected. I must bypass wireguard device to fix it temporarily.
iptables -I FORWARD -o wireguard -j ACCEPT
iptables -I FORWARD -i wireguard -j ACCEPT
It seems that it's a wireguard issue, other vpn client such as openconnect works fine.
After some more thorough testing on my Omnia, at home, I came to the conclusion that, as far as I can tell, this problem only occurs when both SQM (I only tested cake) and software flow offloading are enabled. I know flow offloading is an experimental feature, but I was expecting suboptimal performance rather than a showstopper. I guess I'll just file flow offloading under "don't do that", for the time being. 
Anyone know what the impact is of flow offloading to bufferbloat? On my Turris Omnia (mvebu) and my Netgear R7800, I'm getting higher ping while transferring on DSLReports' benchmark.
Strange behavior with latest 18.06.1 OpenWRT, they cant use LEDE code for that?
https://forum.openwrt.org/t/xiaomi-mi-wifi-router-3g-hw-nat/19703
Any idea what's wrong? random reboot seems to occur after enabling HW NAT.
Unfortunately I couldn't find exact step to reproduce.
Router: Mi Router 3G (MT7621)
Firmware: OpenWrt SNAPSHOT r8021-9e58c20
[ 9226.062793] Unhandled kernel unaligned access[#1]:
[ 9226.067594] CPU: 3 PID: 72 Comm: kworker/3:1 Not tainted 4.14.68 #0
[ 9226.073883] Workqueue: events_long nf_ct_kill_acct [nf_conntrack]
[ 9226.079952] task: 8fd712c0 task.stack: 8fdea000
[ 9226.084457] $ 0 : 00000000 00000001 8e222d98 000d904e
[ 9226.089667] $ 4 : 8f0f348c 00000000 0f55bc3d 00000001
[ 9226.094880] $ 8 : 00000000 00007c00 811ca500 0001225e
[ 9226.100092] $12 : 00000000 00000000 ffffffff 00000764
[ 9226.105306] $16 : 8f0f348c 8e9b7000 8e9b7000 00000000
[ 9226.110519] $20 : 000007da 805a0000 00000019 8f0d0000
[ 9226.115729] $24 : 00000000 8f0c06bc
[ 9226.120938] $28 : 8fdea000 8fdebdc0 805c1760 8f0f0864
[ 9226.126149] Hi : 00000b32
[ 9226.129011] Lo : 76457000
[ 9226.131889] epc : 8f0f086c nf_ct_nat_ext_add+0x218/0x928 [nf_nat]
[ 9226.138129] ra : 8f0f0864 nf_ct_nat_ext_add+0x210/0x928 [nf_nat]
[ 9226.144363] Status: 11007c03 KERNEL EXL IE
[ 9226.148533] Cause : 40800014 (ExcCode 05)
[ 9226.152517] BadVA : 000d904e
[ 9226.155382] PrId : 0001992f (MIPS 1004Kc)
[ 9226.159453] Modules linked in: pppoe ppp_async pppox ppp_generic nf_conntrack_ipv6 mt76x2e mt7603e mt76 mac80211 iptable_nat ipt_REJECT ipt_MASQUERADE cfg80211 xt_time
[ 9226.227867] Process kworker/3:1 (pid: 72, threadinfo=8fdea000, task=8fd712c0, tls=00000000)
[ 9226.236177] Stack : 00000000 000007da 805a0000 00000019 8e9b7000 8f0d0208 8f0d021c 8f0caf28
[ 9226.244516] 8e3a82b0 00000010 00000000 8e9b7000 8e9b7000 8e9b7040 00000013 8f0c066c
[ 9226.252857] 805fedc0 805a01b8 8123fdc0 00000001 8e9b7000 8f0c1c30 8fd715d8 80490000
[ 9226.261193] 805a0000 80056a98 8f0d0000 8f0d0000 00000020 00000002 8f0d01a4 8f0cce54
[ 9226.269530] 81242a00 8fd712c0 8f0d01a4 8fd0f180 8123fa40 81242a00 00000000 00000000
[ 9226.277864] ...
[ 9226.280314] Call Trace:
[ 9226.282764] [<8f0f086c>] nf_ct_nat_ext_add+0x218/0x928 [nf_nat]
[ 9226.288675] Code: 02002025 8e23007c 8e220078 <ac620000> 10400002 00000000 ac430004 24020200 ae22007c
[ 9226.298400]
[ 9226.300064] ---[ end trace bdd2cad862bd9103 ]---
I am going to try again, maybe someone knows how to fix this:
With SW (or HW) flow offload enabled, the PPTP client behind the router stops working.
The proper nat helper module is loaded, maybe nat helper modules are not compatible with Flow offload?
Any help would be appreciated.
Are you still having issues with a combination of mwan3 + wireguard + flow offload? everything is extremely slow for me on Xiaomi R3G with offload enabled (latest snapshot).
Yes, the problem still exists. I use following commangs to make wireguard interface bypass flowoffload.
iptables -A forwarding_rule -i wg -j ACCEPT
iptables -A forwarding_rule -o wg -j ACCEPT
I'm not using mwan3, so I don't know if it has compatible issues with flowoffload.
In my case all traffic goes through wireguard, so there is no need to have flow offload enabled at all then?
How do I check if router is doing hardware flow offloading ?
I built master for Buffalo WZR-HP-G300NH (ar71xx) and iperf was doing 350Mbs without offload and 500Mbs with offload. I expect something more, so I suspect is doing software offloading.
thank you
Currently, HW offloading is only supported in ramips/mt7621.
1 Like
If disabling flowoffload could solve your problem, just do it.
Can I bypass the flow offload with a similar rule for PPTP? If yes, can you give me an example? I have mt7621 based router, and would like to use the offload capability, but it breaks PPTP on the client side behind the router.
Thanks!
I'm afraid I can not help you, I have no experience about pptp config.
I have an notebook run Manjaro under my x86 Openwrt, if I enable flow offload, the remote ssh and scp access(login through wan, scp send file from notebook to my office PC) become very slow(50KB/s).
If I run a speedtest.net test on notebook, result is upload speed=25Mbps, and http server speed on the notebook is fine, and scp send file from office pc to notebook speed is normal too.
If I disable flow offload, the ssh and scp speed back to 1.8MB/s, mach the max upload speed 25Mbps.
Anyone same? I had to disable flow offload now.
yes, for me is the same, but I used the link over wireguard, if I exclude wireguard from flowofload everything is ok
in /etc/firewall.user something like
iptables -A forwarding_rule -i wg0 -j ACCEPT
iptables -A forwarding_rule -o wg0 -j ACCEPT
Hey guys, joining the offload testing party with similar report as last posts - slow upload, but with slightly different conditions.
I use mt7621 platfrorm with software + hw offloading enabled. My ISP uses PPPoE.
Right after firewall restarts I'm getting nice results on speedtests, ~150Mbit downling, ~20Mbit uplink. After a while though bad things starts to happening to uplink - it basically stalls after ~400KB. This is how it looks like for speedtest:
Testing upload speed..........
Upload: 0.46 Mbit/s
And now another interesting observation - I also observed transfer stalling on scp sending (similar to what @Cye3s described above, ofc I'm pretty sure this is protocol agnistic, it's just easy for me to scp). Interesting thing happened - I left transfer "hanging" for a while and it resumed properly after 2-3 minutes - kept working since then (even for new connections) while speedtest still stalls.
I'm definitely not exceeding nf_conntrack_max, during my tests value varied from 200 to 400 / 16k of max.
I remember that I've seen something similar on mwan3 setup and Quallcomm offload implementation that went viral ~year ago. The problem there was that it was properly offloading only for one of broadbands (let's call it WAN1). When I was sending something thorough WAN2 and exceeded offloading threshold (aka offloading got enabled on given connection) it was hanging in exactly the same way and sending following packets on WAN1.
I'll keep digging in this, will take a look on sources as well to understand the mechanism better. I have a feeling that in certain cases once we start offloading transfer stalls for some reason.
PS. while I was writing the post scp problems started to happening again, leaving it for a while to confirm that it will un-stall:
0% 2112KB 736.3KB/s - stalled
PS2. yes it did un-stall after ~20 seconds.
Replying to myself.
Delaying offload a bit seems to be helping. After I added following line into /etc/firewall.user it seems to be stable for over 1h now:
iptables -A forwarding_rule -m conntrack --ctstate RELATED,ESTABLISHED -m connbytes --connbytes 0:10240 --connbytes-dir both --connbytes-mode bytes -j ACCEPT
1 Like