Could be - although I'm getting the same result with flow offload enabled and disabled, so you would hope in the disabled state it's not actually running that code ... but I suppose we'll find out later, about to flash my "last known good" master build and will then work from there.
Update 2: revision 7007 flashes and works, but I can't SSH in to run iperf3
Update: revision 6985 (last before 4.14 changes and kernel partition resize) also sees full performance.
OK, revision 6739 gets full beans:
root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[ 5] local xxx.xxx.xxx.xxx port 46784 connected to 62.210.18.40 port 5207
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 6.71 MBytes 56.3 Mbits/sec
[ 5] 1.00-2.00 sec 18.8 MBytes 157 Mbits/sec
[ 5] 2.00-3.00 sec 23.5 MBytes 197 Mbits/sec
[ 5] 3.00-4.00 sec 32.8 MBytes 275 Mbits/sec
[ 5] 4.00-5.00 sec 45.8 MBytes 384 Mbits/sec
[ 5] 5.00-6.00 sec 45.8 MBytes 384 Mbits/sec
[ 5] 6.00-7.00 sec 45.8 MBytes 384 Mbits/sec
[ 5] 7.00-8.00 sec 45.8 MBytes 384 Mbits/sec
[ 5] 8.00-9.00 sec 45.8 MBytes 384 Mbits/sec
[ 5] 9.00-10.00 sec 45.8 MBytes 384 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 361 MBytes 303 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 356 MBytes 299 Mbits/sec receiver
Going to try a master just before the 4.14 for ipq806x went in to see if that is still fast, then first stable build of 4.14 (after the flurry of commits)
I get the same results with r6985:
root@OpenWrt:~# iperf3 -c ping.online.net -p 5205 -R
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[ 5] local 51.175.71.139 port 58508 connected to 62.210.18.40 port 5205
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 3.34 MBytes 28.0 Mbits/sec
[ 5] 1.00-2.00 sec 8.44 MBytes 70.7 Mbits/sec
[ 5] 2.00-3.00 sec 18.1 MBytes 152 Mbits/sec
[ 5] 3.00-4.00 sec 42.7 MBytes 358 Mbits/sec
[ 5] 4.00-5.00 sec 45.5 MBytes 382 Mbits/sec
[ 5] 5.00-6.00 sec 45.9 MBytes 385 Mbits/sec
[ 5] 6.00-7.00 sec 45.5 MBytes 381 Mbits/sec
[ 5] 7.00-8.00 sec 45.9 MBytes 385 Mbits/sec
[ 5] 8.00-9.00 sec 45.6 MBytes 383 Mbits/sec
[ 5] 9.00-10.00 sec 45.9 MBytes 385 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 351 MBytes 294 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 347 MBytes 291 Mbits/sec receiver
iperf Done.
Seems like a single stream caps out at ~385 Mbit/sec at this revision. I'm able to max out my connection with two parallel streams:
root@OpenWrt:~# iperf3 -c ping.online.net -p 5205 -R -P 2
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[ 5] local 51.175.71.139 port 58526 connected to 62.210.18.40 port 5205
[ 7] local 51.175.71.139 port 58528 connected to 62.210.18.40 port 5205
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 3.39 MBytes 28.5 Mbits/sec
[ 7] 0.00-1.00 sec 4.09 MBytes 34.3 Mbits/sec
[SUM] 0.00-1.00 sec 7.48 MBytes 62.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 1.00-2.00 sec 16.9 MBytes 142 Mbits/sec
[ 7] 1.00-2.00 sec 16.0 MBytes 134 Mbits/sec
[SUM] 1.00-2.00 sec 32.9 MBytes 276 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 2.00-3.00 sec 32.5 MBytes 273 Mbits/sec
[ 7] 2.00-3.00 sec 32.7 MBytes 275 Mbits/sec
[SUM] 2.00-3.00 sec 65.2 MBytes 548 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 3.00-4.00 sec 30.0 MBytes 252 Mbits/sec
[ 7] 3.00-4.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 3.00-4.00 sec 59.9 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 4.00-5.00 sec 30.1 MBytes 253 Mbits/sec
[ 7] 4.00-5.00 sec 29.9 MBytes 250 Mbits/sec
[SUM] 4.00-5.00 sec 60.0 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 5.00-6.00 sec 30.0 MBytes 252 Mbits/sec
[ 7] 5.00-6.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 5.00-6.00 sec 59.9 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 6.00-7.00 sec 30.1 MBytes 252 Mbits/sec
[ 7] 6.00-7.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 6.00-7.00 sec 59.9 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 7.00-8.00 sec 30.1 MBytes 252 Mbits/sec
[ 7] 7.00-8.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 7.00-8.00 sec 59.9 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 8.00-9.00 sec 30.0 MBytes 252 Mbits/sec
[ 7] 8.00-9.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 8.00-9.00 sec 59.9 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.00-10.00 sec 30.1 MBytes 252 Mbits/sec
[ 7] 9.00-10.00 sec 29.9 MBytes 251 Mbits/sec
[SUM] 9.00-10.00 sec 60.0 MBytes 503 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 267 MBytes 224 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 263 MBytes 221 Mbits/sec receiver
[ 7] 0.00-10.00 sec 267 MBytes 224 Mbits/sec 0 sender
[ 7] 0.00-10.00 sec 262 MBytes 220 Mbits/sec receiver
[SUM] 0.00-10.00 sec 534 MBytes 448 Mbits/sec 0 sender
[SUM] 0.00-10.00 sec 525 MBytes 441 Mbits/sec receiver
iperf Done.
Yep, that's to be expected. If you want to go faster with a single stream you could try the SFE patch that dissent1 put up on the old lede repo, that's the old hacky alternative to the nf flow offload that's in 4.14
So it points to there being something between 6985 and now that causes a huge drop in single stream performance.
Sadly I'm logged into work and running a pg_dump on a huge table that I don't want to interrupt, so can't go to the next version I have built, 7007, until that's finished sometime in the next hour or so.
On the R7800 running r7184, with flow offloading enabled:
root@OpenWrt:~# iperf3 -c ping.online.net -p 5205 -R
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[ 5] local xxx.xxx.xxx.xxx port 49666 connected to 62.210.18.40 port 5205
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 15.8 MBytes 132 Mbits/sec
[ 5] 1.00-2.00 sec 16.7 MBytes 140 Mbits/sec
[ 5] 2.00-3.00 sec 16.6 MBytes 140 Mbits/sec
[ 5] 3.00-4.00 sec 16.6 MBytes 139 Mbits/sec
[ 5] 4.00-5.00 sec 16.8 MBytes 140 Mbits/sec
[ 5] 5.00-6.00 sec 16.7 MBytes 140 Mbits/sec
[ 5] 6.00-7.00 sec 16.6 MBytes 140 Mbits/sec
[ 5] 7.00-8.00 sec 16.8 MBytes 141 Mbits/sec
[ 5] 8.00-9.00 sec 16.7 MBytes 140 Mbits/sec
[ 5] 9.00-10.00 sec 16.7 MBytes 140 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 167 MBytes 140 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 166 MBytes 139 Mbits/sec receiver
iperf Done.
But on a Ubuntu machine connected to the R7800 by Ethernet:
$ iperf3 -c ping.online.net -p 5205 -R
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[ 4] local 192.168.1.223 port 51042 connected to 62.210.18.40 port 5205
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 50.1 MBytes 420 Mbits/sec
[ 4] 1.00-2.00 sec 68.9 MBytes 578 Mbits/sec
[ 4] 2.00-3.00 sec 68.6 MBytes 576 Mbits/sec
[ 4] 3.00-4.00 sec 83.1 MBytes 697 Mbits/sec
[ 4] 4.00-5.00 sec 77.6 MBytes 651 Mbits/sec
[ 4] 5.00-6.00 sec 76.7 MBytes 643 Mbits/sec
[ 4] 6.00-7.00 sec 74.4 MBytes 624 Mbits/sec
[ 4] 7.00-8.00 sec 62.0 MBytes 520 Mbits/sec
[ 4] 8.00-9.00 sec 61.8 MBytes 519 Mbits/sec
[ 4] 9.00-10.00 sec 50.2 MBytes 421 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 680 MBytes 571 Mbits/sec 616 sender
[ 4] 0.00-10.00 sec 674 MBytes 565 Mbits/sec receiver
iperf Done.
I ran git bisect to identify the commit that causes the regression and I identified this kernel bump as the cause (r7106):
https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=e52f3e9b13763cbfbd92fca8db0b4d19dc1e309e
I've run an iperf3 test on the preceding commit (r7105) to make sure the speeds are normal:
Link to commit: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=7590c3c58f5e9d580c86da10473d1d29a2f081c9
root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[ 5] local 51.175.71.139 port 48514 connected to 62.210.18.40 port 5207
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.24 MBytes 18.8 Mbits/sec
[ 5] 1.00-2.00 sec 17.1 MBytes 144 Mbits/sec
[ 5] 2.00-3.00 sec 35.6 MBytes 299 Mbits/sec
[ 5] 3.00-4.00 sec 39.4 MBytes 330 Mbits/sec
[ 5] 4.00-5.00 sec 39.3 MBytes 330 Mbits/sec
[ 5] 5.00-6.00 sec 39.1 MBytes 327 Mbits/sec
[ 5] 6.00-7.00 sec 38.7 MBytes 325 Mbits/sec
[ 5] 7.00-8.00 sec 39.1 MBytes 328 Mbits/sec
[ 5] 8.00-9.00 sec 39.3 MBytes 330 Mbits/sec
[ 5] 9.00-10.00 sec 38.8 MBytes 325 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 333 MBytes 280 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 329 MBytes 276 Mbits/sec receiver
Here are the speeds on the kernel bump commit:
root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[ 5] local 51.175.71.139 port 45862 connected to 62.210.18.40 port 5207
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.42 MBytes 20.2 Mbits/sec
[ 5] 1.00-2.00 sec 3.44 MBytes 28.9 Mbits/sec
[ 5] 2.00-3.00 sec 3.61 MBytes 30.3 Mbits/sec
[ 5] 3.00-4.00 sec 3.45 MBytes 28.9 Mbits/sec
[ 5] 4.00-5.00 sec 3.55 MBytes 29.8 Mbits/sec
[ 5] 5.00-6.00 sec 3.53 MBytes 29.6 Mbits/sec
[ 5] 6.00-7.00 sec 3.46 MBytes 29.0 Mbits/sec
[ 5] 7.00-8.00 sec 3.62 MBytes 30.4 Mbits/sec
[ 5] 8.00-9.00 sec 3.45 MBytes 28.9 Mbits/sec
[ 5] 9.00-10.00 sec 3.60 MBytes 30.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 34.6 MBytes 29.0 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 34.1 MBytes 28.6 Mbits/sec receiver
And for good measure I also tested the latest commit (r7290):
Link to commit: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=3b11b225b32883d730a641c5aebef6f018c91649
root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[ 5] local 51.175.71.139 port 43702 connected to 62.210.18.40 port 5207
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.91 MBytes 16.0 Mbits/sec
[ 5] 1.00-2.00 sec 3.46 MBytes 29.0 Mbits/sec
[ 5] 2.00-3.00 sec 3.61 MBytes 30.3 Mbits/sec
[ 5] 3.00-4.00 sec 3.46 MBytes 29.0 Mbits/sec
[ 5] 4.00-5.00 sec 3.61 MBytes 30.3 Mbits/sec
[ 5] 5.00-6.00 sec 3.46 MBytes 29.0 Mbits/sec
[ 5] 6.00-7.00 sec 3.50 MBytes 29.3 Mbits/sec
[ 5] 7.00-8.00 sec 3.58 MBytes 30.1 Mbits/sec
[ 5] 8.00-9.00 sec 3.45 MBytes 28.9 Mbits/sec
[ 5] 9.00-10.00 sec 3.62 MBytes 30.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 34.1 MBytes 28.6 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 33.7 MBytes 28.2 Mbits/sec receiver
NOTE: all tests are performed with flow offload disabled!
It's quite relieving that it's not the switch between kernel 4.9 and 4.14 (which would have been really painful to debug), but a generic and rather isolated -stable bump instead, something that could be bisected even further.
now we should check if it's related to the kernel or some wrong path refresh...
i think the culprit is in the bump to 4.14.48 (so we need to check what change between this 2 kernel version about ipq)
I just tried Kevin's 4.14.51 kernel bump patch and it seems like it fixes the issue:
root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[ 5] local 51.175.71.139 port 40326 connected to 62.210.18.40 port 5207
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 2.39 MBytes 20.1 Mbits/sec
[ 5] 1.00-2.00 sec 6.79 MBytes 56.9 Mbits/sec
[ 5] 2.00-3.00 sec 20.5 MBytes 172 Mbits/sec
[ 5] 3.00-4.00 sec 46.3 MBytes 388 Mbits/sec
[ 5] 4.00-5.00 sec 52.4 MBytes 440 Mbits/sec
[ 5] 5.00-6.00 sec 52.5 MBytes 440 Mbits/sec
[ 5] 6.00-7.00 sec 52.5 MBytes 441 Mbits/sec
[ 5] 7.00-8.00 sec 52.6 MBytes 441 Mbits/sec
[ 5] 8.00-9.00 sec 52.7 MBytes 441 Mbits/sec
[ 5] 9.00-10.00 sec 52.1 MBytes 437 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 395 MBytes 331 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 391 MBytes 328 Mbits/sec receiver
iperf Done.
Here's the patch: https://patchwork.ozlabs.org/patch/932599/
Have I understood correctly that
- 4.14.44 was ok,
- .48 was already bad,
- current .50 is bad and
- forthcoming .51 is again good.
then I Think it was a kernel regression... nothing related to openwrt....
Yep, it sounds like that. I tried identifying the possible fix from the kernel changelog, but did not quickly pick up anything likely, although naturally there are several network related commits.
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/log/?h=linux-4.14.y
(18.06.0-rc1 is planned to be tagged tomorrow, and currently .50 is still there in 18.06. Possibly .51 gets only into rc2 or final. From that perspective it would be really helpful to be certain that .51 really fixes the problem. So hopefully people with speedy connections do test that kernel bump patch.)
well i think that kernel should be upgraded in the release just to be sure...
Could be a problem with every 4.14 device...
That's correct.
It would be nice if someone else with a fast connection could confirm my results.
Can confirm. iperf maxes out now.
Boo. iperf3 drops by half when connected to WAN.
iperf3 -c openwrt
Connecting to host openwrt, port 5201
[ 5] local fd9a:d091:22c5::ddd port 49838 connected to fd9a:d091:22c5::1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 63.2 MBytes 530 Mbits/sec 217 787 KBytes
[ 5] 1.00-2.00 sec 61.1 MBytes 513 Mbits/sec 0 841 KBytes
[ 5] 2.00-3.00 sec 62.3 MBytes 523 Mbits/sec 0 891 KBytes
0[ 5] 3.00-4.00 sec 59.9 MBytes 502 Mbits/sec 0 941 KBytes
[ 5] 4.00-5.00 sec 61.0 MBytes 512 Mbits/sec 0 990 KBytes
[ 5] 5.00-6.00 sec 62.3 MBytes 523 Mbits/sec 0 1.01 MBytes
[ 5] 6.00-7.00 sec 61.1 MBytes 513 Mbits/sec 0 1.05 MBytes
[ 5] 7.00-8.00 sec 61.1 MBytes 512 Mbits/sec 0 1.09 MBytes
[ 5] 8.00-9.00 sec 61.1 MBytes 513 Mbits/sec 0 1.13 MBytes
[ 5] 9.00-10.00 sec 62.4 MBytes 523 Mbits/sec 0 1.17 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 615 MBytes 516 Mbits/sec 217 sender
[ 5] 0.00-10.04 sec 613 MBytes 512 Mbits/sec receiver
iperf Done.
Full speed if it's just the computer connected. Guess that's the one issue with qca8k.
edit: I could add a USB WAN port. But USB has a lot of buffering...
Just built 18.06 with 4.14.51 patch applied, will test in the next hour or so.
@hnyman in think we should include in the release also led patch and the fix with 160... (if someone select 160mhz the firmware crash, not so good for a stable release!!!)
Led patch got merged in trunk...
Are you running qca8k?
You did not look into 18.06?
Both were backported there right after they got into master.