Netgear R7800 exploration (IPQ8065, QCA9984)

Could be - although I'm getting the same result with flow offload enabled and disabled, so you would hope in the disabled state it's not actually running that code ... but I suppose we'll find out later, about to flash my "last known good" master build and will then work from there.

Update 2: revision 7007 flashes and works, but I can't SSH in to run iperf3

Update: revision 6985 (last before 4.14 changes and kernel partition resize) also sees full performance.

OK, revision 6739 gets full beans:

root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[  5] local xxx.xxx.xxx.xxx port 46784 connected to 62.210.18.40 port 5207
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  6.71 MBytes  56.3 Mbits/sec
[  5]   1.00-2.00   sec  18.8 MBytes   157 Mbits/sec
[  5]   2.00-3.00   sec  23.5 MBytes   197 Mbits/sec
[  5]   3.00-4.00   sec  32.8 MBytes   275 Mbits/sec
[  5]   4.00-5.00   sec  45.8 MBytes   384 Mbits/sec
[  5]   5.00-6.00   sec  45.8 MBytes   384 Mbits/sec
[  5]   6.00-7.00   sec  45.8 MBytes   384 Mbits/sec
[  5]   7.00-8.00   sec  45.8 MBytes   384 Mbits/sec
[  5]   8.00-9.00   sec  45.8 MBytes   384 Mbits/sec
[  5]   9.00-10.00  sec  45.8 MBytes   384 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   361 MBytes   303 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   356 MBytes   299 Mbits/sec                  receiver

Going to try a master just before the 4.14 for ipq806x went in to see if that is still fast, then first stable build of 4.14 (after the flurry of commits)

I get the same results with r6985:

root@OpenWrt:~# iperf3 -c ping.online.net -p 5205 -R
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[  5] local 51.175.71.139 port 58508 connected to 62.210.18.40 port 5205
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.34 MBytes  28.0 Mbits/sec                  
[  5]   1.00-2.00   sec  8.44 MBytes  70.7 Mbits/sec                  
[  5]   2.00-3.00   sec  18.1 MBytes   152 Mbits/sec                  
[  5]   3.00-4.00   sec  42.7 MBytes   358 Mbits/sec                  
[  5]   4.00-5.00   sec  45.5 MBytes   382 Mbits/sec                  
[  5]   5.00-6.00   sec  45.9 MBytes   385 Mbits/sec                  
[  5]   6.00-7.00   sec  45.5 MBytes   381 Mbits/sec                  
[  5]   7.00-8.00   sec  45.9 MBytes   385 Mbits/sec                  
[  5]   8.00-9.00   sec  45.6 MBytes   383 Mbits/sec                  
[  5]   9.00-10.00  sec  45.9 MBytes   385 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   351 MBytes   294 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   347 MBytes   291 Mbits/sec                  receiver

iperf Done.

Seems like a single stream caps out at ~385 Mbit/sec at this revision. I'm able to max out my connection with two parallel streams:

root@OpenWrt:~# iperf3 -c ping.online.net -p 5205 -R -P 2
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[  5] local 51.175.71.139 port 58526 connected to 62.210.18.40 port 5205
[  7] local 51.175.71.139 port 58528 connected to 62.210.18.40 port 5205
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  3.39 MBytes  28.5 Mbits/sec                  
[  7]   0.00-1.00   sec  4.09 MBytes  34.3 Mbits/sec                  
[SUM]   0.00-1.00   sec  7.48 MBytes  62.8 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  16.9 MBytes   142 Mbits/sec                  
[  7]   1.00-2.00   sec  16.0 MBytes   134 Mbits/sec                  
[SUM]   1.00-2.00   sec  32.9 MBytes   276 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  32.5 MBytes   273 Mbits/sec                  
[  7]   2.00-3.00   sec  32.7 MBytes   275 Mbits/sec                  
[SUM]   2.00-3.00   sec  65.2 MBytes   548 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  30.0 MBytes   252 Mbits/sec                  
[  7]   3.00-4.00   sec  29.9 MBytes   251 Mbits/sec                  
[SUM]   3.00-4.00   sec  59.9 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  30.1 MBytes   253 Mbits/sec                  
[  7]   4.00-5.00   sec  29.9 MBytes   250 Mbits/sec                  
[SUM]   4.00-5.00   sec  60.0 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  30.0 MBytes   252 Mbits/sec                  
[  7]   5.00-6.00   sec  29.9 MBytes   251 Mbits/sec                  
[SUM]   5.00-6.00   sec  59.9 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  30.1 MBytes   252 Mbits/sec                  
[  7]   6.00-7.00   sec  29.9 MBytes   251 Mbits/sec                  
[SUM]   6.00-7.00   sec  59.9 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  30.1 MBytes   252 Mbits/sec                  
[  7]   7.00-8.00   sec  29.9 MBytes   251 Mbits/sec                  
[SUM]   7.00-8.00   sec  59.9 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  30.0 MBytes   252 Mbits/sec                  
[  7]   8.00-9.00   sec  29.9 MBytes   251 Mbits/sec                  
[SUM]   8.00-9.00   sec  59.9 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  30.1 MBytes   252 Mbits/sec                  
[  7]   9.00-10.00  sec  29.9 MBytes   251 Mbits/sec                  
[SUM]   9.00-10.00  sec  60.0 MBytes   503 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   267 MBytes   224 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   263 MBytes   221 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   267 MBytes   224 Mbits/sec    0             sender
[  7]   0.00-10.00  sec   262 MBytes   220 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec   534 MBytes   448 Mbits/sec    0             sender
[SUM]   0.00-10.00  sec   525 MBytes   441 Mbits/sec                  receiver

iperf Done.

Yep, that's to be expected. If you want to go faster with a single stream you could try the SFE patch that dissent1 put up on the old lede repo, that's the old hacky alternative to the nf flow offload that's in 4.14

So it points to there being something between 6985 and now that causes a huge drop in single stream performance.

Sadly I'm logged into work and running a pg_dump on a huge table that I don't want to interrupt, so can't go to the next version I have built, 7007, until that's finished sometime in the next hour or so.

1 Like

On the R7800 running r7184, with flow offloading enabled:

root@OpenWrt:~# iperf3 -c ping.online.net -p 5205 -R
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[  5] local xxx.xxx.xxx.xxx port 49666 connected to 62.210.18.40 port 5205
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  15.8 MBytes   132 Mbits/sec
[  5]   1.00-2.00   sec  16.7 MBytes   140 Mbits/sec
[  5]   2.00-3.00   sec  16.6 MBytes   140 Mbits/sec
[  5]   3.00-4.00   sec  16.6 MBytes   139 Mbits/sec
[  5]   4.00-5.00   sec  16.8 MBytes   140 Mbits/sec
[  5]   5.00-6.00   sec  16.7 MBytes   140 Mbits/sec
[  5]   6.00-7.00   sec  16.6 MBytes   140 Mbits/sec
[  5]   7.00-8.00   sec  16.8 MBytes   141 Mbits/sec
[  5]   8.00-9.00   sec  16.7 MBytes   140 Mbits/sec
[  5]   9.00-10.00  sec  16.7 MBytes   140 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   167 MBytes   140 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   166 MBytes   139 Mbits/sec                  receiver

iperf Done.

But on a Ubuntu machine connected to the R7800 by Ethernet:

$ iperf3 -c ping.online.net -p 5205 -R
Connecting to host ping.online.net, port 5205
Reverse mode, remote host ping.online.net is sending
[  4] local 192.168.1.223 port 51042 connected to 62.210.18.40 port 5205
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  50.1 MBytes   420 Mbits/sec
[  4]   1.00-2.00   sec  68.9 MBytes   578 Mbits/sec
[  4]   2.00-3.00   sec  68.6 MBytes   576 Mbits/sec
[  4]   3.00-4.00   sec  83.1 MBytes   697 Mbits/sec
[  4]   4.00-5.00   sec  77.6 MBytes   651 Mbits/sec
[  4]   5.00-6.00   sec  76.7 MBytes   643 Mbits/sec
[  4]   6.00-7.00   sec  74.4 MBytes   624 Mbits/sec
[  4]   7.00-8.00   sec  62.0 MBytes   520 Mbits/sec
[  4]   8.00-9.00   sec  61.8 MBytes   519 Mbits/sec
[  4]   9.00-10.00  sec  50.2 MBytes   421 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   680 MBytes   571 Mbits/sec  616             sender
[  4]   0.00-10.00  sec   674 MBytes   565 Mbits/sec                  receiver

iperf Done.

I ran git bisect to identify the commit that causes the regression and I identified this kernel bump as the cause (r7106):

https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=e52f3e9b13763cbfbd92fca8db0b4d19dc1e309e


I've run an iperf3 test on the preceding commit (r7105) to make sure the speeds are normal:

Link to commit: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=7590c3c58f5e9d580c86da10473d1d29a2f081c9

root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[  5] local 51.175.71.139 port 48514 connected to 62.210.18.40 port 5207
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.24 MBytes  18.8 Mbits/sec                  
[  5]   1.00-2.00   sec  17.1 MBytes   144 Mbits/sec                  
[  5]   2.00-3.00   sec  35.6 MBytes   299 Mbits/sec                  
[  5]   3.00-4.00   sec  39.4 MBytes   330 Mbits/sec                  
[  5]   4.00-5.00   sec  39.3 MBytes   330 Mbits/sec                  
[  5]   5.00-6.00   sec  39.1 MBytes   327 Mbits/sec                  
[  5]   6.00-7.00   sec  38.7 MBytes   325 Mbits/sec                  
[  5]   7.00-8.00   sec  39.1 MBytes   328 Mbits/sec                  
[  5]   8.00-9.00   sec  39.3 MBytes   330 Mbits/sec                  
[  5]   9.00-10.00  sec  38.8 MBytes   325 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   333 MBytes   280 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   329 MBytes   276 Mbits/sec                  receiver


Here are the speeds on the kernel bump commit:

root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[  5] local 51.175.71.139 port 45862 connected to 62.210.18.40 port 5207
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.42 MBytes  20.2 Mbits/sec                  
[  5]   1.00-2.00   sec  3.44 MBytes  28.9 Mbits/sec                  
[  5]   2.00-3.00   sec  3.61 MBytes  30.3 Mbits/sec                  
[  5]   3.00-4.00   sec  3.45 MBytes  28.9 Mbits/sec                  
[  5]   4.00-5.00   sec  3.55 MBytes  29.8 Mbits/sec                  
[  5]   5.00-6.00   sec  3.53 MBytes  29.6 Mbits/sec                  
[  5]   6.00-7.00   sec  3.46 MBytes  29.0 Mbits/sec                  
[  5]   7.00-8.00   sec  3.62 MBytes  30.4 Mbits/sec                  
[  5]   8.00-9.00   sec  3.45 MBytes  28.9 Mbits/sec                  
[  5]   9.00-10.00  sec  3.60 MBytes  30.2 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  34.6 MBytes  29.0 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  34.1 MBytes  28.6 Mbits/sec                  receiver


And for good measure I also tested the latest commit (r7290):

Link to commit: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=3b11b225b32883d730a641c5aebef6f018c91649

root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[  5] local 51.175.71.139 port 43702 connected to 62.210.18.40 port 5207
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.91 MBytes  16.0 Mbits/sec                  
[  5]   1.00-2.00   sec  3.46 MBytes  29.0 Mbits/sec                  
[  5]   2.00-3.00   sec  3.61 MBytes  30.3 Mbits/sec                  
[  5]   3.00-4.00   sec  3.46 MBytes  29.0 Mbits/sec                  
[  5]   4.00-5.00   sec  3.61 MBytes  30.3 Mbits/sec                  
[  5]   5.00-6.00   sec  3.46 MBytes  29.0 Mbits/sec                  
[  5]   6.00-7.00   sec  3.50 MBytes  29.3 Mbits/sec                  
[  5]   7.00-8.00   sec  3.58 MBytes  30.1 Mbits/sec                  
[  5]   8.00-9.00   sec  3.45 MBytes  28.9 Mbits/sec                  
[  5]   9.00-10.00  sec  3.62 MBytes  30.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  34.1 MBytes  28.6 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  33.7 MBytes  28.2 Mbits/sec                  receiver


NOTE: all tests are performed with flow offload disabled!

1 Like

It's quite relieving that it's not the switch between kernel 4.9 and 4.14 (which would have been really painful to debug), but a generic and rather isolated -stable bump instead, something that could be bisected even further.

now we should check if it's related to the kernel or some wrong path refresh...

i think the culprit is in the bump to 4.14.48 (so we need to check what change between this 2 kernel version about ipq)

I just tried Kevin's 4.14.51 kernel bump patch and it seems like it fixes the issue:

root@OpenWrt:~# iperf3 -c ping.online.net -p 5207 -R
Connecting to host ping.online.net, port 5207
Reverse mode, remote host ping.online.net is sending
[  5] local 51.175.71.139 port 40326 connected to 62.210.18.40 port 5207
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  2.39 MBytes  20.1 Mbits/sec                  
[  5]   1.00-2.00   sec  6.79 MBytes  56.9 Mbits/sec                  
[  5]   2.00-3.00   sec  20.5 MBytes   172 Mbits/sec                  
[  5]   3.00-4.00   sec  46.3 MBytes   388 Mbits/sec                  
[  5]   4.00-5.00   sec  52.4 MBytes   440 Mbits/sec                  
[  5]   5.00-6.00   sec  52.5 MBytes   440 Mbits/sec                  
[  5]   6.00-7.00   sec  52.5 MBytes   441 Mbits/sec                  
[  5]   7.00-8.00   sec  52.6 MBytes   441 Mbits/sec                  
[  5]   8.00-9.00   sec  52.7 MBytes   441 Mbits/sec                  
[  5]   9.00-10.00  sec  52.1 MBytes   437 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   395 MBytes   331 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   391 MBytes   328 Mbits/sec                  receiver

iperf Done.

Here's the patch: https://patchwork.ozlabs.org/patch/932599/

1 Like

Have I understood correctly that

  • 4.14.44 was ok,
  • .48 was already bad,
  • current .50 is bad and
  • forthcoming .51 is again good.

then I Think it was a kernel regression... nothing related to openwrt....

Yep, it sounds like that. I tried identifying the possible fix from the kernel changelog, but did not quickly pick up anything likely, although naturally there are several network related commits.
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/log/?h=linux-4.14.y

(18.06.0-rc1 is planned to be tagged tomorrow, and currently .50 is still there in 18.06. Possibly .51 gets only into rc2 or final. From that perspective it would be really helpful to be certain that .51 really fixes the problem. So hopefully people with speedy connections do test that kernel bump patch.)

1 Like

well i think that kernel should be upgraded in the release just to be sure...

Could be a problem with every 4.14 device...

That's correct.

It would be nice if someone else with a fast connection could confirm my results.

Can confirm. iperf maxes out now.

1 Like

Boo. iperf3 drops by half when connected to WAN.

iperf3 -c openwrt
Connecting to host openwrt, port 5201
[  5] local fd9a:d091:22c5::ddd port 49838 connected to fd9a:d091:22c5::1 port 5201

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  63.2 MBytes   530 Mbits/sec  217    787 KBytes       
[  5]   1.00-2.00   sec  61.1 MBytes   513 Mbits/sec    0    841 KBytes       
[  5]   2.00-3.00   sec  62.3 MBytes   523 Mbits/sec    0    891 KBytes       
0[  5]   3.00-4.00   sec  59.9 MBytes   502 Mbits/sec    0    941 KBytes       
[  5]   4.00-5.00   sec  61.0 MBytes   512 Mbits/sec    0    990 KBytes       
[  5]   5.00-6.00   sec  62.3 MBytes   523 Mbits/sec    0   1.01 MBytes       
[  5]   6.00-7.00   sec  61.1 MBytes   513 Mbits/sec    0   1.05 MBytes       
[  5]   7.00-8.00   sec  61.1 MBytes   512 Mbits/sec    0   1.09 MBytes       
[  5]   8.00-9.00   sec  61.1 MBytes   513 Mbits/sec    0   1.13 MBytes       
[  5]   9.00-10.00  sec  62.4 MBytes   523 Mbits/sec    0   1.17 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   615 MBytes   516 Mbits/sec  217             sender
[  5]   0.00-10.04  sec   613 MBytes   512 Mbits/sec                  receiver

iperf Done.

Full speed if it's just the computer connected. Guess that's the one issue with qca8k.

edit: I could add a USB WAN port. But USB has a lot of buffering...

Just built 18.06 with 4.14.51 patch applied, will test in the next hour or so.

@hnyman in think we should include in the release also led patch and the fix with 160... (if someone select 160mhz the firmware crash, not so good for a stable release!!!)

Led patch got merged in trunk...

Are you running qca8k?

You did not look into 18.06?
Both were backported there right after they got into master.