Netgear R7800 exploration (IPQ8065, QCA9984)

Ansuel · 2021-05-10T14:43:53.962Z

if done right ath10k can't ignore a patch that improve 5-10% perf without major changes...
The critical part is doing this in the most standard and clean way.

Think some good bench with real difference would also help in take some attention from the reviewers

notgood · 2021-05-10T14:48:04.795Z

slh:

while ipq8065 does 500 MBit/s, at its limits, but still.

With all due respect, but thats just not true, R7800 most certanly can push 900+ Mbit/s even in pure software mode, without any out-of-openwrt-tree drivers and patches, without any extra compiler optimizations.
Being exploration thread, here are the iperf3 results, made a few minutes ago.
Pumping 5 TCP threads through NAT. Freshly built current 21.02 branch, nothing else:

iperf3 -c 1gbit_server_behind_nat -t 60 -P 5 -R

latest 21.02 branch, 2021-05-10
===============

# all off
[SUM]   0.00-60.00  sec  2.75 GBytes   394 Mbits/sec                  receiver

# flow_offloading
[SUM]   0.00-60.00  sec  3.67 GBytes   525 Mbits/sec                  receiver

# packet_steering
[SUM]   0.00-60.00  sec  3.95 GBytes   565 Mbits/sec                  receiver

# irqbalance
[SUM]   0.00-60.00  sec  5.40 GBytes   773 Mbits/sec                  receiver

# packet_steering + irqbalance
[SUM]   0.00-60.00  sec  5.17 GBytes   740 Mbits/sec                  receiver

# packet_steering + flow_offloading
[SUM]   0.00-60.00  sec  5.42 GBytes   776 Mbits/sec                  receiver

# irqbalance + flow_offloading
[SUM]   0.00-60.00  sec  5.90 GBytes   845 Mbits/sec                  receiver

# irqbalance + flow_offloading + packet_steering
[SUM]   0.00-60.00  sec  6.46 GBytes   924 Mbits/sec                  receiver

Current default 21.02 configuration gives user extremely poor experience, no load distribution between cores at all. Majority of users won't bother to investigate, will complain and switch to something else.

slh:

Eagerly awaiting your patches, quick, I'm holding my breath.

Right, I will gladly make PR, installing and enabling irqbalance for ipq806x and ipq40xx targets. But it won't get accepted, right?

castiel652 · 2021-05-10T14:59:20.708Z

Don't have the result right now and I am working on something else.

slh · 2021-05-10T15:45:20.218Z

notgood:

With all due respect, but thats just not true, R7800 most certanly can push 900+ Mbit/s even in pure software mode, without any out-of-openwrt-tree drivers and patches, without any extra compiler optimizations.

I know that synthetic iperf3 benchmarks can achieve those results, but they don't really agree with real-world usage.

Disclaimer: I do own/ use a nbg6817, which -in performance terms- is close to identical to the r7800, I have irqbalance enabled and also tested (and currently use-) software flow-offloading, but I do see the nbg6817 getting close to the ceiling (attaining 90-100% core load for times) on my 430/220 MBit/s ftth connection (plain ethernet/ dhcp, no PPPoE, no SQM). It's still fine to cope with this speed (as long as I stay away from SQM) - and I do concur with the gospel others with faster internet connections have confirmed that it can do around 500 MBit/s without flow-offloading and maybe up to 650-660 MBit/s with flow-offloading, as that matches my observations. I do not agree that it could do 1 GBit/s line-speed (~931 MBit/s) without NSS offloading, not in the field with NAT, firewalling, routing, concurrent traffic patterns to different targets - and even less once WLAN enters the picture[0] (which would be part of its expected duties).

--
[0] by that I mean WLAN active and used, not benchmarking wirelessly over the air.

notgood · 2021-05-10T16:19:37.118Z

In the spirit of exploration, built and tested current 19.07 branch, shorter test, but still.

iperf3 -c 1gbit_server_behind_nat -t 60 -P 5 -R

latest 19.07 branch, 2021-05-10
===============

# all off
[SUM]   0.00-60.00  sec  4.47 GBytes   640 Mbits/sec                  receiver

# flow offloading
[SUM]   0.00-60.00  sec  6.28 GBytes   899 Mbits/sec                  receiver

So there coming from 19.07 to 21.02 there is like 40% NAT performance drop in default OpenWRT configuration. On more or less flagship router platform. Thats what people see, good for PR, right?

Digging a bit deeper, I'd say the most probable suspect is this commit: https://git.openwrt.org/?p=openwrt/openwrt.git;a=commit;h=d3868f15f876507db54afacdef22a7059011a54e
Packet steering was enabled by default in 19.07 and then disabled, without providing sensible default alternative, even though commit message says "hey, we should probably use irqbalance instead"

ACwifidude · 2021-05-10T20:17:36.451Z

@Ansuel

Using an iphone 7 (80mhz 2x2) and a hardwired computer as the server. No NSS. Made two ath10k builds (with encap offloading and without) and used @hnyman build for the -ct build. All builds had matching driver and firmware (no mixing). Performance governor, irqbalance enabled, and global packet steering enabled. iperf3 settings = 5 streams & 30 seconds. Using @castiel652 encap offloading patch for ath10k. Encap offloading looks good!

ath10k with encap offloading:
[SUM]   0.00-30.01  sec  1.63 GBytes   468 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.50 GBytes   428 Mbits/sec  1717             sender

ath10k
[SUM]   0.00-30.01  sec  1.60 GBytes   459 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.14 GBytes   326 Mbits/sec  699             sender

ath10k-ct
[SUM]   0.00-30.01  sec  1.53 GBytes   437 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.21 GBytes   347 Mbits/sec  763             sender

Ansuel · 2021-05-10T20:19:32.891Z

the ath10k driver is matched with ath10k firmware or the ct variant?

ACwifidude · 2021-05-10T20:26:38.637Z

matched driver and firmware for each run. All runs using the latest encap offloading patch

Here is what it looks like with matched ath10k + NSS (I think I have the old ath10k encap offloading patch on my NSS build so after I rebuild with the newer patch I might get more):

[SUM]   0.00-30.01  sec  1.65 GBytes   473 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.81 GBytes   517 Mbits/sec  784             sender

Ansuel · 2021-05-10T20:27:45.894Z

funny how the nss handling gives more or less 100 mb of extra band.
Did we ever tried if nss core + wifi offload works (i mean without the gmac custom driver?)

Wifi use the virtual interface feature so I wonder if it does actually work without the gmac offload part

(I think nss firmware + nss core + wifi patch can be part of openwrt if they improve the wifi speed with 100mb)

ACwifidude · 2021-05-10T20:35:47.581Z

Ansuel:

Did we ever tried if nss core + wifi offload works (i mean without the gmac custom driver?)

Haven't tried that before. Let me know if you want to simplify or improve the patches to make it more friendly for mainstream integration. I'll gladly tweak and test.

GitHub

GitHub - ACwifidude/openwrt: This repository is a mirror of...

This repository is a mirror of https://git.openwrt.org/openwrt/openwrt.git It is for reference only and is not active for check-ins or for reporting issues. We will continue to accept Pull Request...

ACwifidude · 2021-05-11T00:07:36.998Z

The NSS driver is dependent on the NSS gmac driver. Splitting them them looks interesting. Difficult or doable?

Ansuel · 2021-05-11T00:09:44.517Z

actually nope... the nss core (the drv package) can be loaded without the gmac package... what we need to understand is if it can work without it. (if the gmac part is mandatory or the nss firmware can offload for example only wifi packet)

From the kmod part... they can be loaded separately in fact you load nss-dr first and the nss-gmac withour a problem

A quick test would be remove the gmac driver and check if the wifi offload works (but you need to enable the old gmac binding to make the master driver work)

ACwifidude · 2021-05-11T02:02:11.076Z

I’ll give it a try later this week when I have a second.

quarky · 2021-05-11T04:03:26.303Z

Ansuel:

actually nope... the nss core (the drv package) can be loaded without the gmac package... what we need to understand is if it can work without it. (if the gmac part is mandatory or the nss firmware can offload for example only wifi packet)

The nss-drv needs the nss-gmac driver as nss-gmac has hooks for the nss-drv to tap into. So nss-drv needs nss-gmac. Using the in-tree gmac driver will disable all NSS functionalities, as the nss-drv will not load, IIRC.

open.nya · 2021-05-11T20:25:34.751Z

notgood:

Packet steering was enabled by default in 19.07

This statement is not quite true.

According to this commit:

Remove RPS/XPS support from netifd core, move the logic to a hotplug script that uses a different policy... bla bla bla
https://github.com/openwrt/openwrt/commit/916e33fa1e14b97daf8c9bf07a1b36f9767db679

packet steering was broken in 18.06 and later migrated to 19.07 in the same state.
You can examine the source code of referred script to see the cause of the problem.

As a workaround, many users remove or disable the '/etc/hotplug.d/net/20-smp-tune' script and then add several commands to the init script:

https://forum.openwrt.org/t/r7800-cache-scaling-issue/44187/18

@facboy:

#utilize both cpu cores for network processing
	for file in /sys/class/net/*
	do
			echo 3 > $file"/queues/rx-0/rps_cpus"
			echo 3 > $file"/queues/tx-0/xps_cpus"
	done

Or:
https://bugs.openwrt.org/index.php?do=details&task_id=2573

echo 3 > /sys/class/net/eth0/queues/rx-0/rps_cpus
echo 3 > /sys/class/net/eth0/queues/tx-0/xps_cpus

Feel free to search the forum.
You will find many topics related to the question "How to restore normal (original) operation of RPS / XPS on 19.07".

Ansuel · 2021-05-11T22:34:12.253Z

(self promotion since i like to spam and i should be banned 1??!?!)

[PR] [WIP] ipq806x: request for kernel 5.10 + dsa - For Developers - OpenWrt Forum

Ansuel · 2021-05-11T23:08:55.583Z

@ACwifidude i'm checking the ath10k-ct offload patch... Did you say that it doesn't work ?

ACwifidude · 2021-05-12T01:33:54.394Z

Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500) Community Builds, Projects & Packages

@ACwifidude ct patch doesn't do anything. On line 7 static bool ethernetmode;. ethernetmode is always false so all the check to enable encap offload will fail. This is also why it doesn't break anything. On line 295 SUPPORT_FAST_XMIT should be always set. In txrx.c vif is not assigned to anything. I did some test after I fixed those problems but it ended up crashing the firmware.

This was the breakdown for why it doesn’t work and why it doesn’t cause any issues.

I’m in over my head on fixing it.

Ansuel · 2021-05-12T01:37:07.334Z

When you have time (i think you use the normal ath10k driver)
Can you test the candela tech version with the ath10k plain driver?
(just replace the firmware with this one https://www.candelatech.com/downloads/ath10k-9984-10-4b/ath10k-fw-beta/firmware-5-ct-full-community.bin )

IMHO the candela firmware just doesn't have the feature compiled in the firmware as in my case it just crash the firmware and no traffic is transmitted... Also the patch for ath10k-ct has some error anyway that cause kernel crash on any firmware crash so i would just drop it

by testing if the ath10k-ct firmware works with plain ath10k driver we would test if the feature is actually compiled in and we would understand if the problem is in the driver itself that require some extra """care""" to make this work... I checked the driver code a bit and WOW SO MUCH CHECKS... They for sure slow down all the wifi operation... wonder if by just cutting them we would gain some extra mb of traffic...
From what i can understand the candela driver really try to work in the more clean way even by removing some performance... There is actually the version htt-mgm that I think use some type of offload but the extra check slow down all the operation

castiel652 · 2021-05-12T09:26:40.324Z

I briefly tested ct driver with encap offload with upstream firmware and it works fine so I guess the problem could be that ct special feature can't deal with 802.3 frames.