Netgear R7800 exploration (IPQ8065, QCA9984)

anon98444528 · 2022-02-03T23:29:43.687Z

I've already pulled this build off and gone back to a k510+dsa (pr4036 only).

I did a netperf from a 5g wifi client that is capable of doing 450+ mbps to a linux box attached via wire to the r7500v2 (wan port, using vlan's) and, on the 5.15 pr 4748 build, I could only do 200 mbps at best. One r7500v2 cpu is maxed out and the other is at 50+%. Going back to a k510+dsa build, i regain the netperf rates for this client with much much less cpu usage.

I seem to recall that vlans might be an issue - so perhaps nothing new here.

Unfortunately, I confounded these results by also changing from wpad (full) to wpad-wolfssl (full, pr4748 build), iw to iw-full, aht10k-ct htt-full to ath10k-htt. I don't think these changes would explain the excessive cpu usage with pr 4748.

EDIT: Not to mention that my 5.10 builds are from aug-nov time frame last year - I'll update a 5.10 build with pr 4036 to eliminate any such differences.

Ansuel · 2022-02-04T00:10:22.883Z

@anon98444528 can you test this https://github.com/Ansuel/openwrt/tree/5.15-improve (there is an extra commit)

anon98444528 · 2022-02-04T00:22:16.497Z

i will, but i'm going to do an updated 5.10+dsa build first as a baseline otherwise I'll always be second guessing the results.

May take a few days, as the r7500v2 will be in use for a bit.

Ansuel · 2022-02-04T00:23:30.377Z

If you want to have some fun in both tests, comment ds->pcs_poll = true; from qca8k.c

Anyway from what I observed the slowdown seems to be with the internal phy part.
The mdio patch was introduced to enable the assisted learning for multicpu that require multiple write/read to access the fdb... with this we now do it in one go directly. Probably the mdio is faster for the phy part... We can totally both way

I didn't trust the phy part and it would makes sense if it does cause some slowdown (not 200mbps but should be fixed by the extra patch i posted)
In theory disabling the pcs_pool should give our perf back... at worst we can just disable the eth mgmt for the phy mdio part... the mdio part is now optimized and use one write/read instead of 3 so should be fine...

In fact... @nmrh thx for the bench that's exactly what i needed... As it looks stable now...

tell me if you need some instruction on how to disable eth for phy

anon98444528 · 2022-02-04T00:32:36.349Z

Ansuel:

tell me if you need some instruction on how to disable eth for phy

best tell me what you want. But sleep first, it's got to be late for you and I won't get to this quickly.

Ansuel · 2022-02-04T00:39:03.366Z

Anyway to disable the mdio for phy

in qca8k_internal_mdio_write and qca8k_internal_mdio_read just comment the qca8k_phy_eth_command part and the if.

This way phy will use mdio instead of eth mgmt

quarky · 2022-02-04T00:50:14.975Z

Ansuel:

I'm still convinced in all these years that the krait notifier for the safe parent is implemented in a bad way and in a corner case it can happen that all the mux configuration is wrong causing the cpu to be clocked to an abnormal freq or a too low freq... this with the introduction of the regulator actually working makes the problem even worse as in the old days the regulator was set to the max the cpu could handle abnormal freq spike but now if it's set to the correct voltage and a spike happen 99% the system will crash for not sufficient voltage.

Looks like the CPU core undergoing a frequency change should not be context switching then?

Without a definite way to test this 'bug', it'll be difficult to fix, as the issue doesn't manifest itself often. My R7800 can sometimes goes weeks before rebooting by itself. I always thot. it's caused by the NSS cores, but now it looks like it's due to the change in the Krait core's frequency? With the NSS cores, the Krait CPU will likely be mostly idle (for LAN based traffic anyway) and it would likely be more prone to wild swings of the CPU core frequency. Maybe the frequency change should be in discreet next step (e.g. 600MHz -> 800 MHz -> 1.4GHz -> 1.7 GHz), instead of a big jump, like from 600MHz to 1.7GHz?

You think it's feasible to write a test kernel module to change the CPU core frequency to try simulating this bug?

Ansuel · 2022-02-04T01:04:40.706Z

I think a good way to test this would be hammer the mux change freq logic. Or run 2 thread and make the cpu freq change to random value AT THE SAME TIME and stress this very hard. Nothing incremental...

Also in that scheme i posted, the l2 cache is absent and some comments from qsdk said that the l2 can't run at idle freq when the cpu core is NOT at idle makes me suspect that l2 freq is connected to one of the 2 core muxes... Considering that most of the time the glitch are about memory... wonder if it's the cache that is scaled to an abnormal clk and produce all sort of random operation/data...

Anyway i will check now if that "coordinated clk" series apply so we can check if the clk logic is just wrong and has some flaw.

about the kernel module... in theory should be easy to write one extra point to collect tons of data about the muxes and clk states... And check if they are correct. Also adding some debug values to the krait notifier wouldn't hurt.

quarky · 2022-02-04T01:29:30.410Z

Ansuel:

Also in that scheme i posted, the l2 cache is absent and some comments from qsdk said that the l2 can't run at idle freq when the cpu core is NOT at idle makes me suspect that l2 freq is connected to one of the 2 core muxes... Considering that most of the time the glitch are about memory... wonder if it's the cache that is scaled to an abnormal clk and produce all sort of random operation/data...

The L2 cache is shared between the two Krait cores right? This would be tricky since both cores can operate at different frequencies.

Ansuel · 2022-02-04T01:32:20.164Z

yes... the current cpufreq driver address this and scale based on the max freq across the 2 cores...

Anyway I managed to rebase the coordinate clks series now i just need to understand how to change the krait clk to use it...
the amount of changes done to rockchip looks scary tho...
https://patchwork.kernel.org/project/linux-clk/patch/20190305044936.22267-7-dbasehore@chromium.org/
anyway from

[    0.736660] L2 @ QSB rate. Forcing new rate.
[    0.737277] L2 @ 384000 KHz
[    0.737741] CPU0 @ 800000 KHz
[    0.737767] CPU1 @ QSB rate. Forcing new rate.
[    0.738445] CPU1 @ 384000 KHz

looks like l2 is connected to the cpu0 mux as qsb is set only one time for l2 and cpu0

I mean another way to test this would be just remove all the safe_sel logic and check the crash log produced...
If they are more then we add some debug log to the clk so we can check what it does happen.

the krait-cc driver have lots of info in the comments... the code is a mess btw... very ancient qcom stuff self reviewed/accepted... this thing would be NACK directly today...

quarky · 2022-02-04T02:01:23.849Z

Ansuel:

yes... the current cpufreq driver address this and scale based on the max freq across the 2 cores...

Could we be encountering the edge case where the CPU core clocks got scaled to say 1.7GHz while the L2 cache clock is staying at a lower clock frequecy, and this corrupted the data the the 1.7GHz core is fetching from the L2 cache? Would the solution be as simple as getting the L2 cache clock go up first before setting the CPU core clock?

There were reports that 800MHz min CPU clock is somehow stable. I suppose the L2 cache clock at 800MHz CPU clock is compatible with 1.7GHz CPU clock?

Peacefuleight · 2022-02-04T02:06:26.394Z

On my side with archer c2600, it runs but Everytime I reboot the router, the hardware is reseted. But even after I update my setting with backup , the router is reseted after that too

anon98444528 · 2022-02-04T02:15:50.110Z

are you upgrading from a non-dsa build? If so, you may have to update your configs in /etc/config for the change. Not sure exactly what tho as i've been using dsa for a while now.

EDIT: From a community build for the r7800, i used this to get started with what needed to be changed for DSA config migration. See also the "mini-tutorial for dsa network config." This is from some time ago and likely not entirely relavent for your device. Luci seems to do a good job so perhaps just start with a totally new config - just do the basics first and work your way into it. I find dsa easier than swconfig but i might be... different.

Ansuel · 2022-02-04T02:16:35.112Z

If i'm not wrong we scale l2 first and then core. Anyway fun free time project understand all the mess that is qsb, pll and hfpll... first thing i'm noticing is that we never use pll and only source freq from qsb and hfpll.

I need to find the series that introduced support for all of this to understand the logic of this driver...

Ok first piece of the puzzle...

Due to the design, QSB and the top PLL are always a fixed rate and thus
only support one frequency each.

And from there...

When switching rates we can't leave the CPU clocked by the HFPLL because
we need to turn off the output of the PLL when changing its frequency.
This means we have to switch over to the secondary mux and use one of the
fixed sources. This is why we need something like the safe parent patch.

Peacefuleight · 2022-02-04T02:51:35.300Z

I used a dsa build since a while, 5.10.builds.
And I sysupgraded but did not keep the setting.

anon98444528 · 2022-02-04T02:54:58.042Z

This is different than my experience with the r7500v2 (very similar device to the r7800). Not sure what is happening for you - my best suggestions are try a forum search related to configurations not being retained for you device and just starting a new config and see if very simple changes (like just the host name) persist between reboots.

anon98444528 · 2022-02-04T18:27:01.811Z

Ansuel:

can you test this https://github.com/Ansuel/openwrt/tree/5.15-improve (there is an extra commit)

k510+pr4036 (pcs_poll commented out), 450+mbps netperf, cpu0 80%, cpu1 50%
k515+pr4036+pr4748+cherry pick e972109 5.15-improve (pcs_poll commented out) 200 mbps netperf max, cpu0 100%, cpu1 50+%

not sure i got the workflow right for the k515 build to pick up the patch you indicate - you tell me.

I'll try disabling the "mdio for phy" as you suggest above on the k515 build as described above and test that.

EDIT: k510+pr4036 (pcs_poll not commented out) 450+mbps netperf, cpu0 80%, cpu1 50% i.e. no detectable change with or without pcs_poll.

Ansuel · 2022-02-04T19:32:23.384Z

anon98444528:

I'll try disabling the "mdio for phy" as you suggest above on the k515 build as described above and test that.

Hope this show us some results then o.O

anon98444528 · 2022-02-04T19:36:15.676Z

it's built, just need 15 min when no one is using the AP... latter today I should have an answer

btw in the tests above i do have:

echo 2 > /proc/irq/<irq # for eth0>/smp_affinity
echo 2 > /proc/irq/<adm_dma>/smp_affinity

if that matters, i'll test again without these

Ansuel · 2022-02-04T19:40:40.604Z

anon98444528:

it's built, just need 15 min when no one is using the AP... latter today I should have an answer

sure take your time. But still something is very strange here. We really need to understand exactly what is causing the regression.

250mbps is a lot!