R7800 performance

DeadEnd · 2020-07-26T04:04:19.529Z

Can anyone tell me where to check to figure out why my R7800 restarts randomly? For the third time it has restarted at the most inopportune time (conference call, middle of game, etc.). I am running hnyman's build - trying different "tuning" that people post, but the same thing always happens. I have looked in /var/log and found nothing. I looked at the kernel and system logs in the GUI and found nothing. Where do I go to see what is causing these crashes? Anyone know?

Thanks!
DeadEnd

hnyman · 2020-07-26T07:10:01.558Z

DeadEnd:

. Where do I go to see what is causing these crashes? Anyone know?

Serial console output during the actual crash. To prevent flash wear, logs are written to ram, so they do not survive a reboot.

P.s. One known/suspected reason for occasional crashes has been jumbo frames from other devices. There was apparently a bug in the kernel driver for fixed ethernet, but that should have been fixed before kernel 5.4.

DeadEnd · 2020-07-27T02:22:22.458Z

Okay, so not really an easy way to figure it out...
I just update from r13628-870588b6eb-20200625 to r13881-bae4204e34-20200718.
I'll see if my some miracle that fixes anything.

I already made the adjustment of adding

echo 800000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq
echo 800000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq

to the local startup commands as I had read that it fixes some issues with crashing.
It did greatly reduce the amount of transitions the CPU's were doing.
I thought that had fixed it, but the random crashes still persist.
They next thing I might try is changing the governor to performance - this basically just runs at full clock speed correct?

If I get another crash, that'll be the next time.

Thanks for the help!

fantom-x · 2020-07-27T02:31:09.133Z

Could this help?

Crashlog retrieval (MIPS) For Developers

It is supposed to end up in /sys/kernel/debug/crashlog after a crash triggering a reboot. Whether it actually works in practise mostly depends on the bootloader and some other circumstances. For the crash dump to be recoverable, the RAM allocated for it must not be cleared by the bootloader or the kernel after a reset.

hnyman · 2020-07-27T05:29:32.406Z

No.
The crashlog feature is disabled for most common router processor architectures.

github.com

openwrt/openwrt/blob/caf09f2b842d31b70815e1281098bf895b067714/config/Config-kernel.in#L31


	  Sets the Kernel build domain string, which for example will be
	  returned by 'uname -a' on running systems.
	  If not set, uses system hostname at build time.

config KERNEL_PRINTK
	bool "Enable support for printk"
	default y

config KERNEL_CRASHLOG
	bool "Crash logging"
	depends on !(arm || powerpc || sparc || TARGET_uml || i386 || x86_64)
	default y

config KERNEL_SWAP
	bool "Support for paging of anonymous memory (swap)"
	default y if !SMALL_FLASH

config KERNEL_DEBUG_FS
	bool "Compile the kernel with debug filesystem enabled"
	default y
	help

To my knowledge, it only works for some MIPS devices in practice.

hnyman · 2020-07-28T16:40:48.954Z

Think commit in ndb's stanging repo sounds interesting:

https://git.openwrt.org/?p=openwrt/staging/nbd.git;a=commitdiff;h=4eca0cc9acd3a9e56a8a418142b62603464e1a59

kernel: add patch that adds support for running threaded NAPI poll functions

This is helps on workloads with CPU intensive poll functions (e.g. 802.11) on multicore systems

For some drivers (especially 802.11 drivers), doing a lot of work in the NAPI poll function does not perform well. Since NAPI poll is bound to the CPU it was scheduled from, we can easily end up with a few very busy CPUs spending most of their time in softirq/ksoftirqd and some idle ones.

Introduce threaded NAPI for such drivers based on a workqueue. The API is the same except for using netif_threaded_napi_add instead of netif_napi_add.

In my tests with mt76 on MT7621 using threaded NAPI + a thread for tx scheduling improves LAN->WLAN bridging throughput by 10-50%. Throughput without threaded NAPI is wildly inconsistent, depending on the CPU that runs the tx scheduling thread.

With threaded NAPI it seems stable and consistent (and higher than the best results I got without it).

(I haven't tested it yet.)

neheb · 2020-07-30T00:47:48.980Z

Sounds like it helps mostly with mt76.

xeonpj · 2020-07-31T10:42:22.879Z

Any summary of recommended settings? i currently have 600/600 mb wan which is not able to manage. Use 19.07 (kong 2020-01) kernel 4.14.179.

I don't have anything in rc.local, what do you recommend to put?
Thank you.

atiensivu · 2020-08-01T00:16:37.353Z

I suspect this patch might just make the firmware crash faster if there are any undiagnosed race conditions hanging around. I'd love to see if it makes a difference with the R7800 though.

DeadEnd · 2020-08-05T13:28:40.777Z

DeadEnd:

r13881-bae4204e34-20200718

Well so far 9.5 days on the above and no crashes.
Whatever was going on causing the crashes seems to be gone now.
I also found this and put it in my local startup - that might have helped... not sure:

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

# min scaling frequency: set to 800MHz because of L2 cache issues
echo 800000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq
echo 800000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq

sleep 1

# ondemand governor
echo ondemand > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo ondemand > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
echo 100000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate
echo 75 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

# schedutil governor
#echo schedutil > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
#echo schedutil > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
#echo 100000 > /sys/devices/system/cpu/cpufreq/schedutil/rate_limit_us

# performance governor
#echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
#echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor

# Reset stats
echo 1 > /sys/devices/system/cpu/cpufreq/policy0/stats/reset
echo 1 > /sys/devices/system/cpu/cpufreq/policy1/stats/reset

exit 0

I believe ondemand was the default anyway, but when I checked the sampling was much much lower (if I remember correctly) so maybe these tweaks helped too? If anyone thinks one of these settings is wrong or could be improved, I'm always happy to hear it!

Thanks as always for the community support!
DeadEnd

ACwifidude · 2020-08-06T01:59:22.622Z

One thing to consider- Using 800mhz as your minimum frequency you’ll probably never see a frequency increase with an up threshold of 75. This will hurt performance for max 5ghz wifi speeds or as you approach near gig wired speeds.

Most people will use a more aggressive up threshold with 800mhz as their minimum. 20-35 is pretty popular. Personally I use the more aggressive end (90%+ of the time the router is at 800mhz, only a fraction of the time it ramps up to max frequency). This is what my startup settings are:


echo 800000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq 
echo 800000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
echo 20 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold 
echo 60 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
echo 1000000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate

This what the time at each frequency looks like with these settings:


root@OpenWrt:~# cat /sys/devices/system/cpu/cpufreq/policy*/stats/time_in_state
384000 1138
600000 116
800000 26059202
1000000 237
1400000 177
1725000 64726
384000 2191
600000 111
800000 26078669
1000000 131
1400000 89
1725000 44405

wired · 2020-08-06T02:10:58.293Z

I posted those settings so I can comment on them based on my use case and observations

up_threshold 75 works quite well if you want to use pretty much all frequencies between 800-1.7 and have a gradual ramp up. I use mine in AP mode, most services turned off and it does get to 1.7 when laptops back up over wifi.

If you go aggressive 20-35 then you will just be jumping from 800 pretty much to 1.7, you don't use 1 or 1.4, because up_threshold is the load to go all the way up to max, not to the next frequency. At least in my case when I tried aggressive settings I never saw 1.4 being used. Not that it really matters, because ...

I use the performance governor, locked at 1.7, I don't see much benefit in transitioning between frequencies a lot. This is not a phone, you're not trying to save battery.

FWIW, I noticed schedutil to not be aggressive at all, with 800 min freq, I rarely see moves up to higher frequencies.

DeadEnd · 2020-08-06T04:00:57.665Z

Router has been stabling running for over 10 days - so that makes me happy .
Here are the charts for the last week with the 75/10 up threshold/sampling down factor:

image936×593 10.9 KB

image934×594 10.2 KB

So yes, I rarely go up past the 800Mhz - but my WAN is only 30/5 - so there probably isn't a ton of stress on the system anyway . Since WAN speed is probably a non-issue I'm more happy with stability and WiFi between devices. Again though, I would be happy to test other settings to see how they work - but as of now, its not broke, lol.

Cheers!
Thanks for the feedback!
DeadEnd

update: past 15 days now... so I have a good feeling whatever was causing the restart every 4-6 days is now fixed. Very happy as like many others, Work from Home is the new normal and I needed this to be reliable.

Cheers!

excOpen · 2020-08-15T15:29:27.447Z

I didn't see your Exit 0. I'm assuming you also have...

echo 2 > /proc/irq/31/smp_affinity
echo 2 > /proc/irq/32/smp_affinity

... in there somewhere for the irq "balancing"?

wired · 2020-08-15T15:57:22.332Z

Yeah, mine is more complex, I do other things in rc.local, I only posted the governor section.

xeonpj · 2020-08-17T12:00:37.766Z

can someone pass a good configuration? I do not reach 600mbits in WAN.

You put configurations but none are complete, I need to improve performance, thanks!

Is KONG images recommendable?

ACwifidude · 2020-08-17T14:47:55.101Z


echo 800000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq; echo 800000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq; echo 20 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold; echo 60 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor; echo 1000000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate

Run that in the terminal to try out a combo of settings similar to Kong (using the ondemand governor). Was able to get me high 700 / low 800 mbps with software offloading enabled in the firewall. Sets your base frequency to 800mhz, ramps up quickly, and lessens the frequency that it checks to change frequency.

This is what my /etc/rc.local files looks like (same settings, feel free to tweak to the settings you are most comfortable with)


root@OpenWrt:~# cat /etc/rc.local
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

echo 800000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
echo 800000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
echo 20 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 60 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
echo 1000000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate

exit 0

xeonpj · 2020-08-17T15:51:42.626Z

thank you. Also I have installed IRQ balance and put this configuration that I have seen out there, I do not know if it works.

IRQ balance

echo 2 > /proc/irq/31/smp_affinity
echo 2 > /proc/irq/32/smp_affinity

Even so, with those settings, it did not exceed 400mbits approx. I use SQM fq_codel with configuration to 600000 in both up / down channels.

if I deactivate SQM if I get to about 550mbits, approx. I have activated the download of software in firewall.

How do you see those statistics that the partner shows?

ACwifidude · 2020-08-17T17:05:05.445Z

Moving interrupts to different CPUs provides little to no benefit for max throughput with recent builds - the spread of interrupts is reasonably balanced enough recently that it won’t give tremendous gains for most setups (there are always exceptions to this). I’d test your router with and without the settings, you’ll likely see little to no benefit. I use irqbalance too- I like it.

To test max throughput I’d use speedtest.net (not dsl reports). Dslreports speedtest has limited bandwidth from its limited servers.

Wired speedtest.net with a gig connection (try 2-3 different nearby servers) should hit upper 600’s - lower 800 mbps with more aggressive CPU settings, no SQM, and software offloading. Post your results.

xeonpj · 2020-08-17T19:10:56.815Z

results with this in local.rc, firewall software on and sqm disabled

# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

# min scaling frequency: set to 800MHz because of L2 cache issues
echo 800000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq
echo 800000 > /sys/devices/system/cpu/cpufreq/policy1/scaling_min_freq

sleep 1

# ondemand governor
echo ondemand > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
echo ondemand > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor
echo 20 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold 
echo 60 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
echo 1000000 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_rate

exit 0

sqm cake

sqm default

with sqm off everything seems to be going much better. I have compilation from January, can a new one improve results?

increase performance with sqm is it possible?

thanks