R7800 performance

hnyman · August 11, 2018, 5:13am

Has anybody tracked down, where the original values come from ? It is likely in Linux kernel, but have the values been tweaked for specific processors? Or just for a generic architecture family?

If the original values have been drafted for basic calculation tasks, it might well be that sporadic irq driven network traffic needs different values like discussed above.

EDIT:
Based on a brief look at the kernel docs, it might be that the default 95% originates from the Linux kernel itself and no CPU-family specific tailoring has been made.

https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt

up_threshold:

This defines what the average CPU usage between the samplings of
'sampling_rate' needs to be for the kernel to make a decision on
whether it should increase the frequency. For example when it is set
to its default value of '95' it means that between the checking
intervals the CPU needs to be on average more than 95% in use to then
decide that the CPU frequency needs to be increased.

Might quite well be that the default for Openwrt should be changed, as the IRQ driven load by network traffic may not generate constant 95% load.

35% sounds maybe a bit low, though.

EDIT2:
Looks like the default values are hard-coded to Linux. no config options:

This line https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/cpufreq/cpufreq_ondemand.c?h=linux-4.14.y#n372

dbs_data->up_threshold = MICRO_FREQUENCY_UP_THRESHOLD;

simply sets the value set in https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/cpufreq/cpufreq_ondemand.c?h=linux-4.14.y#n27

#define MICRO_FREQUENCY_UP_THRESHOLD (95)

Same for sampling_down_factor.

hnyman · August 11, 2018, 6:30am

One other thing is the division of IRQs on CPU cores. By default all IRQs get assigned to core0, so the load is not split evenly. You might look into R7800 IRQ discussion in R7800 exploration thread, and e.g. to irqbalance

(also manually doing IRQ splitting to cores is possible.)

tolga9009 · October 13, 2018, 6:30am

(also manually doing IRQ splitting to cores is possible.)

I found pinning eth0 / eth1 on their respective CPUs to be a crucial step. It gave me better performance in bufferbloat tests, opposed to running irqbalance without any configuration.

I've used this doc as a guideline: https://digitalpubliclibraryofamerica.atlassian.net/wiki/spaces/TECH/pages/24510493/Using+irqbalance+and+SMP+IRQ+affinity

However, I had issues with irqbalance in the past (like 8 - 10 months ago), leading to spontaneous device reboots. Was running it on my NBG6817 though, not on my R7800. Did anyone else run into it here?

treefiddy · November 4, 2018, 12:46pm

This setting does not help with my WAN - LAN speeds. I'm only getting 300mbps maximum with the latest version of hymnan. Any other solution? Thanks

Furthermore in earlier versions of hymnan the setting Tweaked settings for ondemand scheduler worked but after a while the device freezed and rebooted:
echo 35 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

enri · November 8, 2018, 1:23am

Have you turn on Flow Offloading in Firewall page? I recently set up my R7800 by resetting all the settings, change only the Flow Offloading and the 2 line u mentioned.

with no SQM
My own Lan test peak at 700Mbps at dslreports
Comcast's XMT device Lan test reach 960Mbps

treefiddy · November 8, 2018, 3:26pm

Thanks for the hint.

The following are the standard settings:

ohvo15Z.png878×387

With this settings i am only getting 250-300mbps:

As soon as i activate software and / or hardware flow offloading the speed is much better (maximum @430mbps)

uShbQG6.png820×425

Speed:

Which one is the best setting? I suppose hardware offloading, or? Thanks a lot for your hint!

I also tried other speed tests and direct downloads, the result is the same as seen in the speed test above (measuring network speed)

//Edit:

hnyman just said that only software flow offloading is available for the r7800

treefiddy · November 11, 2018, 10:51am

Thanks again for your hint but somehow after a lot of traffic load (Steam Downloads) and a lot of connections the R7800 crashed and restarted twice. I had the same issue after altering the up_trhreshold and sampling_down_factor after a while (2-3 days). Without altering this settings i never had the issue I will try to reproduce the issue and gather some logs.

Anyone else with an idea how to fix this?

enri · November 11, 2018, 10:57am

single stream is always around 400-450 for me
using dslreport speed test can reach more than 700 at least

i also got crash, i just updated his latest no ct ver of firmware, i think using the old non ct should be the fix for crashing (i am not entirely sure but seems yes)

treefiddy · November 11, 2018, 6:20pm

The version master-r8447-1b4b942bce-20181110-ath10k crashed again today... i will try now master-r8410-900005ee75-20181104-c

What about the master version? does it use ct or ath10k?

locojohn · November 11, 2018, 9:28pm

It uses ct by default, but it's really easy to switch with 'make menuconfig'. I wonder why is it being an issue for you? I use yesterday's master with ath10k firmware driver and everything is rock solid and the speeds on 802.11ac are 850 mbit/s on average.

treefiddy · November 12, 2018, 8:46am

not sure but it seems like enri experienced the same issue. What is your line speed? Maybe this is related to a higher bandwidth / more traffic on WAN?

Do you know how i can trace some log files when the error occurs?

hnyman · November 12, 2018, 9:02am

With gigabit speeds, you should likely forget about CPU frequency scaling "ondemand", and simply set the "performance" scaling governor.

Your crashes may be due to something going wrong with the scaling.

In general, there not been reports of R7800 crashing, so your case(s) seem rather unique, so far.

And you should try to separate the wifi part vs. generic router fixed line part. ath10k-ct or old ath10k have impact just on wifi. The interesting part si, if the crashes are due to the base system or the wifi firmware/driver.

treefiddy · November 12, 2018, 10:35am

Thanks for your attention. But as a beginner i don't know how to troubleshoot this / gather logs. After the device crashes i have to turn it off and after the next start the systemlogs are gone... How do you do it to save logs?

Furthermore i did a complete factory reset and installed master-r8410-900005ee75-20181104-c. I'm using Adblock, DDNS, Dual Stack (ipv4 and ipv6) with software flow offloading enabled but without changing the ondemand scheduler. I will try to reproduce the crash today.

treefiddy · November 13, 2018, 4:09pm

Okay, it crashed again now. But there is nothing specific i could find in the log files at the date and time the device crashed. Only something else from this night which does not seem to be related. But maybe someone could take a look? Thanks again and sorry for being annyoing

Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.606952] print_req_error: I/O error, dev mtdblock0, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.607647] print_req_error: I/O error, dev mtdblock0, sector 8
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.612338] print_req_error: I/O error, dev mtdblock0, sector 16
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.618362] print_req_error: I/O error, dev mtdblock0, sector 24
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.633682] print_req_error: I/O error, dev mtdblock0, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.633703] Buffer I/O error on dev mtdblock0, logical block 0, async page read
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.718530] print_req_error: I/O error, dev mtdblock0, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.718556] Buffer I/O error on dev mtdblock0, logical block 0, async page read
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.723808] print_req_error: I/O error, dev mtdblock1, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.730956] print_req_error: I/O error, dev mtdblock1, sector 8
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.736929] print_req_error: I/O error, dev mtdblock1, sector 16
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.742758] print_req_error: I/O error, dev mtdblock1, sector 24
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.753909] Buffer I/O error on dev mtdblock1, logical block 0, async page read
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.794426] Buffer I/O error on dev mtdblock1, logical block 0, async page read

Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.355073] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
Mon Nov 12 00:45:56 2018 kern.warn kernel: [   26.527894] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
Mon Nov 12 00:45:56 2018 kern.warn kernel: [   26.527934] ath10k_pci 0001:01:00.0: Falling back to user helper
Mon Nov 12 00:45:56 2018 kern.err kernel: [   26.743892] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.744216] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.751760] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.764625] ath10k_pci 0001:01:00.0: firmware ver 10.4-3.5.3-00053 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps crc32 4c56a386
Mon Nov 12 00:45:56 2018 kern.info kernel: [   29.038913] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 dd6d039c
Mon Nov 12 00:45:56 2018 kern.info kernel: [   34.948468] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 512 raw 0 hwcrypto 1
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038016] ath: EEPROM regdomain: 0x0
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038031] ath: EEPROM indicates default country code should be used
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038041] ath: doing EEPROM country->regdmn map search
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038060] ath: country maps to regdmn code: 0x3a
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038073] ath: Country alpha2 being used: US
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038084] ath: Regpair used: 0x3a

[   17.488180] NET: Registered protocol family 24
[   17.492263] PPTP driver version 0.8.5
[   17.510314] ath10k_pci 0000:01:00.0: assign IRQ: got 67
[   17.510917] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
[   17.510996] ath10k_pci 0000:01:00.0: enabling bus mastering
[   17.511443] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   17.691347] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
[   17.691392] ath10k_pci 0000:01:00.0: Falling back to user helper
[   17.720674] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
[   18.078298] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[   18.078349] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[   18.091880] ath10k_pci 0000:01:00.0: firmware ver 10.4-3.5.3-00053 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps crc32 4c56a386
[   20.366121] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 dd6d039c
[   26.254157] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 512 raw 0 hwcrypto 1
[   26.347933] ath: EEPROM regdomain: 0x0
[   26.347948] ath: EEPROM indicates default country code should be used
[   26.347959] ath: doing EEPROM country->regdmn map search
[   26.347977] ath: country maps to regdmn code: 0x3a
[   26.347991] ath: Country alpha2 being used: US
[   26.348002] ath: Regpair used: 0x3a
[   26.353226] ath10k_pci 0001:01:00.0: assign IRQ: got 100
[   26.354259] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
[   26.354406] ath10k_pci 0001:01:00.0: enabling bus mastering
[   26.355073] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   26.527894] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
[   26.527934] ath10k_pci 0001:01:00.0: Falling back to user helper
[   26.743892] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
[   26.744216] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[   26.751760] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[   26.764625] ath10k_pci 0001:01:00.0: firmware ver 10.4-3.5.3-00053 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps crc32 4c56a386
[   29.038913] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 dd6d039c
[   34.948468] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 512 raw 0 hwcrypto 1
[   35.038016] ath: EEPROM regdomain: 0x0
[   35.038031] ath: EEPROM indicates default country code should be used
[   35.038041] ath: doing EEPROM country->regdmn map search
[   35.038060] ath: country maps to regdmn code: 0x3a
[   35.038073] ath: Country alpha2 being used: US
[   35.038084] ath: Regpair used: 0x3a
[   35.046985] kmodloader: done loading kernel modules from /etc/modules.d/*
[   39.360354] print_req_error: 14 callbacks suppressed
[   39.360361] print_req_error: I/O error, dev mtdblock0, sector 0
[   39.365157] print_req_error: I/O error, dev mtdblock0, sector 8
[   39.370565] print_req_error: I/O error, dev mtdblock0, sector 16
[   39.376509] print_req_error: I/O error, dev mtdblock0, sector 24
[   39.383225] print_req_error: I/O error, dev mtdblock0, sector 0
[   39.388182] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   39.399839] print_req_error: I/O error, dev mtdblock0, sector 0
[   39.401113] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   39.409406] print_req_error: I/O error, dev mtdblock1, sector 0
[   39.414918] print_req_error: I/O error, dev mtdblock1, sector 8
[   39.420708] print_req_error: I/O error, dev mtdblock1, sector 16
[   39.426678] print_req_error: I/O error, dev mtdblock1, sector 24
[   39.433121] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   39.441850] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   40.434402] Generic PHY fixed-0:01: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:01, irq=POLL)
[   40.435384] dwmac1000: Master AXI performs any burst length
[   40.443307] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
[   40.448944] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
[   40.457941] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   40.469290] br-lan: port 1(eth1.1) entered blocking state
[   40.470466] br-lan: port 1(eth1.1) entered disabled state
[   40.476309] device eth1.1 entered promiscuous mode
[   40.481315] device eth1 entered promiscuous mode
[   40.488717] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   40.507627] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[   40.508599] dwmac1000: Master AXI performs any burst length
[   40.516729] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[   40.522073] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
[   40.531627] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   40.541388] IPv6: ADDRCONF(NETDEV_UP): eth0.2: link is not ready
[   41.513944] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
[   41.529860] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   41.530200] br-lan: port 1(eth1.1) entered blocking state
[   41.535027] br-lan: port 1(eth1.1) entered forwarding state
[   41.577173] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   41.593907] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   41.593970] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   41.614916] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.2: link becomes ready
[   48.338796] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready
[   54.450130] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   54.455951] br-lan: port 2(wlan1) entered blocking state
[   54.456002] br-lan: port 2(wlan1) entered disabled state
[   54.460992] device wlan1 entered promiscuous mode
[   54.476939] br-lan: port 3(wlan0) entered blocking state
[   54.476965] br-lan: port 3(wlan0) entered disabled state
[   54.481487] device wlan0 entered promiscuous mode
[   54.852974] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   54.853239] br-lan: port 2(wlan1) entered blocking state
[   54.858540] br-lan: port 2(wlan1) entered forwarding state
[   55.234089] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   55.234420] br-lan: port 3(wlan0) entered blocking state
[   55.239561] br-lan: port 3(wlan0) entered forwarding state

Update:

The current version is stable since 4-5 days, i will not change anything now

Firmware Version	OpenWrt SNAPSHOT r8450-a95bef0579 / LuCI Master (git-18.316.62517-99a696d)
Kernel Version	4.14.79
Uptime	4d 15h 39m 35s

locojohn · November 17, 2018, 2:47pm

So I am running build OpenWrt-r8463-59ff8687c7 with irqbalance in /etc/rc.local, but the processor statistics is showing that most tasks are still processed by the first (default?) core:

OpenWrt%20-%20Processor%20-%20LuCI%202018-11-17%2016-46-55

Is there something wrong?

cbz · December 1, 2018, 10:48am

What does /proc/interrupts look like in that case? Are the interrupts still being handled by one core?

locojohn · December 1, 2018, 2:15pm

           CPU0       CPU1
 16:      41612     111819     GIC-0  18 Edge      gp_timer
 18:         33          0     GIC-0  51 Edge      qcom_rpm_ack
 19:          0          0     GIC-0  53 Edge      qcom_rpm_err
 20:          0          0     GIC-0  54 Edge      qcom_rpm_wakeup
 26:          0          0     GIC-0 241 Edge      ahci[29000000.sata]
 27:          0          0     GIC-0 210 Edge      tsens_interrupt
 28:     180811      18578     GIC-0  67 Edge      qcom-pcie-msi
 29:     188100      85549     GIC-0  89 Edge      qcom-pcie-msi
 30:     205667         25     GIC-0 202 Edge      adm_dma
 31:       3798      11330     GIC-0 255 Level     eth0
 32:        131         26     GIC-0 258 Level     eth1
 33:          0          0     GIC-0 130 Level     bam_dma
 34:          0          0     GIC-0 128 Level     bam_dma
 35:          0          0   PCI-MSI   0 Edge      aerdrv
 36:     180811      18578   PCI-MSI   1 Edge      ath10k_pci
 68:          0          0   PCI-MSI   0 Edge      aerdrv
 69:     188100      85549   PCI-MSI   1 Edge      ath10k_pci
101:         10          0     GIC-0 184 Level     msm_serial0
102:          2          0   msmgpio   6 Edge      gpio-keys
103:          2          0   msmgpio  54 Edge      gpio-keys
104:          2          0   msmgpio  65 Edge      gpio-keys
105:          0          0     GIC-0 142 Level     xhci-hcd:usb1
106:          0          0     GIC-0 237 Level     xhci-hcd:usb3
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:      35221      75618  Rescheduling interrupts
IPI3:         36      20041  Function call interrupts
IPI4:          0          0  CPU stop interrupts
IPI5:      49216      85866  IRQ work interrupts
IPI6:          0          0  completion interrupts
Err:          0

cbz · December 1, 2018, 8:04pm

A cursory conclusion would be that the irqs are still not being balanced properly, with core #0 receiving the bulk of interrupts from the devices that are most active.

I am in the middle of debugging a similar situation myself.

locojohn · December 1, 2018, 10:46pm

It would seem so. Hopefully, someone will provide a fix soon.

cbz · December 3, 2018, 11:58am

It's hard to argue with the raw figures you have in your rrd graphs - but I wonder whether a lot of this isn't architecture specific.

Look at a couple of ARM based boards I'm seeing similar imbalances in the IRQs raised - where most of the device IRQs are raised against a single core - however in those cases it looks like at least some of the work is being rescheduled later (presumably via the rescheduling interrupts).

I suppose the test would be to attempt to actually max out one of the cores - and make sure the load was redistributed properly at that point - certainly it looks like at low levels there's a supervisory function assumed by the first core.