R7800 performance


#21

Okay, it crashed again now. But there is nothing specific i could find in the log files at the date and time the device crashed. Only something else from this night which does not seem to be related. But maybe someone could take a look? Thanks again and sorry for being annyoing :frowning:

Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.606952] print_req_error: I/O error, dev mtdblock0, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.607647] print_req_error: I/O error, dev mtdblock0, sector 8
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.612338] print_req_error: I/O error, dev mtdblock0, sector 16
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.618362] print_req_error: I/O error, dev mtdblock0, sector 24
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.633682] print_req_error: I/O error, dev mtdblock0, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.633703] Buffer I/O error on dev mtdblock0, logical block 0, async page read
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.718530] print_req_error: I/O error, dev mtdblock0, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.718556] Buffer I/O error on dev mtdblock0, logical block 0, async page read
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.723808] print_req_error: I/O error, dev mtdblock1, sector 0
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.730956] print_req_error: I/O error, dev mtdblock1, sector 8
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.736929] print_req_error: I/O error, dev mtdblock1, sector 16
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.742758] print_req_error: I/O error, dev mtdblock1, sector 24
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.753909] Buffer I/O error on dev mtdblock1, logical block 0, async page read
Mon Nov 12 00:45:56 2018 kern.err kernel: [   13.794426] Buffer I/O error on dev mtdblock1, logical block 0, async page read

Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.355073] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
Mon Nov 12 00:45:56 2018 kern.warn kernel: [   26.527894] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
Mon Nov 12 00:45:56 2018 kern.warn kernel: [   26.527934] ath10k_pci 0001:01:00.0: Falling back to user helper
Mon Nov 12 00:45:56 2018 kern.err kernel: [   26.743892] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.744216] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.751760] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
Mon Nov 12 00:45:56 2018 kern.info kernel: [   26.764625] ath10k_pci 0001:01:00.0: firmware ver 10.4-3.5.3-00053 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps crc32 4c56a386
Mon Nov 12 00:45:56 2018 kern.info kernel: [   29.038913] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 dd6d039c
Mon Nov 12 00:45:56 2018 kern.info kernel: [   34.948468] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 512 raw 0 hwcrypto 1
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038016] ath: EEPROM regdomain: 0x0
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038031] ath: EEPROM indicates default country code should be used
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038041] ath: doing EEPROM country->regdmn map search
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038060] ath: country maps to regdmn code: 0x3a
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038073] ath: Country alpha2 being used: US
Mon Nov 12 00:45:56 2018 kern.debug kernel: [   35.038084] ath: Regpair used: 0x3a

[   17.488180] NET: Registered protocol family 24
[   17.492263] PPTP driver version 0.8.5
[   17.510314] ath10k_pci 0000:01:00.0: assign IRQ: got 67
[   17.510917] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142)
[   17.510996] ath10k_pci 0000:01:00.0: enabling bus mastering
[   17.511443] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   17.691347] ath10k_pci 0000:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
[   17.691392] ath10k_pci 0000:01:00.0: Falling back to user helper
[   17.720674] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
[   18.078298] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[   18.078349] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[   18.091880] ath10k_pci 0000:01:00.0: firmware ver 10.4-3.5.3-00053 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps crc32 4c56a386
[   20.366121] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 dd6d039c
[   26.254157] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 512 raw 0 hwcrypto 1
[   26.347933] ath: EEPROM regdomain: 0x0
[   26.347948] ath: EEPROM indicates default country code should be used
[   26.347959] ath: doing EEPROM country->regdmn map search
[   26.347977] ath: country maps to regdmn code: 0x3a
[   26.347991] ath: Country alpha2 being used: US
[   26.348002] ath: Regpair used: 0x3a
[   26.353226] ath10k_pci 0001:01:00.0: assign IRQ: got 100
[   26.354259] ath10k_pci 0001:01:00.0: enabling device (0140 -> 0142)
[   26.354406] ath10k_pci 0001:01:00.0: enabling bus mastering
[   26.355073] ath10k_pci 0001:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[   26.527894] ath10k_pci 0001:01:00.0: Direct firmware load for ath10k/QCA9984/hw1.0/firmware-6.bin failed with error -2
[   26.527934] ath10k_pci 0001:01:00.0: Falling back to user helper
[   26.743892] firmware ath10k!QCA9984!hw1.0!firmware-6.bin: firmware_loading_store: map pages failed
[   26.744216] ath10k_pci 0001:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe
[   26.751760] ath10k_pci 0001:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 1
[   26.764625] ath10k_pci 0001:01:00.0: firmware ver 10.4-3.5.3-00053 api 5 features no-p2p,mfp,peer-flow-ctrl,btcoex-param,allows-mesh-bcast,no-ps crc32 4c56a386
[   29.038913] ath10k_pci 0001:01:00.0: board_file api 2 bmi_id 0:2 crc32 dd6d039c
[   34.948468] ath10k_pci 0001:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 512 raw 0 hwcrypto 1
[   35.038016] ath: EEPROM regdomain: 0x0
[   35.038031] ath: EEPROM indicates default country code should be used
[   35.038041] ath: doing EEPROM country->regdmn map search
[   35.038060] ath: country maps to regdmn code: 0x3a
[   35.038073] ath: Country alpha2 being used: US
[   35.038084] ath: Regpair used: 0x3a
[   35.046985] kmodloader: done loading kernel modules from /etc/modules.d/*
[   39.360354] print_req_error: 14 callbacks suppressed
[   39.360361] print_req_error: I/O error, dev mtdblock0, sector 0
[   39.365157] print_req_error: I/O error, dev mtdblock0, sector 8
[   39.370565] print_req_error: I/O error, dev mtdblock0, sector 16
[   39.376509] print_req_error: I/O error, dev mtdblock0, sector 24
[   39.383225] print_req_error: I/O error, dev mtdblock0, sector 0
[   39.388182] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   39.399839] print_req_error: I/O error, dev mtdblock0, sector 0
[   39.401113] Buffer I/O error on dev mtdblock0, logical block 0, async page read
[   39.409406] print_req_error: I/O error, dev mtdblock1, sector 0
[   39.414918] print_req_error: I/O error, dev mtdblock1, sector 8
[   39.420708] print_req_error: I/O error, dev mtdblock1, sector 16
[   39.426678] print_req_error: I/O error, dev mtdblock1, sector 24
[   39.433121] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   39.441850] Buffer I/O error on dev mtdblock1, logical block 0, async page read
[   40.434402] Generic PHY fixed-0:01: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:01, irq=POLL)
[   40.435384] dwmac1000: Master AXI performs any burst length
[   40.443307] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
[   40.448944] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
[   40.457941] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[   40.469290] br-lan: port 1(eth1.1) entered blocking state
[   40.470466] br-lan: port 1(eth1.1) entered disabled state
[   40.476309] device eth1.1 entered promiscuous mode
[   40.481315] device eth1 entered promiscuous mode
[   40.488717] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   40.507627] Generic PHY fixed-0:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=fixed-0:00, irq=POLL)
[   40.508599] dwmac1000: Master AXI performs any burst length
[   40.516729] ipq806x-gmac-dwmac 37200000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[   40.522073] ipq806x-gmac-dwmac 37200000.ethernet eth0: registered PTP clock
[   40.531627] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   40.541388] IPv6: ADDRCONF(NETDEV_UP): eth0.2: link is not ready
[   41.513944] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
[   41.529860] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   41.530200] br-lan: port 1(eth1.1) entered blocking state
[   41.535027] br-lan: port 1(eth1.1) entered forwarding state
[   41.577173] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   41.593907] ipq806x-gmac-dwmac 37200000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[   41.593970] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   41.614916] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.2: link becomes ready
[   48.338796] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready
[   54.450130] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   54.455951] br-lan: port 2(wlan1) entered blocking state
[   54.456002] br-lan: port 2(wlan1) entered disabled state
[   54.460992] device wlan1 entered promiscuous mode
[   54.476939] br-lan: port 3(wlan0) entered blocking state
[   54.476965] br-lan: port 3(wlan0) entered disabled state
[   54.481487] device wlan0 entered promiscuous mode
[   54.852974] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   54.853239] br-lan: port 2(wlan1) entered blocking state
[   54.858540] br-lan: port 2(wlan1) entered forwarding state
[   55.234089] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   55.234420] br-lan: port 3(wlan0) entered blocking state
[   55.239561] br-lan: port 3(wlan0) entered forwarding state

Update:

The current version is stable since 4-5 days, i will not change anything now :slight_smile:

Firmware Version OpenWrt SNAPSHOT r8450-a95bef0579 / LuCI Master (git-18.316.62517-99a696d)
Kernel Version 4.14.79
Uptime 4d 15h 39m 35s

Need a best gaming router
#22

So I am running build OpenWrt-r8463-59ff8687c7 with irqbalance in /etc/rc.local, but the processor statistics is showing that most tasks are still processed by the first (default?) core:

OpenWrt%20-%20Processor%20-%20LuCI%202018-11-17%2016-46-55

Is there something wrong?


#23

What does /proc/interrupts look like in that case? Are the interrupts still being handled by one core?


#24
           CPU0       CPU1
 16:      41612     111819     GIC-0  18 Edge      gp_timer
 18:         33          0     GIC-0  51 Edge      qcom_rpm_ack
 19:          0          0     GIC-0  53 Edge      qcom_rpm_err
 20:          0          0     GIC-0  54 Edge      qcom_rpm_wakeup
 26:          0          0     GIC-0 241 Edge      ahci[29000000.sata]
 27:          0          0     GIC-0 210 Edge      tsens_interrupt
 28:     180811      18578     GIC-0  67 Edge      qcom-pcie-msi
 29:     188100      85549     GIC-0  89 Edge      qcom-pcie-msi
 30:     205667         25     GIC-0 202 Edge      adm_dma
 31:       3798      11330     GIC-0 255 Level     eth0
 32:        131         26     GIC-0 258 Level     eth1
 33:          0          0     GIC-0 130 Level     bam_dma
 34:          0          0     GIC-0 128 Level     bam_dma
 35:          0          0   PCI-MSI   0 Edge      aerdrv
 36:     180811      18578   PCI-MSI   1 Edge      ath10k_pci
 68:          0          0   PCI-MSI   0 Edge      aerdrv
 69:     188100      85549   PCI-MSI   1 Edge      ath10k_pci
101:         10          0     GIC-0 184 Level     msm_serial0
102:          2          0   msmgpio   6 Edge      gpio-keys
103:          2          0   msmgpio  54 Edge      gpio-keys
104:          2          0   msmgpio  65 Edge      gpio-keys
105:          0          0     GIC-0 142 Level     xhci-hcd:usb1
106:          0          0     GIC-0 237 Level     xhci-hcd:usb3
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:      35221      75618  Rescheduling interrupts
IPI3:         36      20041  Function call interrupts
IPI4:          0          0  CPU stop interrupts
IPI5:      49216      85866  IRQ work interrupts
IPI6:          0          0  completion interrupts
Err:          0

#25

A cursory conclusion would be that the irqs are still not being balanced properly, with core #0 receiving the bulk of interrupts from the devices that are most active.

I am in the middle of debugging a similar situation myself.


#26

It would seem so. Hopefully, someone will provide a fix soon.


#27

It's hard to argue with the raw figures you have in your rrd graphs - but I wonder whether a lot of this isn't architecture specific.

Look at a couple of ARM based boards I'm seeing similar imbalances in the IRQs raised - where most of the device IRQs are raised against a single core - however in those cases it looks like at least some of the work is being rescheduled later (presumably via the rescheduling interrupts).

I suppose the test would be to attempt to actually max out one of the cores - and make sure the load was redistributed properly at that point - certainly it looks like at low levels there's a supervisory function assumed by the first core.