Netgear R7800 exploration (IPQ8065, QCA9984)

However I think swconfig dev switch0 show gives a pretty odd output, it also hangs.
I'm not sure if it's because of my change.

Tho' it said in the old kworker forum thread the swconfig output differs from other routers, perhaps it needs investigation too.

there is a patched version here: http://www.desipro.de/openwrt/

1 Like

FWIW

I am symptom free and no apparent issues for 24+ hrs (see misc output below) after

git revert d6366ce3665f010a7ae7061a557689643073040a
git revert 9fc506e9b2e40db8f3a6df5b59556e59351e376f

and building as usual from current head in master.

The first revert is the commit referenced by @huaracheguarache above (also partly reverted by @shelterx changes above) and is likely the only revert necessary. I did the second revert (generic: ar8216: add support for get_port_stats()) since this commit reverts a commit from Sep. 2017 (generic: drop support for get_port_stats() on ar8xxx) which included comments like:

The implementation is not efficient on ar8xxx switches. It triggers high
CPU load and degrades device performance.

but my second revert may not be necessary.

misc output for reference - I don't know if the swconfig dev switch0 show is normal or not but it does not hang...

root@OpenWrt:~# pidstat -H 2
Linux 4.14.109 (OpenWrt) 	04/08/19 	_armv7l_	(2 CPU)

1554765612    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765614      0         8    0.00    0.50    0.00    0.00    0.50     1  rcu_sched
1554765614      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765614      0      8536    0.00    1.49    0.00    0.00    1.49     0  pidstat
1554765614      0     15111    0.00    3.47    0.00    0.00    3.47     0  kworker/0:2

1554765614    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765616      0      2967    0.50    0.00    0.00    0.00    0.50     0  hostapd
1554765616      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765616      0      8536    0.00    1.50    0.00    0.00    1.50     0  pidstat
1554765616      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2

1554765616    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765618      0         7    0.00    0.50    0.00    0.00    0.50     0  ksoftirqd/0
1554765618      0        14    0.00    0.50    0.00    0.00    0.50     1  ksoftirqd/1
1554765618      0      8536    0.00    1.00    0.00    0.00    1.00     0  pidstat
1554765618      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2

1554765618    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765620      0       668    0.50    0.00    0.00    0.00    0.50     1  logd
1554765620      0      5687    0.50    0.00    0.00    0.00    0.50     1  dropbear
1554765620      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765620      0      8536    0.00    1.00    0.00    0.00    1.00     0  pidstat
1554765620      0     15111    0.00    3.48    0.00    0.00    3.48     0  kworker/0:2

1554765620    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765622      0         8    0.00    0.50    0.00    0.00    0.50     1  rcu_sched
1554765622      0       697    0.00    0.50    0.00    0.00    0.50     1  irqbalance
1554765622      0       842    0.00    0.50    0.00    0.00    0.50     1  netifd
1554765622      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765622      0      8536    0.00    1.50    0.00    0.00    1.50     0  pidstat
1554765622      0     15111    0.00    3.50    0.00    0.00    3.50     0  kworker/0:2

1554765622    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765624      0      2967    0.50    0.00    0.00    0.00    0.50     1  hostapd
1554765624      0      3219    0.50    0.00    0.00    0.00    0.50     0  hostapd
1554765624      0      8536    0.00    1.00    0.00    0.00    1.00     0  pidstat
1554765624      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2

1554765624    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765626      0      7353    0.00    1.00    0.00    0.00    1.00     1  kworker/1:2
1554765626      0      8536    0.00    1.00    0.00    0.00    1.00     0  pidstat
1554765626      0     15111    0.00    3.50    0.00    0.00    3.50     0  kworker/0:2

1554765626    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765628      0      8536    0.00    1.00    0.00    0.00    1.00     0  pidstat
1554765628      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2

1554765628    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765630      0         7    0.00    0.50    0.00    0.00    0.50     0  ksoftirqd/0
1554765630      0        14    0.00    0.50    0.00    0.00    0.50     1  ksoftirqd/1
1554765630      0       668    1.00    0.50    0.00    0.00    1.49     1  logd
1554765630      0      2967    0.50    0.00    0.00    0.00    0.50     1  hostapd
1554765630      0      3088    0.50    0.00    0.00    0.00    0.50     1  sh
1554765630      0      5687    0.50    0.00    0.00    0.00    0.50     1  dropbear
1554765630      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765630      0      8536    0.00    0.50    0.00    0.00    0.50     0  pidstat
1554765630      0     15111    0.00    2.49    0.00    0.00    2.49     0  kworker/0:2

1554765630    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765632      0      5687    0.50    0.00    0.00    0.00    0.50     1  dropbear
1554765632      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765632      0      8536    0.00    1.50    0.00    0.00    1.50     0  pidstat
1554765632      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2

1554765632    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765634      0         8    0.00    0.50    0.00    0.00    0.50     0  rcu_sched
1554765634      0      8536    0.50    1.50    0.00    0.00    2.00     1  pidstat
1554765634      0      8537    0.00    0.50    0.00    0.00    0.50     0  kworker/u4:1
1554765634      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2

1554765634    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765636      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765636      0      8536    0.00    1.50    0.00    0.00    1.50     1  pidstat

1554765636    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765638    453      4756    0.00    0.50    0.00    0.00    0.50     1  dnsmasq
1554765638      0      8536    0.00    1.00    0.00    0.00    1.00     1  pidstat
1554765638      0     15111    0.00    3.50    0.00    0.00    3.50     0  kworker/0:2

1554765638    UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
1554765640      0       668    0.50    0.50    0.00    0.00    1.00     1  logd
1554765640      0      7353    0.00    0.50    0.00    0.00    0.50     1  kworker/1:2
1554765640      0      8536    0.00    1.00    0.00    0.00    1.00     0  pidstat
1554765640      0     15111    0.00    3.00    0.00    0.00    3.00     0  kworker/0:2
^C

Average:      UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
Average:        0         7    0.00    0.07    0.00    0.00    0.07     -  ksoftirqd/0
Average:        0         8    0.00    0.11    0.00    0.00    0.11     -  rcu_sched
Average:        0        14    0.00    0.07    0.00    0.00    0.07     -  ksoftirqd/1
Average:        0       668    0.14    0.07    0.00    0.00    0.21     -  logd
Average:        0       697    0.00    0.04    0.00    0.00    0.04     -  irqbalance
Average:        0       842    0.00    0.04    0.00    0.00    0.04     -  netifd
Average:        0      2967    0.11    0.00    0.00    0.00    0.11     -  hostapd
Average:        0      3088    0.04    0.00    0.00    0.00    0.04     -  sh
Average:        0      3219    0.04    0.00    0.00    0.00    0.04     -  hostapd
Average:      453      4756    0.00    0.04    0.00    0.00    0.04     -  dnsmasq
Average:        0      5687    0.11    0.00    0.00    0.00    0.11     -  dropbear
Average:        0      7353    0.00    0.36    0.00    0.00    0.36     -  kworker/1:2
Average:        0      8536    0.04    1.18    0.00    0.00    1.21     -  pidstat
Average:        0      8537    0.00    0.04    0.00    0.00    0.04     -  kworker/u4:1
Average:        0     15111    0.00    2.92    0.00    0.00    2.92     -  kworker/0:2
root@OpenWrt:~# uptime
 19:20:51 up 1 day, 27 min,  load average: 0.03, 0.07, 0.03
root@OpenWrt:~/local# cat /etc/openwrt_release
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='SNAPSHOT'
DISTRIB_REVISION='r9817+2-80568e5854'
DISTRIB_TARGET='ipq806x/generic'
DISTRIB_ARCH='arm_cortex-a15_neon-vfpv4'
DISTRIB_DESCRIPTION='OpenWrt SNAPSHOT r9817+2-80568e5854'
DISTRIB_TAINTS=''
root@OpenWrt:~/local# swconfig dev switch0 show
Global attributes:
	enable_vlan: 1
	enable_mirror_rx: 0
	enable_mirror_tx: 0
	mirror_monitor_port: 0
	mirror_source_port: 0
	arl_age_time: 300
	arl_table: address resolution table
Port 0: MAC :e5:a9
Port 4: MAC :cb:45
Port 5: MAC :d4:22
Port 6: MAC :3f:b4
Port 6: MAC :08:59
Port 6: MAC :72:40
Port 6: MAC :fa:59
Port 6: MAC :f7:7a
Port 6: MAC :75:f7

	igmp_snooping: 0
	igmp_v3: 0
Port 0:
	mib: MIB counters
RxBroad     : 4
RxPause     : 0
RxMulti     : 29
RxFcsErr    : 0
RxAlignErr  : 0
RxRunt      : 0
RxFragment  : 0
Rx64Byte    : 166070
Rx128Byte   : 4092514
Rx256Byte   : 281751
Rx512Byte   : 81826
Rx1024Byte  : 52347
Rx1518Byte  : 55861
RxMaxByte   : 36540
RxTooLong   : 0
RxGoodByte  : 657459667 (627.0 MiB)
RxBadByte   : 0
RxOverFlow  : 0
Filtered    : 10
TxBroad     : 2599
TxPause     : 0
TxMulti     : 222927
TxUnderRun  : 0
Tx64Byte    : 12
Tx128Byte   : 504868
Tx256Byte   : 378053
Tx512Byte   : 145338
Tx1024Byte  : 62086
Tx1518Byte  : 1326862
TxMaxByte   : 4311377
TxOverSize  : 0
TxByte      : 8622077046 (8.0 GiB)
TxCollision : 0
TxAbortCol  : 0
TxMultiCol  : 0
TxSingleCol : 0
TxExcDefer  : 0
TxDefer     : 0
TxLateCol   : 0

	enable_eee: ???
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 0
	link: port:0 link:up speed:1000baseT full-duplex 
Port 1:
	mib: No MIB data
	enable_eee: 0
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 1
	link: port:1 link:down
Port 2:
	mib: MIB counters
RxBroad     : 40
RxPause     : 0
RxMulti     : 806
RxFcsErr    : 0
RxAlignErr  : 0
RxRunt      : 0
RxFragment  : 0
Rx64Byte    : 41068
Rx128Byte   : 60132
Rx256Byte   : 211335
Rx512Byte   : 49860
Rx1024Byte  : 12163
Rx1518Byte  : 25657
RxMaxByte   : 0
RxTooLong   : 0
RxGoodByte  : 110537032 (105.4 MiB)
RxBadByte   : 0
RxOverFlow  : 0
Filtered    : 22
TxBroad     : 356
TxPause     : 0
TxMulti     : 1869
TxUnderRun  : 0
Tx64Byte    : 22811
Tx128Byte   : 44911
Tx256Byte   : 212040
Tx512Byte   : 75682
Tx1024Byte  : 15246
Tx1518Byte  : 28374
TxMaxByte   : 0
TxOverSize  : 0
TxByte      : 126561948 (120.6 MiB)
TxCollision : 0
TxAbortCol  : 0
TxMultiCol  : 0
TxSingleCol : 0
TxExcDefer  : 0
TxDefer     : 0
TxLateCol   : 0

	enable_eee: 0
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 1
	link: port:2 link:down
Port 3:
	mib: No MIB data
	enable_eee: 0
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 1
	link: port:3 link:down
Port 4:
	mib: MIB counters
RxBroad     : 0
RxPause     : 0
RxMulti     : 9
RxFcsErr    : 0
RxAlignErr  : 0
RxRunt      : 0
RxFragment  : 0
Rx64Byte    : 19763
Rx128Byte   : 6643
Rx256Byte   : 477
Rx512Byte   : 1031
Rx1024Byte  : 66
Rx1518Byte  : 3473
RxMaxByte   : 0
RxTooLong   : 0
RxGoodByte  : 7089977 (6.7 MiB)
RxBadByte   : 0
RxOverFlow  : 0
Filtered    : 0
TxBroad     : 5059
TxPause     : 0
TxMulti     : 11360
TxUnderRun  : 0
Tx64Byte    : 9067
Tx128Byte   : 9474
Tx256Byte   : 7546
Tx512Byte   : 5046
Tx1024Byte  : 2885
Tx1518Byte  : 54424
TxMaxByte   : 0
TxOverSize  : 0
TxByte      : 89373242 (85.2 MiB)
TxCollision : 0
TxAbortCol  : 0
TxMultiCol  : 0
TxSingleCol : 0
TxExcDefer  : 0
TxDefer     : 0
TxLateCol   : 0

	enable_eee: 0
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 1
	link: port:4 link:up speed:1000baseT full-duplex txflow rxflow auto
Port 5:
	mib: MIB counters
RxBroad     : 2593
RxPause     : 0
RxMulti     : 222920
RxFcsErr    : 0
RxAlignErr  : 0
RxRunt      : 0
RxFragment  : 0
Rx64Byte    : 68867
Rx128Byte   : 444527
Rx256Byte   : 371812
Rx512Byte   : 143636
Rx1024Byte  : 61840
Rx1518Byte  : 5638203
RxMaxByte   : 0
RxTooLong   : 0
RxGoodByte  : 8595476744 (8.0 GiB)
RxBadByte   : 0
RxOverFlow  : 0
Filtered    : 100
TxBroad     : 4
TxPause     : 0
TxMulti     : 19
TxUnderRun  : 0
Tx64Byte    : 202628
Tx128Byte   : 4061428
Tx256Byte   : 277191
Tx512Byte   : 81227
Tx1024Byte  : 52151
Tx1518Byte  : 92328
TxMaxByte   : 0
TxOverSize  : 0
TxByte      : 639066535 (609.4 MiB)
TxCollision : 0
TxAbortCol  : 0
TxMultiCol  : 0
TxSingleCol : 0
TxExcDefer  : 0
TxDefer     : 0
TxLateCol   : 0

	enable_eee: 0
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 2
	link: port:5 link:up speed:1000baseT full-duplex auto
Port 6:
	mib: MIB counters
RxBroad     : 5028
RxPause     : 0
RxMulti     : 10619
RxFcsErr    : 0
RxAlignErr  : 0
RxRunt      : 0
RxFragment  : 0
Rx64Byte    : 30848
Rx128Byte   : 51012
Rx256Byte   : 219946
Rx512Byte   : 82086
Rx1024Byte  : 17836
Rx1518Byte  : 1482
RxMaxByte   : 81313
RxTooLong   : 0
RxGoodByte  : 217108793 (207.0 MiB)
RxBadByte   : 0
RxOverFlow  : 0
Filtered    : 48
TxBroad     : 33
TxPause     : 0
TxMulti     : 763
TxUnderRun  : 0
Tx64Byte    : 1
Tx128Byte   : 124556
Tx256Byte   : 214510
Tx512Byte   : 51168
Tx1024Byte  : 12232
Tx1518Byte  : 27881
TxMaxByte   : 1261
TxOverSize  : 0
TxByte      : 119345820 (113.8 MiB)
TxCollision : 0
TxAbortCol  : 0
TxMultiCol  : 0
TxSingleCol : 0
TxExcDefer  : 0
TxDefer     : 0
TxLateCol   : 0

	enable_eee: ???
	igmp_snooping: 0
	vlan_prio: 0
	pvid: 0
	link: port:6 link:up speed:1000baseT full-duplex 
VLAN 1:
	vid: 1
	ports: 1 2 3 4 6t 
VLAN 2:
	vid: 2
	ports: 0t 5 
root@OpenWrt:~/local# 

Thanks!

Is it 18.06.2 ?

How can I flash? Via OpenWRT web interface? or R7800's Recovery mode (TFTP)?

3% seems a bit high yes.
For me kworker is 0.50% at most now, average is about 0.15% but I broke swconfig in the process (i think).

So I guess things can be optimized, or just scrap the MIB counter support.

1 Like

I discovered TP-LINK Archer C7(v4) have same problem.
(Sorry, I didn't check this device properly until now....)

I have two Archer C7, One is OpenWRT 18.06.1 Stable, another one is recent SNAPSHOT(r9752).

I ran pidstat -H2,

<OpenWRT 18.06.1>
Average: 0 1664 0.00 4.10 0.00 0.00 4.10 - kworker/0:3

<OpenWRT SNAPSHOT(r9752)>
Average: 0 5615 0.00 10.29 0.00 0.00 10.29 - kworker/0:1

Because this device is somewhat slower(750MHz, 1Core), kworker consumes 4.10%.
But recent SNAPSHOT build, kworker double scored, 10.29%!

Not only R7800(+IPQ806x? Platform). Archer C7 is very popular/favorite OpenWRT-compatible device...

So hang on now, I looked at the commits. Did they manage to reimplement the MiB counter stuff that was considered bad in this commit made in 2017?

I just got a confirmation from the author of that commit they seem to have ignored his warnings.
So I guess that's it, good work everyone. Now we need the new commits reverted/fixed.

2 Likes

The problem in generic: drop support for get_port_stats() on ar8xxx is that get_port_stats() function is called every 0.1 second. Doing mdio operations is a heavy CPU load on ipq806x because the mdio is emulated using GPIO.

generic: ar8216: add support for get_port_stats() returns existing mib counter values available instead of polling every time. This should fix the problem mentioned above but I don't own an ipq806x router and can't verify that.

I think the high CPU load is caused by generic: ar8216: mib_work_func: read all port mibs everytime
This commits adds 7x mdio operations as it polls 7 port counters instead of original 1 port counter.

1 Like

My earlier hesitation in responding results from not really understanding why these commits have been made. The comment

AR7240 and AR9341 have buggy hardware switch LED trigger. The AR7240
one doesn't blink and the blinking of port0/port5 is reversed on
AR9341...

implies the purpose is to get some leds working. While I don't want the above performance hit on my r7500v2, I also don't want to block improving another device.

I'm willing to help test if that will prevent future unexpected performance hits.

EDIT:

I see some alternatives are being discussed on IRC...

I've created a bug report for this now. I made some "off-forum" discussions first, because I just wanted to make sure I wasn't giving the wrong information.
Here it is for those who are interested.

(I noticed I tagged someone wrong in the report.. can't edit, sorry. :blush: I was about to change it but clicked save.)

According to the dev (who is good in my opinion) these images are based off 18.06, are "pre alpha," and (at least up until now) are really only meant for development purposes by the group of people working on them...

Flash at your own risk via TFTP only

There is a patch titled 901-switch-mib-disable.patch but I have not tried it.

That might help, I am not sure if it disables everything tho' or if it's still does some other polling.
That is why i commented out ar8xxx_mib_fetch_port_stat(priv, i, false); too.
I haven't had the time to try leaving the above function in.

Kind of what i suspected, I went all the way back to using the files in this commit

There are spikes at about 20% every 2s , however kworker is only using about 2,5%
So there are probably some other polls going on in the arxxxx files that's causing it, not just MiB counters.
swconfig works now however so the mib_fetch_port_stat i commented out in the newer driver broke swconfig, but as I've said earlier, it also made kworker silent at 0% cpu,

Just tried the patch(s). It is good.

I noticed that Pi-Hole (3.1M entries) on a BananaPro and setting DNS cache to 0 (default: 150) resulted in faster internet than using adblock (764K entries) on my R7800 with the build of a couple of days ago.

i'm currently checking the patch and dsa test
https://git.openwrt.org/?p=openwrt/staging/chunkeey.git;a=commit;h=9f5604224d719b5a735b51a6de858843305ddbc7
https://git.openwrt.org/?p=openwrt/staging/chunkeey.git;a=commit;h=fa435be4def8048ebea4c39f9155312d79c04a08

Usb is broken and the dts lack of dsa settings for the switch to work... someone should comunicate him that this is also needed

With this dsa works... still working on broken usb3 ports

Also usb-of-simple doesn't go deprecated with kernel 4.19
It looks like usb-dwc3-qcom is for some special qcom device

@chunkeey

1 Like

do you have a try with the 4.19 and dsa patch?
it seems the dsa patch is not incompatible after linux 4.19.25

Anyone here with a PPPOE ISP and using DSA?

A year or so ago when I tested DSA on ipq8064 the router would enter a reboot-loop once a PPPOE session is initiated.

Once I unplugged the WAN port the router boots normally.

all my message was related to the 4.19 kernel

i know, i use that patch for a time
after linux4.19.25,it seems that the kernel source don't have those functions which the dsa patch modifies