High idle CPU usage (ds-lite ?)

madez · January 13, 2023, 4:41pm

On my TP-Link Archer C5 v1 running OpenWrt 22.03.2 r19803-9a599fee93 / LuCI openwrt-22.03 branch git-22.288.45147-96ec0cd I'm experiencing constant high CPU usage:

CPU:  69% usr  29% sys   0% nic   0% idle   0% io   0% irq   0% sirq
Load average: 1.96 1.04 0.76 5/58 16744
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 1688     1 root     S     1548   1%   1% /usr/sbin/odhcpd
 1630     1 root     S     2928   2%   1% /sbin/netifd
 1237     1 root     S     2312   2%   1% /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
    1     0 root     S     1696   1%   0% /sbin/procd
  592     1 ubus     S     1412   1%   0% /sbin/ubusd
11236 11227 root     R     1324   1%   0% top
11225  1459 root     S     1228   1%   0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
    8     2 root     SW       0   0%   0% [ksoftirqd/0]
 1811     1 root     S     4188   3%   0% /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80 -C /etc/uhttpd.crt -K /etc/uhttpd.key -s 0.0.0.0:443 -s [::]:443
 2173  1630 root     S     1124   1%   0% odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
27456     2 root     IW       0   0%   0% [kworker/u2:1-ev]
14620     2 root     IW       0   0%   0% [kworker/u2:0-ev]
 1567  1564 network  S     4424   4%   0% /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1590  1563 network  S     4424   4%   0% /usr/sbin/hostapd -s -g /var/run/hostapd/global
 2470     1 root     S     2656   2%   0% {ntpd} /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/share/libubox/jshn.sh -- /usr/sb
 1563     1 root     S     2656   2%   0% {hostapd} /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 2155     1 root     S     2656   2%   0% {dnsmasq} /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r /tmp/dnsmasq.d -r /tmp/ho
 1564     1 root     S     2656   2%   0% {wpa_supplicant} /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
16718 16716 root     R     2272   2%   0% utpl -S /usr/share/firewall4/main.uc
 2157  2155 dnsmasq  S     1520   1%   0% /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
16727  2173 root     R     1516   1%   0% {dhcpv6.script} /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated
16715     1 root     S     1344   1%   0% {fw3} /bin/sh /sbin/fw3 -q network wan6
 1183     1 logd     S     1340   1%   0% /sbin/logd -S 64
16743     1 root     R     1340   1%   0% {sysntpd} /bin/sh /etc/rc.common /etc/init.d/sysntpd running
16716 16715 root     S     1316   1%   0% flock -x /var/run/fw4.lock utpl -S /usr/share/firewall4/main.uc
11227 11225 root     S     1312   1%   0% -ash
 2523  2470 ntp      S     1308   1%   0% /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org
 1459     1 root     S     1208   1%   0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
  628     1 root     S     1108   1%   0% /sbin/urngd
  593     1 root     S      996   1%   0% /sbin/askfirst /usr/libexec/login.sh
16744     1 root     R        4   0%   0% {sysntpd} /bin/sh /etc/rc.common /etc/init.d/sysntpd reload
  371     2 root     IW       0   0%   0% [kworker/0:2-eve]
  405     2 root     IW<      0   0%   0% [kworker/0:1H-ev]
  126     2 root     SW       0   0%   0% [kcompactd0]
24604     2 root     IW       0   0%   0% [kworker/u2:2-ev]
  221     2 root     SW       0   0%   0% [spi0]
  531     2 root     SWN      0   0%   0% [jffs2_gcd_mtd4]
    4     2 root     IW<      0   0%   0% [kworker/0:0H-kb]
  786     2 root     IW<      0   0%   0% [ath10k_tx_compl]
  784     2 root     IW<      0   0%   0% [ath10k_wq]
  733     2 root     IW<      0   0%   0% [cfg80211]
  785     2 root     IW<      0   0%   0% [ath10k_aux_wq]
  417     2 root     SW       0   0%   0% [irq/23-keys]
  373     2 root     IW<      0   0%   0% [ipv6_addrconf]
  416     2 root     SW       0   0%   0% [irq/16-keys]
  140     2 root     SW       0   0%   0% [watchdogd]
  138     2 root     IW<      0   0%   0% [kblockd]
  139     2 root     IW<      0   0%   0% [blkcg_punt_bio]
  174     2 root     SW       0   0%   0% [kswapd0]
    9     2 root     IW<      0   0%   0% [netns]
   39     2 root     SW       0   0%   0% [oom_reaper]
    6     2 root     IW<      0   0%   0% [mm_percpu_wq]
    2     0 root     SW       0   0%   0% [kthreadd]
    7     2 root     SW       0   0%   0% [rcu_tasks_trace]
  124     2 root     IW<      0   0%   0% [writeback]
  176     2 root     IW<      0   0%   0% [kthrotld]
 4131     2 root     IW<      0   0%   0% [wg-crypt-1blu]
15671     2 root     IW       0   0%   0% [kworker/0:1]

Any ideas what could be causing this?

darksky · January 13, 2023, 5:02pm

Seems odd that the output is showing a high CPU but not a corresponding process. Does another util offer anything? Try htop which by default sorts by CPU%.

madez · January 13, 2023, 5:07pm

Here is the output of htop:

  CPU[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||      89.9%] Tasks: 22, 0 thr, 30 kthr; 1 running
  Mem[||||||||||||||||||||||||||||||||||||||                                                               20.5M/119M] Load average: 3.25 2.86 2.41 
  Swp[                                                                                                          0K/0K] Uptime: 1 day, 21:27:15

  [Main] [I/O]
  PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
28656 root        20   0  3400  2944  1140 R   2.5  2.4  0:01.35 htop
 1237 root        20   0  2312  1400  1044 D   1.9  1.1  6:29.61 /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1688 root        20   0  1656  1012   880 S   1.3  0.8 14:37.05 /usr/sbin/odhcpd
  592 ubus        20   0  1412   948   812 S   0.6  0.8  3:27.95 /sbin/ubusd
 1630 root        20   0  2944  2280   996 S   0.6  1.9 13:18.93 /sbin/netifd
 1811 root        20   0  4220  2356  2080 S   0.6  1.9  1:07.61 /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80 -C /etc/uhttpd.crt -K /etc/uhttpd.key -s 0.0.0.0:
11225 root        20   0  1228   964   908 S   0.6  0.8  0:05.27 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
    1 root        20   0  1696  1092   900 S   0.0  0.9  1:30.89 /sbin/procd
  593 root        20   0   996   672   636 S   0.0  0.5  0:00.04 /sbin/askfirst /usr/libexec/login.sh
  628 root        20   0  1108   728   676 S   0.0  0.6  0:00.46 /sbin/urngd
 1183 logd        20   0  1340   904   760 S   0.0  0.7  0:00.14 /sbin/logd -S 64
 1459 root        20   0  1208   952   908 S   0.0  0.8  0:00.03 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 1563 root        20   0  2656  1036   940 S   0.0  0.8  0:00.06 /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1564 root        20   0  2656  1036   940 S   0.0  0.8  0:00.00 /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1567 network     20   0  4424  1776  1668 S   0.0  1.5  0:06.38 /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1590 network     20   0  4424  1584  1476 S   0.0  1.3  0:02.56 /usr/sbin/hostapd -s -g /var/run/hostapd/global
 2155 root        20   0  2656  1040   940 S   0.0  0.9  0:00.01 /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r /tmp/dnsmasq
 2157 dnsmasq     20   0  1520  1092   944 S   0.0  0.9  3:52.45 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 2173 root        20   0  1124   816   764 S   0.0  0.7  1:24.95 odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
 2470 root        20   0  2656  1040   940 S   0.0  0.9  0:00.11 /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/share/libubox/js
 2523 ntp         20   0  1308   984   944 S   0.0  0.8  0:01.41 /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org
11227 root        20   0  1312  1052  1008 S   0.0  0.9  0:00.04 -ash

I find also weird that the CPU usage is high but no process is shown. Maybe the CPU clock gets lower when idle and thus with little load it seems saturated?

darksky · January 13, 2023, 5:14pm

I haven't seen that before. Your load avg is definetly high/CPU% high but no obvious process. I think htop offers an option to show 'hide kernel threads' which you can toggle from F2>display options. Is that enabled?

madez · January 13, 2023, 5:16pm

Yes, that was enabled by default. I disabled it. Here is the new output:


  CPU[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||97.6%] Tasks: 24, 0 thr, 30 kthr; 1 running
  Mem[||||||||||||||||||||||||||||||||||||||                                                               20.5M/119M] Load average: 2.04 2.27 2.29 
  Swp[                                                                                                          0K/0K] Uptime: 1 day, 21:37:05

  [Main] [I/O]
  PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
15101 root        20   0  3416  3052  1140 R   2.4  2.5  0:01.11 htop
 1630 root        20   0  2944  2284   996 S   0.0  1.9 13:22.08 /sbin/netifd
11225 root        20   0  1228   964   908 S   0.6  0.8  0:06.59 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
    8 root        20   0     0     0     0 S   0.6  0.0  8:21.58 ksoftirqd/0
  126 root        20   0     0     0     0 S   0.0  0.0  0:06.95 kcompactd0
  371 root        20   0     0     0     0 I   0.0  0.0  3:28.62 kworker/0:2-events_power_efficient
  592 ubus        20   0  1412   948   812 S   0.6  0.8  3:29.38 /sbin/ubusd
 1688 root        20   0  1656  1012   880 S   0.0  0.8 14:40.35 /usr/sbin/odhcpd
 2157 dnsmasq     20   0  1520  1092   944 S   0.0  0.9  3:54.13 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
    1 root        20   0  1696  1092   900 S   0.0  0.9  1:31.28 /sbin/procd
    2 root        20   0     0     0     0 S   0.0  0.0  0:00.00 kthreadd
    4 root         0 -20     0     0     0 I   0.0  0.0  0:00.04 kworker/0:0H-kblockd
    6 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 mm_percpu_wq
    7 root        20   0     0     0     0 S   0.0  0.0  0:00.00 rcu_tasks_trace
    9 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 netns
   39 root        20   0     0     0     0 S   0.0  0.0  0:00.00 oom_reaper
  124 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 writeback
  138 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 kblockd
  139 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 blkcg_punt_bio
  140 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 watchdogd
  174 root        20   0     0     0     0 S   0.0  0.0  0:00.00 kswapd0
  176 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 kthrotld
  221 root        20   0     0     0     0 S   0.0  0.0  0:00.55 spi0
  373 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ipv6_addrconf
  405 root         0 -20     0     0     0 I   0.0  0.0  0:11.76 kworker/0:1H-events_highpri
  416 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 irq/16-keys
  417 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 irq/23-keys
  531 root        30  10     0     0     0 S   0.0  0.0  0:00.29 jffs2_gcd_mtd4
  593 root        20   0   996   672   636 S   0.0  0.5  0:00.04 /sbin/askfirst /usr/libexec/login.sh
  628 root        20   0  1108   728   676 S   0.0  0.6  0:00.46 /sbin/urngd
  733 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 cfg80211
  784 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ath10k_wq
  785 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ath10k_aux_wq
  786 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ath10k_tx_compl
 1183 logd        20   0  1340   904   760 S   0.0  0.7  0:00.14 /sbin/logd -S 64
 1237 root        20   0  2308  1392  1044 D   4.9  1.1  6:44.12 /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1459 root        20   0  1208   952   908 S   0.0  0.8  0:00.03 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 1563 root        20   0  2656  1036   940 S   0.0  0.8  0:00.06 /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1564 root        20   0  2656  1036   940 S   0.0  0.8  0:00.00 /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1567 network     20   0  4424  1776  1668 S   0.0  1.5  0:06.40 /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1590 network     20   0  4424  1584  1476 S   0.0  1.3  0:02.57 /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1811 root        20   0  4220  2356  2080 S   0.6  1.9  1:09.58 /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80 -C /etc/uhttpd.crt -K /etc/uhttpd.key -s 0.0.0.0:
 2155 root        20   0  2656  1040   940 S   0.0  0.9  0:00.01 /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r /tmp/dnsmasq
 2173 root        20   0  1124   816   764 S   0.0  0.7  1:25.25 odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
 2470 root        20   0  2656  1040   940 S   0.0  0.9  0:00.11 /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/share/libubox/js
 2523 ntp         20   0  1308   984   944 S   0.0  0.8  0:01.42 /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org
 4131 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 wg-crypt-1blu
 4552 root        20   0     0     0     0 S   0.6  0.0  0:00.04 kworker/u2:0-events_unbound
11227 root        20   0  1312  1052  1008 S   0.0  0.9  0:00.04 -ash
15671 root        20   0     0     0     0 I   0.0  0.0  0:00.00 kworker/0:1
23835 root        20   0     0     0     0 I   0.0  0.0  0:02.21 kworker/u2:2+events_unbound
27456 root        20   0     0     0     0 I   0.0  0.0  0:01.98 kworker/u2:1-events_unbound
15871 root        20   0  1124   472   420 S   0.0  0.4  0:00.00 odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
15928 root        20   0  1412  1076   944 S   1.2  0.9  0:00.02 /bin/sh /etc/rc.common /etc/init.d/sysntpd running

darksky · January 13, 2023, 5:20pm

I do not know. Does this high CPU% persist after a reboot?

madez · January 13, 2023, 5:27pm

Yes. This is the output of top after a reboot:

Mem: 31044K used, 91276K free, 60K shrd, 0K buff, 11536K cached
CPU:  91% usr   8% sys   0% nic   0% idle   0% io   0% irq   0% sirq
Load average: 2.49 0.73 0.25 5/64 3538
  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3528  2190 root     R     1496   1%  25% {dhcpv6.script} /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated
 3515  3394 root     R     1556   1%  17% utpl -S /usr/share/firewall4/main.uc
 1594  1568 network  S     4424   4%   0% /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1571  1567 network  S     4424   4%   0% /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1815     1 root     S     4132   3%   0% /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80 -C /etc/uhttpd.crt -K /etc/uhttpd.key -s 0.0.0.0:443 -s [::]:443
 2549     1 root     S     2656   2%   0% {ntpd} /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/share/libubox/jshn.sh -- /usr/sb
 1568     1 root     S     2656   2%   0% {wpa_supplicant} /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 2690     1 root     S     2656   2%   0% {dnsmasq} /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r /tmp/dnsmasq.d -r /tmp/ho
 1567     1 root     S     2656   2%   0% {hostapd} /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1237     1 root     S     2128   2%   0% /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1634     1 root     S     1832   1%   0% /sbin/netifd
    1     0 root     S     1672   1%   0% /sbin/procd
 1690     1 root     S     1532   1%   0% /usr/sbin/odhcpd
 2699  2690 dnsmasq  S     1476   1%   0% /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
 3433  1634 root     S     1396   1%   0% {hotplug-call} /bin/sh /sbin/hotplug-call iface
 3447  3433 root     S     1396   1%   0% {hotplug-call} /bin/sh /sbin/hotplug-call iface
 3536     1 root     R     1388   1%   0% {sysntpd} /bin/sh /etc/rc.common /etc/init.d/sysntpd running
 3393     1 root     S     1344   1%   0% {fw3} /bin/sh /sbin/fw3 -q network wan6
  592     1 ubus     S     1340   1%   0% /sbin/ubusd
 1183     1 logd     S     1340   1%   0% /sbin/logd -S 64
 3537     1 root     R     1340   1%   0% {sysntpd} /bin/sh /etc/rc.common /etc/init.d/sysntpd reload
 3532  2390 root     R     1320   1%   0% top
 3498  3447 root     S     1320   1%   0% {fw4} /bin/sh /sbin/fw4 -q network wan6
 3394  3393 root     S     1316   1%   0% flock -x /var/run/fw4.lock utpl -S /usr/share/firewall4/main.uc
 2390  2330 root     S     1312   1%   0% -ash
 2583  2549 ntp      S     1308   1%   0% /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org
 3501  3498 root     S     1308   1%   0% flock -x /var/run/fw4.lock utpl -S /usr/share/firewall4/main.uc
 2330  1463 root     S     1228   1%   0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
 1463     1 root     S     1208   1%   0% /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 2190  1634 root     S     1124   1%   0% odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
  628     1 root     S     1108   1%   0% /sbin/urngd
  593     1 root     S      996   1%   0% /sbin/askfirst /usr/libexec/login.sh
  405     2 root     IW<      0   0%   0% [kworker/0:1H-ev]
   40     2 root     IW       0   0%   0% [kworker/u2:2-ev]
  531     2 root     SWN      0   0%   0% [jffs2_gcd_mtd4]
  371     2 root     IW       0   0%   0% [kworker/0:2-eve]
    8     2 root     SW       0   0%   0% [ksoftirqd/0]
  126     2 root     SW       0   0%   0% [kcompactd0]
    4     2 root     IW<      0   0%   0% [kworker/0:0H-kb]
  221     2 root     SW       0   0%   0% [spi0]
    5     2 root     IW       0   0%   0% [kworker/u2:0-ev]
   41     2 root     IW       0   0%   0% [kworker/0:1-eve]
   10     2 root     IW       0   0%   0% [kworker/u2:1-ev]
  784     2 root     IW<      0   0%   0% [ath10k_wq]
  733     2 root     IW<      0   0%   0% [cfg80211]
  785     2 root     IW<      0   0%   0% [ath10k_aux_wq]
  417     2 root     SW       0   0%   0% [irq/23-keys]
  786     2 root     IW<      0   0%   0% [ath10k_tx_compl]
  373     2 root     IW<      0   0%   0% [ipv6_addrconf]
  174     2 root     SW       0   0%   0% [kswapd0]
  139     2 root     IW<      0   0%   0% [blkcg_punt_bio]
  138     2 root     IW<      0   0%   0% [kblockd]
  140     2 root     SW       0   0%   0% [watchdogd]
  176     2 root     IW<      0   0%   0% [kthrotld]
   39     2 root     SW       0   0%   0% [oom_reaper]
  124     2 root     IW<      0   0%   0% [writeback]
    7     2 root     SW       0   0%   0% [rcu_tasks_trace]
    6     2 root     IW<      0   0%   0% [mm_percpu_wq]
    2     0 root     SW       0   0%   0% [kthreadd]
    3     2 root     IW       0   0%   0% [kworker/0:0-eve]
    9     2 root     IW<      0   0%   0% [netns]
  416     2 root     SW       0   0%   0% [irq/16-keys]
  903     2 root     IW<      0   0%   0% [wg-crypt-1blu]

This is a vanilla OpenWRT with very few additional packages. I think I only installed ds-lite, htop, tcpdump, and luci-app-wireguard. I did have issues with that configuration before, which I described here: Troubleshooting packet loss

All symptoms were present before installing htop, tcpdump, and luci-app-wireguard, and no wireguard connection is active.

I tried another device but the problem is the same. So I assume it might be a problem with ds-lite or my ISPs IPv6 behavior. But shouldn't I still see what is keeping the CPU busy?

Mijzelf · January 13, 2023, 5:44pm

Disable 'Hide kernel threads' in the setup of htop.

chrischn · January 13, 2023, 8:07pm

try order by cpu load.
i think, you dont see all processes.

madez · January 13, 2023, 10:21pm

I feel stupid, so let me try to not pretend to feel smarter.

I delete the configuration of htop:
root@OpenWrt:~# rm -r .config/htop/
I run htop:
root@OpenWrt:~# htop
I hit F2 to enter the setup
I move the curser to Hide kernel threads and uncheck it with enter.
~I hit F10 and htop locks up. Sigh, I restart and skip this step.~
I hit esc
I resize the window to make sure that htop vertical output is not truncated
I hit F6, move the cursor to PERCENT_CPU and hit enter
I copy the output of a frame of htop:


  CPU[||||||||||||||||||||||||||||||||||||||||||||||||||||||                           61.9%] Tasks: 24, 0 thr, 29 kthr; 1 running
  Mem[|||||||||||||||||||||||||                                                   18.6M/119M] Load average: 0.53 0.63 0.66 
  Swp[                                                                                 0K/0K] Uptime: 04:57:03

  [Main] [I/O]
  PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
 6894 root        20   0  2740  2480  1116 R   2.2  2.0  0:03.40 htop
  371 root        20   0     0     0     0 I   0.0  0.0  0:22.70 kworker/0:2-events_power_efficient
  592 ubus        20   0  1348   916   812 S   0.0  0.7  0:19.55 /sbin/ubusd
 1634 root        20   0  1972  1308   996 S   0.6  1.1  1:30.53 /sbin/netifd
 2699 dnsmasq     20   0  1520  1092   944 S   0.0  0.9  0:32.64 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
    1 root        20   0  1676  1080   896 S   0.0  0.9  0:10.29 /sbin/procd
    2 root        20   0     0     0     0 S   0.0  0.0  0:00.00 kthreadd
    4 root         0 -20     0     0     0 I   0.0  0.0  0:00.04 kworker/0:0H-kblockd
    5 root        20   0     0     0     0 I   0.6  0.0  0:00.56 kworker/u2:0-events_unbound
    6 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 mm_percpu_wq
    7 root        20   0     0     0     0 S   0.0  0.0  0:00.00 rcu_tasks_trace
    8 root        20   0     0     0     0 R   1.1  0.0  0:27.49 ksoftirqd/0
    9 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 netns
   39 root        20   0     0     0     0 S   0.0  0.0  0:00.00 oom_reaper
   40 root        20   0     0     0     0 I   0.0  0.0  0:00.96 kworker/u2:2-events_unbound
   41 root        20   0     0     0     0 I   0.0  0.0  0:00.01 kworker/0:1-events
  124 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 writeback
  126 root        20   0     0     0     0 S   0.0  0.0  0:00.68 kcompactd0
  138 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 kblockd
  139 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 blkcg_punt_bio
  140 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 watchdogd
  174 root        20   0     0     0     0 S   0.0  0.0  0:00.00 kswapd0
  176 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 kthrotld
  221 root        20   0     0     0     0 S   0.0  0.0  0:00.05 spi0
  373 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ipv6_addrconf
  405 root         0 -20     0     0     0 I   0.0  0.0  0:07.06 kworker/0:1H-events_highpri
  416 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 irq/16-keys
  417 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 irq/23-keys
  531 root        30  10     0     0     0 S   0.0  0.0  0:00.81 jffs2_gcd_mtd4
  593 root        20   0   996   672   636 S   0.0  0.5  0:00.05 /sbin/askfirst /usr/libexec/login.sh
  628 root        20   0  1108   728   676 S   0.0  0.6  0:00.46 /sbin/urngd
  733 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 cfg80211
  784 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ath10k_wq
  785 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ath10k_aux_wq
  786 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 ath10k_tx_compl
 1183 logd        20   0  1340   880   760 S   0.0  0.7  0:00.07 /sbin/logd -S 64
 1237 root        20   0  2128  1192  1044 S   0.0  1.0  0:00.20 /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1463 root        20   0  1208   952   908 S   0.0  0.8  0:00.01 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 1567 root        20   0  2656  1040   940 S   0.0  0.9  0:00.00 /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run
 1568 root        20   0  2656  1040   940 S   0.0  0.9  0:00.06 /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant 
 1571 network     20   0  4424  1588  1480 S   0.0  1.3  0:00.82 /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1594 network     20   0  4424  1780  1668 S   0.0  1.5  0:00.31 /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1690 root        20   0  1548  1000   880 S   1.1  0.8  1:36.58 /usr/sbin/odhcpd
 1815 root        20   0  4132  2200  2020 S   0.0  1.8  0:00.35 /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]
 2190 root        20   0  1124   816   764 S   0.6  0.7  0:09.10 odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
 2549 root        20   0  2656  1040   940 S   0.0  0.9  0:00.06 /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/js
 2583 ntp         20   0  1308   984   944 S   0.0  0.8  0:00.21 /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.nt
 2690 root        20   0  2656  1040   940 S   0.0  0.9  0:00.01 /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hos
 2903 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 wg-crypt-1blu
 3520 root        20   0  1228   964   908 S   1.1  0.8  0:01.77 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
 3534 root        20   0  1312  1056  1008 S   0.0  0.9  0:00.02 -ash
 8268 root        20   0  1560  1316  1008 S  11.6  1.1  0:00.21 /bin/sh /etc/rc.common /etc/init.d/sysntpd reload
 8303 root        20   0  1376  1040   944 R   0.0  0.9  0:00.00 /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated

madez · January 15, 2023, 2:29am

My idea to further look into this is using linux kernel profiling so I can look in detail what is keeping the system so busy.

If you know that this idea is suboptimal or if you know how to achieve what I'm trying, let me know, because I don't really know what I'm doing.

hnyman · January 15, 2023, 7:59am

Sorting by the "TIME+" column on the htop screen may reveals something...

madez:

  PID USER       PRI  NI  VIRT   RES   SHR S  CPU%-MEM%   TIME+  Command
15101 root        20   0  3416  3052  1140 R   2.4  2.5  0:01.11 htop
 1630 root        20   0  2944  2284   996 S   0.0  1.9 13:22.08 /sbin/netifd
11225 root        20   0  1228   964   908 S   0.6  0.8  0:06.59 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
    8 root        20   0     0     0     0 S   0.6  0.0  8:21.58 ksoftirqd/0
  126 root        20   0     0     0     0 S   0.0  0.0  0:06.95 kcompactd0
  371 root        20   0     0     0     0 I   0.0  0.0  3:28.62 kworker/0:2-events_power_efficient
  592 ubus        20   0  1412   948   812 S   0.6  0.8  3:29.38 /sbin/ubusd
 1688 root        20   0  1656  1012   880 S   0.0  0.8 14:40.35 /usr/sbin/odhcpd
 2157 dnsmasq     20   0  1520  1092   944 S   0.0  0.9  3:54.13 /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
    1 root        20   0  1696  1092   900 S   0.0  0.9  1:31.28 /sbin/procd
    2 root        20   0     0     0     0 S   0.0  0.0  0:00.00 kthreadd
    4 root         0 -20     0     0     0 I   0.0  0.0  0:00.04 kworker/0:0H-kblockd
    6 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 mm_percpu_wq

To me it looks like odhcpd and netifd have consumed pretty much all CPU time.

Quite possible.
But no idea why it would not show in the CPU%.

Note that odhcpd is about propagating the IPv6 info to LAN, while odhcp6c the ISP facing part.

Just for reference, the htop output of my own router sorted by TIME+.
netifd and odhcpd do not rise into similar dominance as in your data. (After a full day's uptime, their total time consumed is much less as in your example.

    0[||                                     2.5%] Tasks: 29, 10 thr, 68 kthr; 2 running
    1[|                                      0.4%] Load average: 0.06 0.03 0.01
  Mem[||||||||||||||||||                114M/463M] Uptime: 1 day, 13:09:55
  Swp[                                      0K/0K]

  [Main] [I/O]
  PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%   TIME+ -Command
 4312 root        39  19  1328   904   816 S   0.4  0.2  4:24.60 /usr/sbin/nlbwmon -o /var/lib/nlbwmo
 1777 network     20   0  4984  3272  2960 S   0.0  0.7  4:11.93 /usr/sbin/hostapd -s -g /var/run/hos
 7023 dnsmasq     20   0  4192  3776   888 S   0.4  0.8  2:22.28 /usr/sbin/dnsmasq -C /var/etc/dnsmas
 1721 root        20   0  2464  1304  1160 S   0.0  0.3  2:10.81 /usr/sbin/unetd -h /var/run/unetd/ho
24368 root        25   5  5544  2908  2224 S   0.0  0.6  0:53.34 /usr/sbin/collectd -C /tmp/collectd.
 1883 root        20   0  1880  1252  1064 S   0.0  0.3  0:39.75 /sbin/netifd
   13 root        20   0     0     0     0 I   0.0  0.0  0:37.25 rcu_sched
 2044 root        20   0  1472   972   852 S   0.0  0.2  0:35.35 /usr/sbin/odhcpd
   18 root        20   0     0     0     0 S   0.0  0.0  0:32.46 ksoftirqd/1
 1639 root        20   0  1984  1240  1140 S   0.0  0.3  0:28.66 /usr/sbin/qosify
   12 root        20   0     0     0     0 S   0.0  0.0  0:25.43 ksoftirqd/0
 3673 root        20   0  1084   748   628 S   0.0  0.2  0:19.53 /usr/sbin/irqbalance -f -c 2 -t 10
   26 root        20   0     0     0     0 S   0.0  0.0  0:17.44 kcompactd0
24381 root        25   5  5544  2908  2224 S   0.0  0.6  0:15.48 /usr/sbin/collectd -C /tmp/collectd.
24387 root        25   5  5544  2908  2224 S   0.0  0.6  0:14.98 /usr/sbin/collectd -C /tmp/collectd.

Note that odhcp6c triggers the /lib/netifd/dhcpv6.script,

 2510 root   ...  0:00.39  odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2

which is visible in one of your examples. That means that there is new IPv6 address info from upstream and then interface has possible new address(es) and that also then may trigger the firewall reload, also visible in your extract below. And also then triggers odhcpd to inform LAN clients about the new addresses & routing.

madez:

  PID  PPID USER     STAT   VSZ %VSZ %CPU COMMAND
 3528  2190 root     R     1496   1%  25% {dhcpv6.script} /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated
 3515  3394 root     R     1556   1%  17% utpl -S /usr/share/firewall4/main.uc

Pure hunch, but

something in your IPv6 & ds-lite config causes frequent interface updates & firewall reloads.
Possibly your ISP constantly/erroneously sends new RAs that get noticed and processed by odhcp6c ?
the firewall rules reload etc. are not visible in the process chart, as the utpl -S /usr/share/firewall4/main.uc is just a momentary thing, not a permanent process.
(firewall is just netfilter rules in kernel, just the rule settings is an external process)

DS-lite is somewhat rare, so there is not much practice experience about it.

Might also be that the time spent on those momentary child processes gets accounted to the parent process?

You might also try seeing the processes as a tree:

  PID USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%   TIME+ - UTIME+   STIME+  Command
    1 root        20   0  1704  1140   924 S   0.0  0.2  0:08.34  0:00.44  0:07.90 /sbin/procd
 4312 root        39  19  1436   924   816 S   0.0  0.2  4:27.98  0:38.93  3:49.05 `- /usr/sbin/nlbwmon -o /var/lib/nlbwmon -b 524288 -i 24h -r
 1721 root        20   0  2464  1304  1160 S   0.0  0.3  2:12.29  1:05.81  1:06.48 `- /usr/sbin/unetd -h /var/run/unetd/hosts -u /var/run/unetd/
24368 root        25   5  5544  2904  2224 S   0.0  0.6  0:56.29  0:24.72  0:31.57 `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24381 root        25   5  5544  2904  2224 S   0.0  0.6  0:16.36  0:06.00  0:10.36 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24387 root        25   5  5544  2904  2224 S   0.0  0.6  0:15.69  0:07.19  0:08.50 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24388 root        25   5  5544  2904  2224 S   0.0  0.6  0:15.21  0:06.91  0:08.30 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24382 root        25   5  5544  2904  2224 S   0.0  0.6  0:01.66  0:00.87  0:00.79 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24385 root        25   5  5544  2904  2224 S   0.0  0.6  0:01.64  0:00.78  0:00.86 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24383 root        25   5  5544  2904  2224 S   0.0  0.6  0:01.63  0:00.89  0:00.74 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24386 root        25   5  5544  2904  2224 S   0.0  0.6  0:01.62  0:00.84  0:00.78 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24384 root        25   5  5544  2904  2224 S   0.0  0.6  0:01.57  0:00.76  0:00.81 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
24380 root        25   5  5544  2904  2224 S   0.0  0.6  0:00.63  0:00.05  0:00.58 |  `- /usr/sbin/collectd -C /tmp/collectd.conf -f
 1883 root        20   0  1880  1252  1064 S   0.7  0.3  0:40.21  0:04.27  0:35.94 `- /sbin/netifd
 2510 root        20   0   936   684   632 S   0.0  0.1  0:00.40  0:00.00  0:00.40 |  `- odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
 2503 root        20   0  1176   848   812 S   0.0  0.2  0:00.10  0:00.03  0:00.07 |  `- udhcpc -p /var/run/udhcpc-eth0.2.pid -s /lib/netifd/dhc
 2044 root        20   0  1580   984   852 S   0.0  0.2  0:35.83  0:14.02  0:21.81 `- /usr/sbin/odhcpd
 1639 root        20   0  1984  1240  1140 S   0.0  0.3  0:28.99  0:05.66  0:23.33 `- /usr/sbin/qosify
 3673 root        20   0  1088   752   628 S   0.0  0.2  0:19.76  0:05.33  0:14.43 `- /usr/sbin/irqbalance -f -c 2 -t 10
 3700 root        20   0  1088   752   628 S   0.0  0.2  0:00.00  0:00.00  0:00.00 |  `- /usr/sbin/irqbalance -f -c 2 -t 10
  302 ubus        20   0  1236   852   740 S   0.0  0.2  0:14.05  0:03.26  0:10.79 `- /sbin/ubusd
 1132 root        20   0  3820  2440  1796 S   0.0  0.5  0:01.72  0:00.68  0:01.04 `- /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 2539 root        20   0  4224  3472  3072 S   0.0  0.7  0:01.64  0:00.69  0:00.95 `- /usr/sbin/uhttpd -f -h /www -r router1 -x /cgi-bin -u /ubu

I added "ds-lite ?" to the topic title, as somebody might have ran into this earlier.

madez · January 16, 2023, 2:29pm

I installed the latest stable 22.03.3 and didn't install ds-lite. I get assigned 3 IPv6 addresses and a gateway for some minutes without connectivity, and then Luci shows me additionally that I received a delegated prefix, a fourth IPv6 address, and two DNS servers. Yet still no connectivity.

After restarting the modem and the router a couple of times, IPv6 connectivity started working. The CPU load is still high:


  CPU[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 29, 0 thr, 28 kthr; 1 running
  Mem[||||||||||||||||||||||||||||||||                                                                18.5M/119M] Load average: 0.63 0.61 0.55 
  Swp[                                                                                                     0K/0K] Uptime: 00:23:56

  [Main] [I/O]
  PID+USER       PRI  NI  VIRT   RES   SHR S  CPU% MEM%   TIME+  Command
    1 root        20   0  1676  1080   900 S   0.0  0.9  0:02.46 /sbin/procd
  593 ubus        20   0  1352   920   812 S   0.0  0.8  0:03.23 `- /sbin/ubusd
  594 root        20   0   996   736   700 S   0.0  0.6  0:00.04 `- /sbin/askfirst /usr/libexec/login.sh
  629 root        20   0  1108   728   676 S   0.0  0.6  0:00.45 `- /sbin/urngd
 1138 logd        20   0  1340   872   760 S   0.0  0.7  0:00.08 `- /sbin/logd -S 64
 1208 root        20   0  2236  1312  1040 S   0.0  1.1  0:19.06 `- /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
 1413 root        20   0  1208   952   908 S   0.0  0.8  0:00.00 `- /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3
 3140 root        20   0  1228   964   908 S   0.0  0.8  0:00.41 |  `- /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
 3171 root        20   0  1312  1052  1008 S   0.0  0.9  0:00.03 |  |  `- -ash
20868 root        20   0  1228   964   908 S   4.5  0.8  0:01.40 |  `- /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -p 22 -K 300 -T 3 -2 9
20869 root        20   0  1312  1056  1008 S   0.0  0.9  0:00.03 |     `- -ash
27568 root        20   0  2972  2720  1140 R  22.7  2.2  0:00.58 |        `- htop
 1517 root        20   0  2656  1036   940 S   0.0  0.8  0:00.00 `- /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1521 network     20   0  4436  1600  1488 S   0.0  1.3  0:00.26 |  `- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 1518 root        20   0  2656  1036   940 S   0.0  0.8  0:00.06 `- /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1544 network     20   0  4436  1764  1652 S   0.0  1.4  0:00.21 |  `- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
 1584 root        20   0  1828  1160   996 S   4.5  0.9  0:06.70 `- /sbin/netifd
 2166 root        20   0  1124   816   764 S   0.0  0.7  0:00.77 |  `- odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
27722 root        20   0  1524  1276  1008 S  31.8  1.0  0:00.29 |  |  `- /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated
27759 root        20   0  1344  1088  1008 S   4.5  0.9  0:00.01 |  |     `- /bin/sh /sbin/fw3 -q network wan6
27760 root        20   0  1316   996   944 S   0.0  0.8  0:00.00 |  |        `- flock -x /var/run/fw4.lock utpl -S /usr/share/firewall4/main.uc
27761 root        20   0  1340   908   824 R   0.0  0.7  0:00.00 |  |           `- utpl -S /usr/share/firewall4/main.uc
27758 root        20   0  1392  1120  1008 R   4.5  0.9  0:00.01 |  `- /bin/sh /sbin/hotplug-call iface
 1642 root        20   0  1544   992   880 S   4.5  0.8  0:07.23 `- /usr/sbin/odhcpd
 1763 root        20   0  4168  2244  2028 S   0.0  1.8  0:04.65 `- /usr/sbin/uhttpd -f -h /www -r OpenWrt -x /cgi-bin -u /ubus -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80 -C /etc/uhttpd.crt -K /etc/uhttpd.ke
 2193 root        20   0  2656  1040   940 S   0.0  0.9  0:00.01 `- /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/sha
 2196 ntp         20   0  1308   984   944 S   0.0  0.8  0:00.04 |  `- /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 0.openwrt.pool.ntp.org -p 1.openwrt.pool.ntp.org -p 2.openwrt.pool.ntp.org -p 3.openwrt.pool.ntp.org
 2471 root        20   0  2656  1040   940 S   0.0  0.9  0:00.01 `- /sbin/ujail -t 5 -n dnsmasq -u -l -r /bin/ubus -r /etc/TZ -r /etc/dnsmasq.conf -r /etc/ethers -r /etc/group -r /etc/hosts -r /etc/passwd -w /tmp/dhcp.leases -r
 2475 dnsmasq     20   0  1480  1052   944 S   0.0  0.9  0:02.42    `- /usr/sbin/dnsmasq -C /var/etc/dnsmasq.conf.cfg01411c -k -x /var/run/dnsmasq/dnsmasq.cfg01411c.pid
    2 root        20   0     0     0     0 S   0.0  0.0  0:00.00 kthreadd
    4 root         0 -20     0     0     0 I   0.0  0.0  0:00.04 `- kworker/0:0H-kblockd
    6 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- mm_percpu_wq
    7 root        20   0     0     0     0 S   0.0  0.0  0:00.00 `- rcu_tasks_trace
    8 root        20   0     0     0     0 S   0.0  0.0  0:01.71 `- ksoftirqd/0
    9 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- netns
   10 root        20   0     0     0     0 I   0.0  0.0  0:01.23 `- kworker/u2:1-events_unbound
   36 root        20   0     0     0     0 S   0.0  0.0  0:00.00 `- oom_reaper
   37 root        20   0     0     0     0 I   0.0  0.0  0:00.02 `- kworker/u2:2-events_unbound
   38 root        20   0     0     0     0 I   0.0  0.0  0:00.00 `- kworker/0:1-events
  125 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- writeback
  127 root        20   0     0     0     0 S   0.0  0.0  0:00.10 `- kcompactd0
  139 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- kblockd
  140 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- blkcg_punt_bio
  141 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 `- watchdogd
  175 root        20   0     0     0     0 S   0.0  0.0  0:00.00 `- kswapd0
  177 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- kthrotld
  222 root        20   0     0     0     0 S   0.0  0.0  0:00.23 `- spi0
  372 root        20   0     0     0     0 I   0.0  0.0  0:01.84 `- kworker/0:2-events_power_efficient
  374 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- ipv6_addrconf
  406 root         0 -20     0     0     0 I   0.0  0.0  0:07.35 `- kworker/0:1H-kblockd
  417 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 `- irq/16-keys
  418 root       -51   0     0     0     0 S   0.0  0.0  0:00.00 `- irq/23-keys
  532 root        30  10     0     0     0 S   0.0  0.0  0:00.01 `- jffs2_gcd_mtd4
  719 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- cfg80211
  755 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- ath10k_wq
  756 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- ath10k_aux_wq
  757 root         0 -20     0     0     0 I   0.0  0.0  0:00.00 `- ath10k_tx_compl

So this is likely not an issue with ds-lite.

I can isolate the router from my home network and sniff the traffic on the wan side until connectivity starts working after a restart when it works, and until some time runs out after a restart when it doesn't work. Would that help? How is best to share it? May it contain private data from the ISP or other clients that I should not publish?

hnyman · January 16, 2023, 3:28pm

madez:

 1584 root        20   0  1828  1160   996 S   4.5  0.9  0:06.70 `- /sbin/netifd
 2166 root        20   0  1124   816   764 S   0.0  0.7  0:00.77 |  `- odhcp6c -s /lib/netifd/dhcpv6.script -P0 -t120 eth0.2
27722 root        20   0  1524  1276  1008 S  31.8  1.0  0:00.29 |  |  `- /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated
27759 root        20   0  1344  1088  1008 S   4.5  0.9  0:00.01 |  |     `- /bin/sh /sbin/fw3 -q network wan6
27760 root        20   0  1316   996   944 S   0.0  0.8  0:00.00 |  |        `- flock -x /var/run/fw4.lock utpl -S /usr/share/firewall4/main.uc
27761 root        20   0  1340   908   824 R   0.0  0.7  0:00.00 |  |           `- utpl -S /usr/share/firewall4/main.uc
27758 root        20   0  1392  1120  1008 R   4.5  0.9  0:00.01 |  `- /bin/sh /sbin/hotplug-call iface

That part /bin/sh /lib/netifd/dhcpv6.script eth0.2 ra-updated looks like firewall refresh triggered by new RA (Router Advertisement) info from ISP.

I am not sure if a misconfigured ISP network sending too frequent RA traffic could cause your router to choke, in style of a DDoS attack

possbly dhcpv6 specialists like @dedeckeh might have clues for you.

madez · January 16, 2023, 9:04pm

Here you can see the rate of Router Advertisements I'm receiving:

And I get hammered with around 50 Neighbor Solicitations every second in bursts at a rate of 10k/s. The frequency of these bursts is remarkably stable. Here is one shown:

darksky · January 17, 2023, 10:57am

As a test, does the CPU load drop if you disconnect your WAN interface physically from the modem?

madez · January 17, 2023, 11:37am

Yes, with the cable to wan disconnected the CPU usage drops to very low numbers (<2%) and stays there.

darksky · January 17, 2023, 12:58pm

Nice, so at least you understand what is driving the high usage. I still don't understand why you are not shown which process is using the CPU in real time thought. Are you able to disable ipv6 or does your ISP require it? I have been running a ipv4 only setup without issue.

madez · January 17, 2023, 1:07pm

It is an IPv6 only connection. IPv4 is provided over ds-lite.

It is indeed not okay that I'm not shown what is using the CPU. That is a bug in itself.

darksky · January 17, 2023, 1:21pm

OK, please ignore my stupid questions