7812

This is last build

root@OpenWrt:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
 11:    1952638    4052698    1889271    1879997     GIC-0  20 Level     arch_timer
 13:          0          0          0          0     GIC-0  39 Level     arch_mem_timer
 16:          2          0          0          0     GIC-0 354 Edge      smp2p
 17:          0          0          0          0     GIC-0 216 Level     4a9000.thermal-sensor
 18:          0          0          0          0     GIC-0 239 Level     bam_dma
 21:          0          0          0          0     GIC-0 270 Level     bam_dma
 22:          4          0          0          0     GIC-0 340 Level     msm_serial0
 23:      49461          0          0          0     GIC-0 178 Level     bam_dma
 24:          0          0          0          0     GIC-0  35 Edge      wdt_bark
 25:          0          0          0          0     GIC-0 357 Edge      q6v5 wdog
 29:          5          0          0          0     GIC-0 348 Edge      ce0
 30:   11629887          0          0          0     GIC-0 347 Edge      ce1
 31:    4824296          0          0          0     GIC-0 346 Edge      ce2
 32:     333442          0          0          0     GIC-0 343 Edge      ce3
 34:       3745          0          0          0     GIC-0 443 Edge      ce5
 36:     835083          0          0          0     GIC-0  72 Edge      ce7
 38:          0          0          0          0     GIC-0 334 Edge      ce9
 39:          1          0          0          0     GIC-0 333 Edge      ce10
 40:          0          0          0          0     GIC-0  69 Edge      ce11
 47:          0          0          0          0     GIC-0 323 Edge      reo2ost-exception
 48:      62546          0          0          0     GIC-0 322 Edge      wbm2host-rx-release
 49:       1353          0          0          0     GIC-0 321 Edge      reo2host-status
 50:     283373          0          0          0     GIC-0 320 Edge      reo2host-destination-ring4
 51:     321557          0          0          0     GIC-0 271 Edge      reo2host-destination-ring3
 52:     299514          0          0          0     GIC-0 268 Edge      reo2host-destination-ring2
 53:     257816          0          0          0     GIC-0 267 Edge      reo2host-destination-ring1
 57:    6158191          0          0          0     GIC-0 263 Edge      ppdu-end-interrupts-mac3
 58:          0          0          0          0     GIC-0 262 Edge      ppdu-end-interrupts-mac2
 59:    1499298          0          0          0     GIC-0 261 Edge      ppdu-end-interrupts-mac1
 60:          1          0          0          0     GIC-0 260 Edge      rxdma2host-monitor-status-ring-mac3
 61:          0          0          0          0     GIC-0 256 Edge      rxdma2host-monitor-status-ring-mac2
 62:          1          0          0          0     GIC-0 255 Edge      rxdma2host-monitor-status-ring-mac1
 63:          1          0          0          0     GIC-0 235 Edge      host2rxdma-host-buf-ring-mac3
 64:          0          0          0          0     GIC-0 215 Edge      host2rxdma-host-buf-ring-mac2
 65:          1          0          0          0     GIC-0 212 Edge      host2rxdma-host-buf-ring-mac1
 66:          0          0          0          0     GIC-0 211 Edge      rxdma2host-destination-ring-mac3
 67:          0          0          0          0     GIC-0 210 Edge      rxdma2host-destination-ring-mac2
 68:          0          0          0          0     GIC-0 209 Edge      rxdma2host-destination-ring-mac1
 73:      55784          0          0          0     GIC-0 191 Edge      wbm2host-tx-completions-ring3
 74:      82920          0          0          0     GIC-0 190 Edge      wbm2host-tx-completions-ring2
 75:    1081775          0          0          0     GIC-0 189 Edge      wbm2host-tx-completions-ring1
 77:         17          0          0          0     GIC-0  47 Edge      cpr3
 78:   24918870          0          0          0     GIC-0 377 Level     edma_txcmpl
 79:          0          0          0          0     GIC-0 385 Level     edma_rxfill
 80:   28492112          0          0          0     GIC-0 393 Level     edma_rxdesc
 81:          0          0          0          0     GIC-0 376 Level     edma_misc
 82:          0          0          0          0       MSI   0 Edge      PCIe PME, aerdrv
 95:          0          0          0          0  pmic_arb 51380237 Edge      pm-adc5
 96:          0          0          0          0     smp2p   0 Edge      q6v5 fatal
 97:          1          0          0          0     smp2p   1 Edge      q6v5 ready
 98:          0          0          0          0     smp2p   2 Edge      q6v5 handover
 99:          0          0          0          0     smp2p   3 Edge      q6v5 stop
100:          0          0          0          0   msmgpio  34 Edge      keys
101:         31          0          0          0       MSI 524288 Edge      ath10k_pci
102:         64          0          0          0     GIC-0 353 Edge      glink-native
IPI0:   1209233    8408649    4275022    3218285       Rescheduling interrupts
IPI1:     35427      50830      38942      56419       Function call interrupts
IPI2:         0          0          0          0       CPU stop interrupts
IPI3:         0          0          0          0       CPU stop (for crash dump) interrupts
IPI4:         0          0          0          0       Timer broadcast interrupts
IPI5:      1968        587        566        580       IRQ work interrupts
IPI6:         0          0          0          0       CPU wake-up interrupts
7813

and do you have ax3600 or ...?

7814

Sorry.
Yes, Ax3600

7815

strange... I've irqs shifted by 1 on mine :-/

7816

As per my test with qnap, 2.5-3GB is reachable without NSS cores

1 Like
7817

Could you please elaborate on your test conditions:
It's full-duplex WAN-LAN speed ?
It's the latest FW with GRO and you have Packet stearing ON, router/nat SW offload ON ?

Thanks

1 Like
7818

sw offload and packet steering, wan-lan of course, an older image, not the new one
but the soc is a bit more powerfull than this one, with nss on original fw it goes to ~9,6GB, tested on VM with dell m640 hosts with MXL 10/40GbE blade switch

7819

It's 2.5-3Gbits full-duplex speed?
Probably you can do more now with GRO.
There will be more cpu load with multiple connections, but still impressive speed, with a low power consumption cpu, without NSS.

7820

If that was with NAT and routing then I am surprised it can do that.
But its still crap if that's with SW offloading.

Anyway, been looking if its possible to add the missing offloads but any kind of useful info is missing.

1 Like
7821

And this was on the 1 gig AX3600 LAN port? :open_mouth:

7822

Off course not, he even wrote it was on the Qnap 301W which has two 10G ports

7823

Ahh, clear, haha. I was very surprised lol

7824

iperf3 --bidir already did 1Gbit full-duplex (dhcp)wan <-> lan speed in AX3600 (with packet stearing and NAT SW offload, before gro).

I'll do some wan-lan tests with gro (for personal interest) and will post results.

WAN<->WiredLAN
iperf3 full-duplex tests

@robimarko
[TL;DR]:
--> SW flow offloading: ON, Packet Steering: ON, gro ON/OFF:
gro ON gives me slower speed in WAN-LAN bidir, see test#2 ~912/740Mbits.
Same conditions but with gro off, see test#3 >930/930Mbits.
I also tested the previous build without gro implementation, see test#1 >930/930Mbits.
I didn't mistaken switched the option or tests, can't say what's going on.

--> SW flow offloading: OFF, Packet Steering: ON, gro ON/OFF:
In test#4 and test#5 we can see a lot of times that packet steering isn't working as it should (when I'm repeating exactly the same test), so comparing gro on/off is inconclusive in these 2 tests.

--> SW flow offloading: ON, Packet Steering: OFF, gro ON/OFF:
Still need to do two more test here, will do tomorrow or when possible and update here.

--> SW flow offloading: OFF, Packet Steering: OFF, gro ON/OFF:
Still need to do two more test here, will do tomorrow or when possible and update here.

Notes:
Both computers run latest mint (with kernel 5.15.0-33-generic #34~20.04.1-Ubuntu).
Results are from iperf3 client that runs in lan: iperf3 -c 192.168.x.x -t 30 --bidir
I had and still have IPV6 disabled in router.
(iperf3 is a test benchmark, not the real world with multiple users/connections.)

Test#1: Robi previous build (2022-06-22 - r0-a6d607b).
SW flow offloading: ON, Packet Steering: ON, GRO: Not implemented

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.26 GBytes   932 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec    0             sender
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   930 Mbits/sec                  receiver

(I run the test several times and it was always > 930Mbits.)
image

Test#2: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: ON, Packet Steering: ON, GRO: ON

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   913 Mbits/sec  415             sender
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   912 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  2.60 GBytes   744 Mbits/sec  631             sender
[  7][RX-C]   0.00-30.00  sec  2.60 GBytes   743 Mbits/sec                  receiver

(I run several times and it's always around the same values.
I revisited this test several times, rebooted router, made sure there's no other traffic in wired/wireless lan or wan.)
image

root@X2:~# ethtool -k eth0 | grep generic-receive-offload
generic-receive-offload: on
root@X2:~# ethtool -k eth1 | grep generic-receive-offload
generic-receive-offload: on
root@X2:~# ethtool -k eth2 | grep generic-receive-offload
generic-receive-offload: on
root@X2:~# ethtool -k eth3 | grep generic-receive-offload
generic-receive-offload: on

Test#3: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: ON, Packet Steering: ON, GRO: OFF (eth0, eth1, eth2, eth3)

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.25 GBytes   932 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   932 Mbits/sec    1             sender
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec                  receiver

(I run the test several times, results always > 930Mbits)
image

root@X2:~# ethtool -k eth0 | grep generic-receive-offload
generic-receive-offload: off
root@X2:~# ethtool -k eth1 | grep generic-receive-offload
generic-receive-offload: off
root@X2:~# ethtool -k eth2 | grep generic-receive-offload
generic-receive-offload: off
root@X2:~# ethtool -k eth3 | grep generic-receive-offload
generic-receive-offload: off

Test#4: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: OFF, Packet Steering: ON, GRO: ON
I get Inconsistent results: Not same speed in all runs of the same test.

- RUN#1 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.24 GBytes   927 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.23 GBytes   926 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec   877 MBytes   245 Mbits/sec  563             sender
[  7][RX-C]   0.00-30.00  sec   874 MBytes   244 Mbits/sec                  receiver
- RUN#2 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   913 Mbits/sec  222             sender
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   912 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  2.16 GBytes   619 Mbits/sec  292             sender
[  7][RX-C]   0.00-30.00  sec  2.16 GBytes   618 Mbits/sec                  receiver
- RUN#3 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.17 GBytes   908 Mbits/sec  228             sender
[  5][TX-C]   0.00-30.00  sec  3.17 GBytes   907 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  2.63 GBytes   753 Mbits/sec  297             sender
[  7][RX-C]   0.00-30.00  sec  2.63 GBytes   752 Mbits/sec                  receiver
- RUN#4 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.20 GBytes   916 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.20 GBytes   915 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec   585 MBytes   164 Mbits/sec  310             sender
[  7][RX-C]   0.00-30.00  sec   584 MBytes   163 Mbits/sec                  rec

Test#5: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: OFF, Packet Steering: ON, GRO: OFF
I get Inconsistent results: Not same speed in all runs of the same test.

- RUN#1 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.11 GBytes   891 Mbits/sec   83             sender
[  5][TX-C]   0.00-30.00  sec  3.11 GBytes   890 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  1.75 GBytes   500 Mbits/sec  210             sender
[  7][RX-C]   0.00-30.00  sec  1.74 GBytes   500 Mbits/sec                  receiver

image

- RUN#2 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  2.96 GBytes   849 Mbits/sec  584             sender
[  5][TX-C]   0.00-30.01  sec  2.96 GBytes   848 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  3.06 GBytes   875 Mbits/sec  576             sender
[  7][RX-C]   0.00-30.01  sec  3.05 GBytes   874 Mbits/sec                  receiver

image

- RUN#3  - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  2.81 GBytes   804 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  2.81 GBytes   803 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec   258 MBytes  72.0 Mbits/sec  540             sender
[  7][RX-C]   0.00-30.00  sec   256 MBytes  71.7 Mbits/sec                  receiver

image

Looks like packet steering sometimes doesn't work at all.

2 Likes
7825

The ath11k uses REGULATORY_WIPHY_SELF_MANAGED (check the reg.c under the ath11k folder)

"A self-managed wiphys only employs regulatory information obtained from
the FW and driver and does not use other cfg80211 sources like
beacon-hints, country-code IEs and hints from other devices on the same
system. Conversely, a self-managed wiphy does not share its regulatory
hints with other devices in the system. If a system contains several
devices, one or more of which are self-managed, there might be
contradictory regulatory settings between them. Usage of flag is
generally discouraged. Only use it if the FW/driver is incompatible
with non-locally originated hints."

7826

Interesting, sysupgrade failed for me for the first time on Qnap since it could not kill hostapd.

root@OpenWrt:/# sysupgrade http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap
_301w-squashfs-sysupgrade.bin
Downloading 'http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap_301w-squashfs-sysupgrade.bin'
Connecting to 192.168.2.32:8000
Writing to '/tmp/sysupgrade.img'
/tmp/sysupgrade.img  100% |*******************************| 11900k  0:00:00 ETA
Download completed (12186393 bytes)
Mon Jun 27 09:39:36 UTC 2022 upgrade: Saving config files...
Mon Jun 27 09:39:36 UTC 2022 upgrade: Commencing upgrade. Closing all shell sessions.
[54226.481128] device wlan1 left promiscuous mode
[54226.481219] br-lan: port 6(wlan1) entered disabled state
Watchdog handover: fd=3
- watchdog -
Watchdog did not previously reset the system
Mon Jun 27 09:39:37 UTC 2022 upgrade: Sending TERM to remaining processes ...
Mon Jun 27 09:39:37 UTC 2022 upgrade: Sending signal TERM to hostapd (1652)
Mon Jun 27 09:39:37 UTC 2022 upgrade: Sending signal TERM to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending KILL to remaining processes ...
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1652)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Failed to kill all processes.
sysupgrade aborted with return code: 256
[54231.291513] reboot: Restarting system

Second attempt worked:

root@OpenWrt:/# sysupgrade http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap
_301w-squashfs-sysupgrade.bin
Downloading 'http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap_301w-squashfs-sysupgrade.bin'
Connecting to 192.168.2.32:8000
Writing to '/tmp/sysupgrade.img'
/tmp/sysupgrade.img  100% |*******************************| 11900k  0:00:00 ETA
Download completed (12186393 bytes)
Mon Jun 27 09:42:46 UTC 2022 upgrade: Saving config files...
Mon Jun 27 09:42:46 UTC 2022 upgrade: Commencing upgrade. Closing all shell sessions.
[  157.393575] device wlan1 left promiscuous mode
[  157.393692] br-lan: port 6(wlan1) entered disabled state
Watchdog handover: fd=3
- watchdog -
Watchdog did not previously reset the system
[  157.450212] ath11k c000000.wifi: No VIF found for vdev 0
[  157.450261] ath11k c000000.wifi: invalid vdev id 0 in bcn_tx_status
[  157.514293] device wlan0 left promiscuous mode
[  157.514384] br-lan: port 7(wlan0) entered disabled state
Mon Jun 27 09:42:46 UTC 2022 upgrade: Sending TERM to remaining processes ...
Mon Jun 27 09:42:46 UTC 2022 upgrade: Sending signal TERM to hostapd (1467)
Mon Jun 27 09:42:46 UTC 2022 upgrade: Sending signal TERM to hostapd (1497)
Mon Jun 27 09:42:50 UTC 2022 upgrade: Sending KILL to remaining processes ...
[  167.737549] stage2 (2688): drop_caches: 3
Mon Jun 27 09:42:56 UTC 2022 upgrade: Switching to ramdisk...
Mon Jun 27 09:42:58 UTC 2022 upgrade: Performing system upgrade...
flashing kernel to /dev/mmcblk0p1
flashing rootfs to /dev/mmcblk0p4
[  170.231918] loop0: detected capacity change from 0 to 1034112
Format new rootfs_data at position 7405568.
mke2fs 1.46.5 (30-Dec-2021)
/dev/loop0 contains a ext4 file system labelled 'rootfs_data'
        last mounted on / on Thu Jan  1 00:00:07 1970
Creating filesystem with 517056 1k blocks and 129024 inodes
Filesystem UUID: 30c82bf2-6ec9-4878-bf1a-26daaa2c3377
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

[  172.442441] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
Saving config to rootfs_data at position 7405568.
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource busy
[  172.691218] reboot: Restarting system

@Ansuel Check the VIF message

1 Like
7827

Sysupgrade runs as root?

It's just unbelievable that a program can survive a kill -9 executed from root.

# ps|grep hostapd
 1966 root      2664 S    {hostapd} /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 2012 network   5004 S    /usr/sbin/hostapd -s -g /var/run/hostapd/global
13421 root      1312 S    grep hostapd
1 Like
7828

Yeah, everything in OpenWrt runs as root.
SIGKILL should be able to kill everything except for the PID1

2 Likes
7829

I've found this in internet. I suppose there are more cases where the kill does not work:

One says kill -9 <pid> always works. That's a misbelief . There are situations where even kill -9 does not kill the process. For example when a process has the state D (uninterruptable sleep). A process comes into this state everytime it waits for I/O (normally not very long). So, if a process waits for I/O (on a defect harddisk for example) and it is not programmed properly (with a timeout), then you simply cannot kill the process . No matter what you do. You just can try to make the file accessible that the process continues.

2 Likes
7830

@robimarko
AX3600 is my first multi-core router cpu, so I don't know what to expect about packet steering.
Packet steering not working as it should, is a stand alone issue?
I ask, so I don't address a issue here that is not related to AX3600 builds.

7831

maybe this is the root cause:

This is missing during the fist sysupgrade attempt:

device wlan0 left promiscuous mode

So wlan0 hasn't been terminated, maybe that's the reason why hostapd cannot be killed?

During the second sysupgrade attempt, both wifis have been terminated successfully.