Adding OpenWrt support for Xiaomi AX3600 (Part 1)

sqrwv · 2022-06-26T16:33:41.635Z

It's 2.5-3Gbits full-duplex speed?
Probably you can do more now with GRO.
There will be more cpu load with multiple connections, but still impressive speed, with a low power consumption cpu, without NSS.

robimarko · 2022-06-26T17:02:07.998Z

If that was with NAT and routing then I am surprised it can do that.
But its still crap if that's with SW offloading.

Anyway, been looking if its possible to add the missing offloads but any kind of useful info is missing.

KingCZE · 2022-06-26T18:27:08.632Z

And this was on the 1 gig AX3600 LAN port?

robimarko · 2022-06-26T18:30:46.361Z

Off course not, he even wrote it was on the Qnap 301W which has two 10G ports

KingCZE · 2022-06-26T18:32:26.254Z

Ahh, clear, haha. I was very surprised lol

sqrwv · 2022-06-26T18:42:12.570Z

iperf3 --bidir already did 1Gbit full-duplex (dhcp)wan <-> lan speed in AX3600 (with packet stearing and NAT SW offload, before gro).

I'll do some wan-lan tests with gro (for personal interest) and will post results.

WAN<->WiredLAN
iperf3 full-duplex tests

@robimarko
[TL;DR]:
--> SW flow offloading: ON, Packet Steering: ON, gro ON/OFF:
gro ON gives me slower speed in WAN-LAN bidir, see test#2 ~912/740Mbits.
Same conditions but with gro off, see test#3 >930/930Mbits.
I also tested the previous build without gro implementation, see test#1 >930/930Mbits.
I didn't mistaken switched the option or tests, can't say what's going on.

--> SW flow offloading: OFF, Packet Steering: ON, gro ON/OFF:
In test#4 and test#5 we can see a lot of times that packet steering isn't working as it should (when I'm repeating exactly the same test), so comparing gro on/off is inconclusive in these 2 tests.

--> SW flow offloading: ON, Packet Steering: OFF, gro ON/OFF:
Still need to do two more test here, will do tomorrow or when possible and update here.

--> SW flow offloading: OFF, Packet Steering: OFF, gro ON/OFF:
Still need to do two more test here, will do tomorrow or when possible and update here.

Notes:
Both computers run latest mint (with kernel 5.15.0-33-generic #34~20.04.1-Ubuntu).
Results are from iperf3 client that runs in lan: iperf3 -c 192.168.x.x -t 30 --bidir
I had and still have IPV6 disabled in router.
(iperf3 is a test benchmark, not the real world with multiple users/connections.)

Test#1: Robi previous build (2022-06-22 - r0-a6d607b).
SW flow offloading: ON, Packet Steering: ON, GRO: Not implemented

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.26 GBytes   932 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec    0             sender
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   930 Mbits/sec                  receiver

(I run the test several times and it was always > 930Mbits.)

Test#2: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: ON, Packet Steering: ON, GRO: ON

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   913 Mbits/sec  415             sender
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   912 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  2.60 GBytes   744 Mbits/sec  631             sender
[  7][RX-C]   0.00-30.00  sec  2.60 GBytes   743 Mbits/sec                  receiver

(I run several times and it's always around the same values.
I revisited this test several times, rebooted router, made sure there's no other traffic in wired/wireless lan or wan.)

root@X2:~# ethtool -k eth0 | grep generic-receive-offload
generic-receive-offload: on
root@X2:~# ethtool -k eth1 | grep generic-receive-offload
generic-receive-offload: on
root@X2:~# ethtool -k eth2 | grep generic-receive-offload
generic-receive-offload: on
root@X2:~# ethtool -k eth3 | grep generic-receive-offload
generic-receive-offload: on

Test#3: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: ON, Packet Steering: ON, GRO: OFF (eth0, eth1, eth2, eth3)

[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.25 GBytes   932 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   932 Mbits/sec    1             sender
[  7][RX-C]   0.00-30.00  sec  3.25 GBytes   931 Mbits/sec                  receiver

(I run the test several times, results always > 930Mbits)

root@X2:~# ethtool -k eth0 | grep generic-receive-offload
generic-receive-offload: off
root@X2:~# ethtool -k eth1 | grep generic-receive-offload
generic-receive-offload: off
root@X2:~# ethtool -k eth2 | grep generic-receive-offload
generic-receive-offload: off
root@X2:~# ethtool -k eth3 | grep generic-receive-offload
generic-receive-offload: off

Test#4: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: OFF, Packet Steering: ON, GRO: ON
I get Inconsistent results: Not same speed in all runs of the same test.

- RUN#1 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.24 GBytes   927 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.23 GBytes   926 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec   877 MBytes   245 Mbits/sec  563             sender
[  7][RX-C]   0.00-30.00  sec   874 MBytes   244 Mbits/sec                  receiver
- RUN#2 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   913 Mbits/sec  222             sender
[  5][TX-C]   0.00-30.00  sec  3.19 GBytes   912 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  2.16 GBytes   619 Mbits/sec  292             sender
[  7][RX-C]   0.00-30.00  sec  2.16 GBytes   618 Mbits/sec                  receiver
- RUN#3 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.17 GBytes   908 Mbits/sec  228             sender
[  5][TX-C]   0.00-30.00  sec  3.17 GBytes   907 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  2.63 GBytes   753 Mbits/sec  297             sender
[  7][RX-C]   0.00-30.00  sec  2.63 GBytes   752 Mbits/sec                  receiver
- RUN#4 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.20 GBytes   916 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  3.20 GBytes   915 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec   585 MBytes   164 Mbits/sec  310             sender
[  7][RX-C]   0.00-30.00  sec   584 MBytes   163 Mbits/sec                  rec

Test#5: Robi latest build (2022-06-23 - r0-6b0cd97).
SW flow offloading: OFF, Packet Steering: ON, GRO: OFF
I get Inconsistent results: Not same speed in all runs of the same test.

- RUN#1 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  3.11 GBytes   891 Mbits/sec   83             sender
[  5][TX-C]   0.00-30.00  sec  3.11 GBytes   890 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  1.75 GBytes   500 Mbits/sec  210             sender
[  7][RX-C]   0.00-30.00  sec  1.74 GBytes   500 Mbits/sec                  receiver

- RUN#2 - - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  2.96 GBytes   849 Mbits/sec  584             sender
[  5][TX-C]   0.00-30.01  sec  2.96 GBytes   848 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec  3.06 GBytes   875 Mbits/sec  576             sender
[  7][RX-C]   0.00-30.01  sec  3.05 GBytes   874 Mbits/sec                  receiver

- RUN#3  - - - - - - - - - - - - - - - - - - - - - - - -
[ ID][Role] Interval           Transfer     Bitrate         Retr
[  5][TX-C]   0.00-30.00  sec  2.81 GBytes   804 Mbits/sec    0             sender
[  5][TX-C]   0.00-30.00  sec  2.81 GBytes   803 Mbits/sec                  receiver
[  7][RX-C]   0.00-30.00  sec   258 MBytes  72.0 Mbits/sec  540             sender
[  7][RX-C]   0.00-30.00  sec   256 MBytes  71.7 Mbits/sec                  receiver

Looks like packet steering sometimes doesn't work at all.

rmandrad · 2022-06-27T07:20:25.424Z

The ath11k uses REGULATORY_WIPHY_SELF_MANAGED (check the reg.c under the ath11k folder)

"A self-managed wiphys only employs regulatory information obtained from
the FW and driver and does not use other cfg80211 sources like
beacon-hints, country-code IEs and hints from other devices on the same
system. Conversely, a self-managed wiphy does not share its regulatory
hints with other devices in the system. If a system contains several
devices, one or more of which are self-managed, there might be
contradictory regulatory settings between them. Usage of flag is
generally discouraged. Only use it if the FW/driver is incompatible
with non-locally originated hints."

robimarko · 2022-06-27T09:42:29.452Z

Interesting, sysupgrade failed for me for the first time on Qnap since it could not kill hostapd.

root@OpenWrt:/# sysupgrade http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap
_301w-squashfs-sysupgrade.bin
Downloading 'http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap_301w-squashfs-sysupgrade.bin'
Connecting to 192.168.2.32:8000
Writing to '/tmp/sysupgrade.img'
/tmp/sysupgrade.img  100% |*******************************| 11900k  0:00:00 ETA
Download completed (12186393 bytes)
Mon Jun 27 09:39:36 UTC 2022 upgrade: Saving config files...
Mon Jun 27 09:39:36 UTC 2022 upgrade: Commencing upgrade. Closing all shell sessions.
[54226.481128] device wlan1 left promiscuous mode
[54226.481219] br-lan: port 6(wlan1) entered disabled state
Watchdog handover: fd=3
- watchdog -
Watchdog did not previously reset the system
Mon Jun 27 09:39:37 UTC 2022 upgrade: Sending TERM to remaining processes ...
Mon Jun 27 09:39:37 UTC 2022 upgrade: Sending signal TERM to hostapd (1652)
Mon Jun 27 09:39:37 UTC 2022 upgrade: Sending signal TERM to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending KILL to remaining processes ...
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1652)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Sending signal KILL to hostapd (1682)
Mon Jun 27 09:39:41 UTC 2022 upgrade: Failed to kill all processes.
sysupgrade aborted with return code: 256
[54231.291513] reboot: Restarting system

Second attempt worked:

root@OpenWrt:/# sysupgrade http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap
_301w-squashfs-sysupgrade.bin
Downloading 'http://192.168.2.32:8000/openwrt-ipq807x-generic-qnap_301w-squashfs-sysupgrade.bin'
Connecting to 192.168.2.32:8000
Writing to '/tmp/sysupgrade.img'
/tmp/sysupgrade.img  100% |*******************************| 11900k  0:00:00 ETA
Download completed (12186393 bytes)
Mon Jun 27 09:42:46 UTC 2022 upgrade: Saving config files...
Mon Jun 27 09:42:46 UTC 2022 upgrade: Commencing upgrade. Closing all shell sessions.
[  157.393575] device wlan1 left promiscuous mode
[  157.393692] br-lan: port 6(wlan1) entered disabled state
Watchdog handover: fd=3
- watchdog -
Watchdog did not previously reset the system
[  157.450212] ath11k c000000.wifi: No VIF found for vdev 0
[  157.450261] ath11k c000000.wifi: invalid vdev id 0 in bcn_tx_status
[  157.514293] device wlan0 left promiscuous mode
[  157.514384] br-lan: port 7(wlan0) entered disabled state
Mon Jun 27 09:42:46 UTC 2022 upgrade: Sending TERM to remaining processes ...
Mon Jun 27 09:42:46 UTC 2022 upgrade: Sending signal TERM to hostapd (1467)
Mon Jun 27 09:42:46 UTC 2022 upgrade: Sending signal TERM to hostapd (1497)
Mon Jun 27 09:42:50 UTC 2022 upgrade: Sending KILL to remaining processes ...
[  167.737549] stage2 (2688): drop_caches: 3
Mon Jun 27 09:42:56 UTC 2022 upgrade: Switching to ramdisk...
Mon Jun 27 09:42:58 UTC 2022 upgrade: Performing system upgrade...
flashing kernel to /dev/mmcblk0p1
flashing rootfs to /dev/mmcblk0p4
[  170.231918] loop0: detected capacity change from 0 to 1034112
Format new rootfs_data at position 7405568.
mke2fs 1.46.5 (30-Dec-2021)
/dev/loop0 contains a ext4 file system labelled 'rootfs_data'
        last mounted on / on Thu Jan  1 00:00:07 1970
Creating filesystem with 517056 1k blocks and 129024 inodes
Filesystem UUID: 30c82bf2-6ec9-4878-bf1a-26daaa2c3377
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

[  172.442441] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled.
Saving config to rootfs_data at position 7405568.
umount: can't unmount /dev: Resource busy
umount: can't unmount /tmp: Resource busy
[  172.691218] reboot: Restarting system

@Ansuel Check the VIF message

xabolcs · 2022-06-27T12:18:15.125Z

Sysupgrade runs as root?

It's just unbelievable that a program can survive a kill -9 executed from root.

# ps|grep hostapd
 1966 root      2664 S    {hostapd} /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
 2012 network   5004 S    /usr/sbin/hostapd -s -g /var/run/hostapd/global
13421 root      1312 S    grep hostapd

robimarko · 2022-06-27T12:18:51.280Z

Yeah, everything in OpenWrt runs as root.
SIGKILL should be able to kill everything except for the PID1

McGiverGim · 2022-06-27T12:59:30.302Z

I've found this in internet. I suppose there are more cases where the kill does not work:

One says kill -9 <pid> always works. That's a misbelief . There are situations where even kill -9 does not kill the process. For example when a process has the state D (uninterruptable sleep). A process comes into this state everytime it waits for I/O (normally not very long). So, if a process waits for I/O (on a defect harddisk for example) and it is not programmed properly (with a timeout), then you simply cannot kill the process . No matter what you do. You just can try to make the file accessible that the process continues.

sqrwv · 2022-06-27T13:24:52.418Z

@robimarko
AX3600 is my first multi-core router cpu, so I don't know what to expect about packet steering.
Packet steering not working as it should, is a stand alone issue?
I ask, so I don't address a issue here that is not related to AX3600 builds.

kirdes · 2022-06-27T14:01:17.560Z

maybe this is the root cause:

This is missing during the fist sysupgrade attempt:

device wlan0 left promiscuous mode

So wlan0 hasn't been terminated, maybe that's the reason why hostapd cannot be killed?

During the second sysupgrade attempt, both wifis have been terminated successfully.

robimarko · 2022-06-27T14:02:58.485Z

Well, hostapd is supposed to bring the WLAN interface down before exiting

Ansuel · 2022-06-27T14:03:19.304Z

@robimarko think we can repo the issue by just init a sysupgrade while we are stressing the wifi for example with an iperf... IMHO there is a problem with terminating the packet processing and this cause longer delay... (since ath11k works entirely on async ring handling)

guess ath11k still have problem with interface removal and can suffer of some corner case where the delay is very long

btw interesting patch
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/commit/?h=pending&id=ff1c40050ac3ca6a55ccc7970e37503e75465977

XiaoliChan · 2022-06-27T15:37:51.189Z

I build a firmware with the newest upstream nss_packages, I found this log about nss-dp

[ 83.543235] NAPI poll function edma_tx_napi+0x0/0xf0 [qca_nss_dp] returned 108, exceeding its budget of 64.

robimarko · 2022-06-27T15:38:59.758Z

That makes sense why it could not get killed.

Yeah, that patch is interesting, been in pending for some time, I will pick it instead of waiting to get merged

Ansuel · 2022-06-27T16:13:20.201Z

we still need to polish that but main intention is to check if it does improve anything...

sqrwv · 2022-06-27T16:16:51.189Z

I randomly found someone with the same results I tested, packet_stearing being faster than gro in NAT (different soc):
https://github.com/openwrt/openwrt/blob/c6f16b63fa7f3a44cd4a71d7e195d6046d612075/target/linux/bcm53xx/patches-5.15/600-net-disable-GRO-by-default.patch
Quote:

In many cases GRO improves network performance however it comes at a cost of chacksums calculations. In case of slow CPU and missing hardware csum calculation support GRO can actually decrease network speed.

On BCM4708 disabling GRO results in following NAT masquarade speed changes:

364 Mb/s → 396 Mb/s (packet steering disabled)

341 Mb/s → 566 Mb/s (packet steering enabled)

robimarko · 2022-06-27T16:19:37.907Z

Yeah, cause GRO will bunch up smaller packets to get processed as one big packet, but that does invalidate the checksum and it needs to be recalculated.
This ethernet adapter for sure has RX checksum offloading, but there is no clues how to use it but the CPU is way faster than the BCM4708 which is just old dual core A9.

In our case even without checksum offloading its faster