Adding OpenWrt support for Xiaomi AX3600 (Part 1)

robimarko · October 30, 2022, 8:36pm

I am doing some testing with NAPI SKB receive lists, it does certainly improve RX performance decently, but GRO is still way, way more efficient.

netif_receive_skb_list implemented:

-----------------------------------------------------------
Server listening on 5201 (test #9)
-----------------------------------------------------------
Accepted connection from 192.168.1.1, port 52960
[  5] local 192.168.1.104 port 5201 connected to 192.168.1.1 port 52974
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   181 MBytes  1.52 Gbits/sec    0    807 KBytes       
[  5]   1.00-2.00   sec   176 MBytes  1.48 Gbits/sec    0    857 KBytes       
[  5]   2.00-3.00   sec   179 MBytes  1.50 Gbits/sec    0    857 KBytes       
[  5]   3.00-4.00   sec   178 MBytes  1.49 Gbits/sec    0    898 KBytes       
[  5]   4.00-5.00   sec   179 MBytes  1.50 Gbits/sec    0    898 KBytes       
[  5]   5.00-6.00   sec   179 MBytes  1.50 Gbits/sec    0    898 KBytes       
[  5]   6.00-7.00   sec   178 MBytes  1.49 Gbits/sec    0    898 KBytes       
[  5]   7.00-8.00   sec   179 MBytes  1.50 Gbits/sec    0    898 KBytes       
[  5]   8.00-9.00   sec   178 MBytes  1.49 Gbits/sec    0    898 KBytes       
[  5]   9.00-10.00  sec   179 MBytes  1.50 Gbits/sec    0    898 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.74 GBytes  1.50 Gbits/sec    0             sender

Plain old netif_receive_skb:

-----------------------------------------------------------
Server listening on 5201 (test #10)
-----------------------------------------------------------
Accepted connection from 192.168.1.1, port 53040
[  5] local 192.168.1.104 port 5201 connected to 192.168.1.1 port 53046
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   161 MBytes  1.35 Gbits/sec    0    814 KBytes       
[  5]   1.00-2.00   sec   156 MBytes  1.31 Gbits/sec    0    854 KBytes       
[  5]   2.00-3.00   sec   156 MBytes  1.31 Gbits/sec    0    895 KBytes       
[  5]   3.00-4.00   sec   154 MBytes  1.29 Gbits/sec    0    895 KBytes       
[  5]   4.00-5.00   sec   155 MBytes  1.30 Gbits/sec    0    895 KBytes       
[  5]   5.00-6.00   sec   159 MBytes  1.33 Gbits/sec    0    895 KBytes       
[  5]   6.00-7.00   sec   158 MBytes  1.32 Gbits/sec    0    895 KBytes       
[  5]   7.00-8.00   sec   158 MBytes  1.32 Gbits/sec    0    895 KBytes       
[  5]   8.00-9.00   sec   160 MBytes  1.34 Gbits/sec    0    895 KBytes       
[  5]   9.00-10.00  sec   158 MBytes  1.32 Gbits/sec    0    895 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.54 GBytes  1.32 Gbits/sec    0             sender

GRO implemented:

-----------------------------------------------------------
Server listening on 5201 (test #11)
-----------------------------------------------------------
Accepted connection from 192.168.1.1, port 39150
[  5] local 192.168.1.104 port 5201 connected to 192.168.1.1 port 39162
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   315 MBytes  2.64 Gbits/sec    0   1.01 MBytes       
[  5]   1.00-2.00   sec   316 MBytes  2.65 Gbits/sec    0   1.01 MBytes       
[  5]   2.00-3.00   sec   318 MBytes  2.66 Gbits/sec    0   1.06 MBytes       
[  5]   3.00-4.00   sec   318 MBytes  2.66 Gbits/sec    0   1.11 MBytes       
[  5]   4.00-5.00   sec   318 MBytes  2.66 Gbits/sec    0   1.11 MBytes       
[  5]   5.00-6.00   sec   319 MBytes  2.67 Gbits/sec    0   1.17 MBytes       
[  5]   6.00-7.00   sec   316 MBytes  2.65 Gbits/sec    0   1.17 MBytes       
[  5]   7.00-8.00   sec   304 MBytes  2.55 Gbits/sec    0   1.17 MBytes       
[  5]   8.00-9.00   sec   310 MBytes  2.60 Gbits/sec    0   1.22 MBytes       
[  5]   9.00-10.00  sec   316 MBytes  2.65 Gbits/sec    0   1.22 MBytes       
[  5]  10.00-10.00  sec  1.25 MBytes  2.91 Gbits/sec    0   1.22 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.08 GBytes  2.64 Gbits/sec    0             sender

GRO implemented, but disabled via ethtool:

-----------------------------------------------------------
Server listening on 5201 (test #3)
-----------------------------------------------------------
Accepted connection from 192.168.1.1, port 53568
[  5] local 192.168.1.104 port 5201 connected to 192.168.1.1 port 53574
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   138 MBytes  1.16 Gbits/sec    0    772 KBytes       
[  5]   1.00-2.00   sec   135 MBytes  1.13 Gbits/sec    0    772 KBytes       
[  5]   2.00-3.00   sec   136 MBytes  1.14 Gbits/sec    0    772 KBytes       
[  5]   3.00-4.00   sec   138 MBytes  1.15 Gbits/sec    0    772 KBytes       
[  5]   4.00-5.00   sec   138 MBytes  1.15 Gbits/sec    0    772 KBytes       
[  5]   5.00-6.00   sec   138 MBytes  1.15 Gbits/sec    0    772 KBytes       
[  5]   6.00-7.00   sec   136 MBytes  1.14 Gbits/sec    0    772 KBytes       
[  5]   7.00-8.00   sec   135 MBytes  1.13 Gbits/sec    0    772 KBytes       
[  5]   8.00-9.00   sec   135 MBytes  1.13 Gbits/sec    0    772 KBytes       
[  5]   9.00-10.00  sec   138 MBytes  1.15 Gbits/sec    0    772 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  1.33 GBytes  1.15 Gbits/sec    0             sender

All tests were done on Qnap 301W on a 10G link to my PC using iperf3 in reverse mode as the 301W being the client (server sends, client receives)

robimarko · October 30, 2022, 8:45pm

@dchard It finally hit me why you see the reduced performance with GRO and PPoE.
That is because due to PPoE header, once its stripped checksum needs to be recalculated and since that is not offloaded CPU has to do it for PPoE packets like usual and then due to GRO batching packets checksum needs to be recalculated again.

Basically, it gets worse with GRO due to PPoE.

dchard · October 30, 2022, 8:59pm

Can you implement GRO with avoiding the recalculation? Or would the NAPI approach play better with PPPoE?

robimarko · October 30, 2022, 9:04pm

No, you can't avoid checksum recalculation as GRO works by batching packets to do processing in one go.

For now a non GRO improvement would be implementation of netif_receive_skb_list.
I am now fixing up the GRO implementation to allow for it to be properly disabled via ethtool and simply use netif_receive_skb then.

GRO fixup has been pushed, it now properly disables GRO with ethtool.

anon54515283 · October 30, 2022, 9:14pm

Did a new build with your latest commit (02d341496b121354cb3862ad6d550707ebe1d32e). Both units are booting fine without any issue like above. Booted each unit 3 times, just to be sure.

helicida · October 31, 2022, 12:00am

Installed and everything is working fine at the moment. Thank you!

anon75569510 · October 31, 2022, 4:30am

hows the range of this device?

dchard · October 31, 2022, 10:24am

@robimarko if the firmware is crashing, is there a way to get the crash dump or some sort of log out of it? I know we cannot do anything with it, but it maybe interesting for QCA (in case they respond to any of my requests....).

robimarko · October 31, 2022, 10:25am

You are already getting the trace when Q6 crashes, its quite sparse but thats about it.

There is also QDSS which should allow dumping a full trace but I have no idea how to implement that in mainline kernel

webm · October 31, 2022, 10:34am

Sorry for bothering and excuse my stupidity but I haven't flashed both partitions and now after sysupgrading I bricked my ax3600. Only orange system light.

What can I do?

P.S.
Nevermind, I manage to fix it with this MIWIFIRepairTool

Xiaomi Mi Router 4A Gigabit Edition (R4AG/R4A Gigabit) -- fully supported and flashable with OpenWRTInvasion - For Developers - OpenWrt Forum

AndyMcSchopf · October 31, 2022, 11:05am

Try the TFTP recovery from https://openwrt.org/inbox/toh/xiaomi/xiaomi_ax3600... That always worked for me.

webm · October 31, 2022, 11:26am

I fix it with MIWIFIRepairTool

iwill · October 31, 2022, 1:53pm

Thank you~~~~~~~

dimfish · October 31, 2022, 4:05pm

Yep we need to setup compat_version in uci-defaults.

I've added this in new version 5.15.76 r21182

Initialize device compat_version

To perform correct sysupgrade be sure that you are on version with extended partitions and run:

uci set system.@system[0].compat_version='2.0'
uci commit

PS. It also means that any old Backup can be restored and then updated with command above.

See original Increased Partitions Post.

Ansuel · October 31, 2022, 4:18pm

Now that I think about it with the updated layout in theory someone can migrate to it without losing anything. Am I wrong?

dimfish · October 31, 2022, 4:21pm

Yep, I've already checked this on my devices:

Generate Backup archive
Flash new partition layouts
Restore Backup.
Run uci set system.@system[0].compat_version='2.0'

LOS · October 31, 2022, 5:38pm

Hmm, good question but it should be possible, some conversion image can be created and flashed probably without having to restore backup.
As you have already made mem-increase-patch ubinized its very good start.

Ansuel · October 31, 2022, 5:54pm

since the partition layout is just the same (just second partition increased and repurposed)
in theory someone should be able to enter tempfs (like what is done in sysupgrade)

create the backup file...
flash the 2 factory bin
mount the second partition
apply the backup file
reboot
resize the rootfs partition to the end of the nand (no idea if this step is required in tmpfs like done in sysupgrade or not... but can also be done in a script

In theory this should work... but honestly it should be better to just ignore all these advanced step and flash initramfs, flash the 2 bin, reboot and restore the backup... there are plenty of way to migrate as we are just repurposing a partition as rootfs (and declaring fixed-partition to expand it to the end of the nand) nothing more.

Transportator · November 1, 2022, 3:56pm

It is awesome to see the progress and how it is usable and daily drivable on an AX3600. Thanks for all of the progress done my the users
As a newbie in OpenWRT (I am an ex Asus Merlin user) I had to use videos and read some tutorials to install and use OpenWRT.

Now using it for several days am I pretry happy about it.
But I still have some questions about OpenWRT for AX3600.

How can I update the firmware/software?
Just download the sysupgrade.bin and upload it via Luci GUI update option?
In the tutorial / post I read someone recommended me this fork:

Any differences between the original Robimarko version and this?
Can I just install/switch to the original version?

Again thanks for the work you all of have done.

juliank · November 1, 2022, 11:24pm

It's based on Robimarko's with few more packages included.

You can see here what packages are in the build:

https://github.com/ns2switch/ipq807x-openwrt-builder/releases/download/ipq807x-latest%40v5.6/config.buildinfo

Generally speaking, even though some release is based on Robimarko's repo it could potentially cocontain changes that are malicious so make sure you trust the source.