Adding OpenWrt support for Xiaomi AX3600 (Part 1)

robimarko · December 31, 2021, 12:09pm

I already fixed SSDK to compile in 5.15, its waiting for me to add the missing resets to the clock driver that it uses.

robimarko · January 1, 2022, 6:41pm

@dchard I completely missed your post.
Yeah, it's gotta be related to the DP buffer size as QCA uses 8k for 512M instead of the default 32k.
I will look into adding this in the same fashion as peer/vdev limits were added but only 5.15 is up and running.

Since I basically did 5.15 from scratch, a lot of stupid hacks and bugs were found, especially when it comes to the PCI and now the clock driver.
I basically looked at what was being actually done by QCA and found various stupid stuff that was just applied with their patches instead of being done properly.
Now it's way cleaner, it's missing the NSS crypto PLL, but I don't care about that now.

I am now working on the wired networking, SSDK looks to be working, NSS-DP is next so this is usable.
But the NSS offloading patches, I won't include those at all as they are all over the place and hacky AF, those basically need to be redone from scratch.

Shockingly, the NSS-DP just works in 5.15 so the branch will be usable (Devices need to be added) after I push the changes.

dchard · January 1, 2022, 9:05pm

Its a lot on your plate (not that it is anything new), the DP buffer sizes are the least of your problem

So, if I get it correctly: wired networking works (minus the offload capabilities), wifi works, so if I can live without PPPoE offload, I can start testing it?

robimarko · January 1, 2022, 9:07pm

Yeah, everything except for the offloading should work.
Everything except for the networking DTSI and the actual device support has been pushed, I hope to include the devices today and then it will work without offloading.

I didn't want to push device support as there have been changes to how networking nodes are added etc

dchard · January 1, 2022, 9:09pm

OK, will wait for that push, and then start building it.

robimarko · January 1, 2022, 9:35pm

Most probably wont happen today, its kind of painful re-adding devices

robimarko · January 1, 2022, 11:19pm

@dchard Ok, so I actually imported the devices and reworked them to work with newer DTS.
I have only runtime tested AX9000 but they are all pretty similar.

Don't forget to include the NSS-DP package as it's not pulled in by default.

dchard · January 1, 2022, 11:29pm

Thanks! It seems Openwrt GIT server is down, so I will build it tomorrow. Lets hope it will be back soon.

robimarko · January 1, 2022, 11:32pm

Yeah, IPv4 is broken on it, I was following the IRC to see whether they will figure out whats wrong but no luck yet, hopefully, its revived tomorrow.

BTW, the firewall doesn't work in 5.15, you can find details about it in the 5.15 PR.

dchard · January 2, 2022, 9:59am

The qca-nss-drv fails to build. Do I need to disable that part for now?

  CC [M]  /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_bridge.o
In file included from /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_hal/include/nss_hal.h:26,
                 from /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_tx_rx_common.h:25,
                 from /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_bridge.c:17:
/home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_core.h: In function 'nss_core_dma_cache_maint':
/home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_core.h:116:17: error: implicit declaration of function 'dmac_inv_range' [-Werror=implicit-function-declaration]
  116 |                 dmac_inv_range(start, start + size);
      |                 ^~~~~~~~~~~~~~
/home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_core.h:119:17: error: implicit declaration of function 'dmac_clean_range' [-Werror=implicit-function-declaration]
  119 |                 dmac_clean_range(start, start + size);
      |                 ^~~~~~~~~~~~~~~~
/home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_core.h:122:17: error: implicit declaration of function 'dmac_flush_range' [-Werror=implicit-function-declaration]
  122 |                 dmac_flush_range(start, start + size);
      |                 ^~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[5]: *** [scripts/Makefile.build:277: /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/nss_bridge.o] Error 1
make[4]: *** [Makefile:1868: /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81] Error 2
make[4]: Leaving directory '/home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/linux-5.15.10'
make[3]: *** [Makefile:122: /home/aaa/xiaomi/openwrt/build_dir/target-aarch64_cortex-a53_musl/linux-ipq807x_generic/qca-nss-drv-2021-04-26-1cf4bf81/.built] Error 2
make[3]: Leaving directory '/home/aaa/xiaomi/openwrt/feeds/nss_packages/qca/qca-nss-drv'
time: package/feeds/nss_packages/qca-nss-drv/compile#1.02#0.18#1.18
    ERROR: package/feeds/nss_packages/qca-nss-drv failed to build.
make[2]: *** [package/Makefile:116: package/feeds/nss_packages/qca-nss-drv/compile] Error 1
make[2]: Leaving directory '/home/aaa/xiaomi/openwrt'
make[1]: *** [package/Makefile:110: /home/aaa/xiaomi/openwrt/staging_dir/target-aarch64_cortex-a53_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/aaa/xiaomi/openwrt'
make: *** [/home/aaa/xiaomi/openwrt/include/toplevel.mk:230: world] Error 2

robimarko · January 2, 2022, 10:08am

NSS-DRV is not working yet and you dont need it as there is no offloading.
You just need NSS-DP which will pull in SSDK as well

dchard · January 2, 2022, 10:12am

Thx! Building now.

robimarko · January 2, 2022, 10:50am

I was finally doing some testing, and the WAN to LAN routing isn't working for some reason.
It looks related to the FW3 issue that requires a call to iptables first and then it will work.

dchard · January 2, 2022, 10:52am

Yep, run into that. Can you tell me what to do to fix it? Checked the 5.15 PR but seen no sign of this firewall issue....

robimarko · January 2, 2022, 10:53am

I am just looking into fixing it, they have found the kernel commit that changes the behaviour to not register xt_tables by default.

Its in the 5.15 PR, but you need to expand the comments:

github.com/openwrt/openwrt

[WIP,RFT] treewide: add support for kernel 5.15

openwrt:master ← Ansuel:ipq806x-5.15

opened 09:40PM - 04 Nov 21 UTC

Ansuel

+45473 -45

This is in initial stage assuming 5.15 is the next LTS. This is an initial step …for 5.15 support. The generic part **in theory** should be finished and ipq806x works right with 5.15 **This require https://github.com/openwrt/openwrt/pull/4036** As dsa driver is now stable lets just use it for the new kernel directly. From: https://openwrt.org/meetings/20210920#how_to_handle_next_lts_kernel_515 **We need to wait for next release branch as handling another kernel would add too much work on it.** Don't expect this to be merged shortly and treat this as something to try new dsa feature for example. Or also if you want prepare another target to the migration. Porting status: --- - [x] All the hack pending patch ported - [x] Dropped all the backport patches present in 5.15 - [x] Compiled all kmods and fixed the mk with the missing new flags (feed kmods still require to be changed... (antfs require changes) - [x] Refreshed generic config - [x] Refreshed ipq806x config - [ ] Port feeds kmods (can't be done here as they are in a different branch) - [ ] Wider testing Target ported: --- - [x] generic **(when openwrt will be ready to accept this a check of the new patch is needed)** - [x] ipq806x (require fw4 for some reason no wan network with fw3. Router is reachable with ssh) - [x] ipq40xx (Router reachable not wan tested) Current breakage: --- - IRQ doesn't update affinity warning - ~~No serial output~~ (i had the mangle bootloader args config disabled) - ~~mmc broken in some version (this is also present in 5.10 @digitalcircuit is trying to fix that)~~ (fixed in 5.10) - No idea of breakage related to initial port of hack and pending generic patch to 5.15 - fw4 is required or the router can only be reached locally. (someone tested this with x86 target and didn't report any error) (we found more info... iptables are not init by default anymore probably fw3 require some changes to adapt to the new logic) Noteworthy changes and notice: --- - Patch pending 613 optional tcp window check has been standardized and the __read_mostly tag dropped. We should check if this cause some performance degradation as the value will be cached less and the access should be generally slower. - I didn't refresh this as any fuzz should be checked to make sure we are not applying something in the really wrong place. Reported problem with feeds package & some package: --- - ksmbd fails to build due removed locks_verify_truncate() in upstream kernels. - Can't manage to fix a build problem with KEYS kernel flag enabled. Makefile doesn't correctly detect a dependency. Tested device: - ipq806x: Netgear R7800 - ipq40xx: FritzBox 7530 Changelog: --- - 18/12/2021: Refreshed the pr, rebased on today master, added missing patch. Compiled tested only on r7800

dchard · January 2, 2022, 11:04am

Other than this firewall issue, the build boots as it should, PPPoE and wifi works, no strange messages in the kernel log. Only problem is the missing iptables rules. Someone said restarting the firewall service works, someone said that does not fix the issue...

robimarko · January 2, 2022, 11:07am

Restarting firewall wont work, if you make a call to iptables -L once then it will register the tables but even after restarting the firewall and seeing the rules forwarding wont work

robimarko · January 2, 2022, 11:51am

@dchard I added a hack to revert the kernel change and now it works as a temporary fix.
So, just pull the branch head and FW3 will work again.

I am planning to add the 512MB reserved memory DTS today, don't know if I will have the time for it.

dchard · January 2, 2022, 1:03pm

Something is really wrong. Pinging to various IPs are working (no packet loss), but name resolution as well as TCP/IP traffic does fail mostly (and I am not using the router's DNS forwarder). I needed to go back to 5.10. The weird thing is, nothing in the kernel log nor in the system log...

Seems like this firewall/forwarding issue is not fully resolved yet.

robimarko · January 2, 2022, 1:46pm

Hm, this is really weird