Qualcommax NSS Build

yeah i tried rmandrad repo and got some errors lol

1 Like

you need to select the nss firmware and eip

suggest you use qosmio repo for the latest

Yeah I did

Oh ok cool

After using NSS builds for several months on QNAP QHora 301W I can say the following.

Builds based on @bitthief repo.
NSS offloading works as expected for devices connected with cables. Still no NSS WiFi offloading.
I add netfilter: optional tcp window check patch to my build too in order to resolve the SSL errors I experience without it.
With this build after sysupgrade (keeping settings) the DHCP is not working right after the reboot and devices cannot get IP addresses. I have to turn off the power and after the next reboot everything is OK. Same issue was mentioned from others too and is present currently as far as I know on @AgustinLorenzo builds too.
When packet steering is enabled LAN becomes unusable.
These two are the only glitches I have found. All other things work flawlessly on my device. Rebooting takes a little over 30 seconds.

Builds based on @qosmio NSS-WiFi offloading repo (currently I only use it for testing purposes on my QNAP).
I add latest ath11k firmware to this build.
Issues with WLAN are mentioned in the above several posts.
The DHCP is working from the first reboot.
Additionally NSS doesn't work right out of the box. I have to reboot the device and only after the second reboot the NSS offload works OK (zero CPU usage) including NSS wifi offloading.
Running a torrent client on a PC connected to the router with cable for some reason destroys the LAN network and I have difficulties accessing the router web interface and all other clients connected via cable experience connection drops too.

Several other questions.
@dimfish
I see that you still use the obsolete HACK patch by Ansuel to workaround failing sysupgrade. And it was stated that it doesn't work OK.
As @hnyman committed this sysupgrade: Add 2 sec sleep into process KILL loop, I think it is not needed anymore.
Maybe you can review the commits of your NSS-6.1 repo.
Does your repo contain all necessary settings for QNAP? I ask because I see that you don't compile builds for it.

Thanks to all for your great work on this.

Still got the same error weirdly

small update, just wanted to provide an uptime report and more wifili stats

________________________________________________________________________________
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< INTERNAL: UNKNOWN >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________
________________________________________________________________________________
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PDEV 0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________
#txrx
	wifili[0]_rx_msdu_error                 = 0               error
	wifili[0]_rx_inv_peer_rcv               = 0               special
	wifili[0]_rx_wds_srcport_exception      = 0               exception
	wifili[0]_rx_wds_srcport_exception_fail = 0               drop
	wifili[0]_rx_deliverd                   = 59388413        special
	wifili[0]_rx_deliver_drops              = 0               drop
	wifili[0]_rx_intra_bss_ucast            = 0               special
	wifili[0]_rx_intra_bss_ucast_fail       = 0               drop
	wifili[0]_rx_intra_bss_mcast            = 0               special
	wifili[0]_rx_intra_bss_mcast_fail       = 0               drop
	wifili[0]_rx_sg_rcv_send                = 0               special
	wifili[0]_rx_sg_rcv_fail                = 0               drop
	wifili[0]_rx_mcast_echo                 = 0               special
	wifili[0]_rx_inv_tid                    = 0               special
	wifili[0]_stats_rx_frag_inv_sc          = 0               special
	wifili[0]_stats_rx_frag_inv_fc          = 0               special
	wifili[0]_stats_rx_frag_non_frag        = 0               special
	wifili[0]_stats_rx_frag_retry           = 0               special
	wifili[0]_stats_rx_frag_ooo             = 0               special
	wifili[0]_stats_rx_frag_ooo_seq         = 0               special
	wifili[0]_stats_rx_frag_all_frag_rcv    = 0               special
	wifili[0]_stats_rx_frag_deliver         = 0               special
	wifili[0]_tx_enqueue                    = 311604588       special
	wifili[0]_tx_enqueue_drop               = 0               drop
	wifili[0]_tx_dequeue                    = 311604494       special
	wifili[0]_tx_hw_enqueue_fail            = 0               drop
	wifili[0]_tx_sent_count                 = 311604831       special
#tcl ring
	wifili[0]_tcl_no_hw_desc = 0               special
	wifili[0]_tcl_ring_full  = 0               special
	wifili[0]_tcl_ring_sent  = 311602437       special
#tcl comp
	wifili[0]_tx_desc_free_inv_bufsrc       = 0               error
	wifili[0]_tx_desc_free_inv_cookie       = 0               special
	wifili[0]_tx_desc_free_hw_ring_empty    = 0               special
	wifili[0]_tx_desc_free_reaped           = 312331835       special
	wifili[0]_tx_capture_enqueue_count      = 0               special
	wifili[0]_tx_capture_enqueue_fail_count = 0               error
#reo ring
	wifili[0]_reo_error          = 0               error
	wifili[0]_reo_reaped         = 59389374        special
	wifili[0]_reo_inv_cookie     = 0               special
	wifili[0]_stats_reo_frag_rcv = 0               special
#tx sw pool
	wifili[0]_tx_desc_in_use            = 0               special
	wifili[0]_tx_desc_alloc_fail        = 0               special
	wifili[0]_tx_desc_already_allocated = 0               special
	wifili[0]_tx_desc_invalid_free      = 0               special
	wifili[0]_tx_desc_free_src_fw       = 8696640         special
	wifili[0]_tx_desc_free_completion   = 302907279       special
	wifili[0]_tx_desc_no_pb             = 0               special
	wifili[0]_tx_desc_queuelimit_drop   = 0               drop
#tx ext sw pool
	wifili[0]_ext_tx_desc_in_use            = 0               special
	wifili[0]_ext_tx_desc_alloc_fail        = 0               special
	wifili[0]_ext_tx_desc_already_allocated = 0               special
	wifili[0]_ext_tx_desc_invalid_free      = 0               special
#rxdma pool
	wifili[0]_rx_desc_no_pb      = 14793189        special
	wifili[0]_rx_desc_alloc_fail = 0               special
	wifili[0]_rx_desc_in_use     = 4095            special
#rxdma ring
	wifili[0]_rxdma_hw_desc_unavailable = 0               special
	wifili[0]_rxdma_buf_replenished     = 60121572        special
________________________________________________________________________________
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PDEV 1 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________
#txrx
	wifili[1]_rx_msdu_error                 = 0               error
	wifili[1]_rx_inv_peer_rcv               = 0               special
	wifili[1]_rx_wds_srcport_exception      = 0               exception
	wifili[1]_rx_wds_srcport_exception_fail = 0               drop
	wifili[1]_rx_deliverd                   = 5800291         special
	wifili[1]_rx_deliver_drops              = 0               drop
	wifili[1]_rx_intra_bss_ucast            = 0               special
	wifili[1]_rx_intra_bss_ucast_fail       = 0               drop
	wifili[1]_rx_intra_bss_mcast            = 0               special
	wifili[1]_rx_intra_bss_mcast_fail       = 0               drop
	wifili[1]_rx_sg_rcv_send                = 3               special
	wifili[1]_rx_sg_rcv_fail                = 0               drop
	wifili[1]_rx_mcast_echo                 = 0               special
	wifili[1]_rx_inv_tid                    = 0               special
	wifili[1]_stats_rx_frag_inv_sc          = 0               special
	wifili[1]_stats_rx_frag_inv_fc          = 0               special
	wifili[1]_stats_rx_frag_non_frag        = 0               special
	wifili[1]_stats_rx_frag_retry           = 0               special
	wifili[1]_stats_rx_frag_ooo             = 0               special
	wifili[1]_stats_rx_frag_ooo_seq         = 0               special
	wifili[1]_stats_rx_frag_all_frag_rcv    = 0               special
	wifili[1]_stats_rx_frag_deliver         = 0               special
	wifili[1]_tx_enqueue                    = 14745283        special
	wifili[1]_tx_enqueue_drop               = 0               drop
	wifili[1]_tx_dequeue                    = 14745304        special
	wifili[1]_tx_hw_enqueue_fail            = 0               drop
	wifili[1]_tx_sent_count                 = 14745417        special
#tcl ring
	wifili[1]_tcl_no_hw_desc = 0               special
	wifili[1]_tcl_ring_full  = 0               special
	wifili[1]_tcl_ring_sent  = 14745415        special
#tcl comp
	wifili[1]_tx_desc_free_inv_bufsrc       = 0               error
	wifili[1]_tx_desc_free_inv_cookie       = 0               special
	wifili[1]_tx_desc_free_hw_ring_empty    = 0               special
	wifili[1]_tx_desc_free_reaped           = 15050094        special
	wifili[1]_tx_capture_enqueue_count      = 0               special
	wifili[1]_tx_capture_enqueue_fail_count = 0               error
#reo ring
	wifili[1]_reo_error          = 0               error
	wifili[1]_reo_reaped         = 5794282         special
	wifili[1]_reo_inv_cookie     = 0               special
	wifili[1]_stats_reo_frag_rcv = 0               special
#tx sw pool
	wifili[1]_tx_desc_in_use            = 0               special
	wifili[1]_tx_desc_alloc_fail        = 0               special
	wifili[1]_tx_desc_already_allocated = 0               special
	wifili[1]_tx_desc_invalid_free      = 0               special
	wifili[1]_tx_desc_free_src_fw       = 8654669         special
	wifili[1]_tx_desc_free_completion   = 6090685         special
	wifili[1]_tx_desc_no_pb             = 0               special
	wifili[1]_tx_desc_queuelimit_drop   = 0               drop
#tx ext sw pool
	wifili[1]_ext_tx_desc_in_use            = 0               special
	wifili[1]_ext_tx_desc_alloc_fail        = 0               special
	wifili[1]_ext_tx_desc_already_allocated = 0               special
	wifili[1]_ext_tx_desc_invalid_free      = 0               special
#rxdma pool
	wifili[1]_rx_desc_no_pb      = 0               special
	wifili[1]_rx_desc_alloc_fail = 0               special
	wifili[1]_rx_desc_in_use     = 0               special
#rxdma ring
	wifili[1]_rxdma_hw_desc_unavailable = 0               special
	wifili[1]_rxdma_buf_replenished     = 6103133         special
________________________________________________________________________________
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PDEV 2 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________
#txrx
	wifili[2]_rx_msdu_error                 = 0               error
	wifili[2]_rx_inv_peer_rcv               = 0               special
	wifili[2]_rx_wds_srcport_exception      = 0               exception
	wifili[2]_rx_wds_srcport_exception_fail = 0               drop
	wifili[2]_rx_deliverd                   = 0               special
	wifili[2]_rx_deliver_drops              = 0               drop
	wifili[2]_rx_intra_bss_ucast            = 0               special
	wifili[2]_rx_intra_bss_ucast_fail       = 0               drop
	wifili[2]_rx_intra_bss_mcast            = 0               special
	wifili[2]_rx_intra_bss_mcast_fail       = 0               drop
	wifili[2]_rx_sg_rcv_send                = 0               special
	wifili[2]_rx_sg_rcv_fail                = 0               drop
	wifili[2]_rx_mcast_echo                 = 0               special
	wifili[2]_rx_inv_tid                    = 0               special
	wifili[2]_stats_rx_frag_inv_sc          = 0               special
	wifili[2]_stats_rx_frag_inv_fc          = 0               special
	wifili[2]_stats_rx_frag_non_frag        = 0               special
	wifili[2]_stats_rx_frag_retry           = 0               special
	wifili[2]_stats_rx_frag_ooo             = 0               special
	wifili[2]_stats_rx_frag_ooo_seq         = 0               special
	wifili[2]_stats_rx_frag_all_frag_rcv    = 0               special
	wifili[2]_stats_rx_frag_deliver         = 0               special
	wifili[2]_tx_enqueue                    = 0               special
	wifili[2]_tx_enqueue_drop               = 0               drop
	wifili[2]_tx_dequeue                    = 0               special
	wifili[2]_tx_hw_enqueue_fail            = 0               drop
	wifili[2]_tx_sent_count                 = 0               special
#tcl ring
	wifili[2]_tcl_no_hw_desc = 0               special
	wifili[2]_tcl_ring_full  = 0               special
	wifili[2]_tcl_ring_sent  = 0               special
#tcl comp
	wifili[2]_tx_desc_free_inv_bufsrc       = 0               error
	wifili[2]_tx_desc_free_inv_cookie       = 0               special
	wifili[2]_tx_desc_free_hw_ring_empty    = 0               special
	wifili[2]_tx_desc_free_reaped           = 0               special
	wifili[2]_tx_capture_enqueue_count      = 0               special
	wifili[2]_tx_capture_enqueue_fail_count = 0               error
#reo ring
	wifili[2]_reo_error          = 0               error
	wifili[2]_reo_reaped         = 0               special
	wifili[2]_reo_inv_cookie     = 0               special
	wifili[2]_stats_reo_frag_rcv = 0               special
#tx sw pool
	wifili[2]_tx_desc_in_use            = 0               special
	wifili[2]_tx_desc_alloc_fail        = 0               special
	wifili[2]_tx_desc_already_allocated = 0               special
	wifili[2]_tx_desc_invalid_free      = 0               special
	wifili[2]_tx_desc_free_src_fw       = 0               special
	wifili[2]_tx_desc_free_completion   = 0               special
	wifili[2]_tx_desc_no_pb             = 0               special
	wifili[2]_tx_desc_queuelimit_drop   = 0               drop
#tx ext sw pool
	wifili[2]_ext_tx_desc_in_use            = 0               special
	wifili[2]_ext_tx_desc_alloc_fail        = 0               special
	wifili[2]_ext_tx_desc_already_allocated = 0               special
	wifili[2]_ext_tx_desc_invalid_free      = 0               special
#rxdma pool
	wifili[2]_rx_desc_no_pb      = 9059145         special
	wifili[2]_rx_desc_alloc_fail = 0               special
	wifili[2]_rx_desc_in_use     = 4095            special
#rxdma ring
	wifili[2]_rxdma_hw_desc_unavailable = 0               special
	wifili[2]_rxdma_buf_replenished     = 0               special
#wbm ring
	wifili_wbm_ie_local_alloc_fail = 0               error
	wifili_wbm_src_dma             = 772923          special
	wifili_wbm_src_dma_code_inv    = 0               special
	wifili_wbm_src_reo             = 259979          special
	wifili_wbm_src_reo_code_nullq  = 483             special
	wifili_wbm_src_reo_code_inv    = 246258          error
	wifili_wbm_src_inv             = 0               error

I noticed most of my devices that were failing to connect, have all reconnected (they were all 2G smart devices) over the last few days. Still haven't pinpointed what exactly is causing it.

These are the warnings I still get every couple hours (MAC addresses masked)

[Sat Dec 23 12:01:49 2023] ath11k c000000.wifi: encap mismatch in nss tx skb encap type 1vif encap type 2
[Sat Dec 23 12:01:49 2023] ath11k c000000.wifi: failed to transmit frame -22
[Sat Dec 23 12:01:53 2023] ath11k c000000.wifi: peer (xx:xx:xx:xx:xx:xx) not found for nss peer delete
[Sat Dec 23 12:06:51 2023] ath11k c000000.wifi: encap mismatch in nss tx skb encap type 1vif encap type 2
[Sat Dec 23 12:06:51 2023] ath11k c000000.wifi: failed to transmit frame -22
[Sat Dec 23 12:06:55 2023] ath11k c000000.wifi: peer (xx:xx:xx:xx:xx:xx) not found for nss peer delete

As others have mentioned, I'll try to revert just the nss peer delete related code, easier said than done, since it looks to be part of the patch that introduces NSS offload into ath11k...

6 Likes

Great news, and good luck on the peer delete code.

Maybe I'll ask Santa to solve this :santa: Best present there is!

1 Like

Am I understanding correctly that your build is, for the most part, fully functional except for some reconnect glitches on 2.4 GHz? Should be able to keep up with 1.5 Gbps fiber?

i am using the same build with no issues apart from iot devices ... it should keep up with 1.5 Gbps ... I have 1.2Gbps

1 Like

That's great to hear!
Now if I understand correctly, nobody is hosting builds, its a build it yourself deal, right? So I need to get set up for that. Any hints? I've seen you above mention some firmware selection, not sure how that works.

Thanks :slight_smile:

not necessarily I know that @AgustinLorenzo and @dimfish post their own build but of course you can build your own.

I suggest you use this forum and search/ask but first have a read @ https://openwrt.org/docs/guide-developer/start

I managed to get through the generic parts, but compile failed on parts that are very target-specific;

/home/adam/openwrt-ipq/build_dir/target-aarch64_cortex-a53_musl/linux-qualcommax_ipq807x/qca-nss-drv-2023-08-06-1ab18403/nss_rps.c:289:24: error: 'nss_rps_ipv4_hash_bitmap_cfg' defined but not used [-Werror=unused-function]
  289 | static nss_tx_status_t nss_rps_ipv4_hash_bitmap_cfg(struct nss_ctx_instance *nss_ctx, int hash_bitmap)
      |                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
make[5]: *** [scripts/Makefile.build:250: /home/adam/openwrt-ipq/build_dir/target-aarch64_cortex-a53_musl/linux-qualcommax_ipq807x/qca-nss-drv-2023-08-06-1ab18403/nss_rps.o] Error 1
make[4]: *** [Makefile:2014: /home/adam/openwrt-ipq/build_dir/target-aarch64_cortex-a53_musl/linux-qualcommax_ipq807x/qca-nss-drv-2023-08-06-1ab18403] Error 2
make[3]: *** [Makefile:279: /home/adam/openwrt-ipq/build_dir/target-aarch64_cortex-a53_musl/linux-qualcommax_ipq807x/qca-nss-drv-2023-08-06-1ab18403/.built] Error 2
make[3]: Leaving directory '/home/adam/openwrt-ipq/feeds/nss_packages/qca/qca-nss-drv'
time: package/feeds/nss_packages/qca-nss-drv/compile#8.00#2.11#10.06
    ERROR: package/feeds/nss_packages/qca-nss-drv failed to build.
make[2]: *** [package/Makefile:128: package/feeds/nss_packages/qca-nss-drv/compile] Error 1
make[2]: Leaving directory '/home/adam/openwrt-ipq'
make[1]: *** [package/Makefile:122: /home/adam/openwrt-ipq/staging_dir/target-aarch64_cortex-a53_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/adam/openwrt-ipq'
make: *** [/home/adam/openwrt-ipq/include/toplevel.mk:232: world] Error 2

Got some time this weekend to work on the wifi offload patches some more, and can happily say I've been able to resolve almost all of the issues I was encountering :slight_smile:

1.) Resolved encapsulation mismatch errors

    [Sun Dec 24 12:03:21 2023] ath11k c000000.wifi: encap mismatch in nss tx skb encap type 1vif encap type 2
    [Sun Dec 24 12:03:21 2023] ath11k c000000.wifi: failed to transmit frame -22
    [Sun Dec 24 12:08:25 2023] ath11k c000000.wifi: encap mismatch in nss tx skb encap type 1vif encap type 2
    [Sun Dec 24 12:08:25 2023] ath11k c000000.wifi: failed to transmit frame -22

2.) Properly handle nss peer deletion.

    [Sun Dec 17 02:43:25 2023] ath11k c000000.wifi: peer not found for nss peer delete
    [Sun Dec 17 02:43:28 2023] ath11k c000000.wifi: peer not found for nss peer delete
    [Sun Dec 17 02:43:30 2023] ath11k c000000.wifi: peer not found for nss peer delete

3.) Clients should also be able to switch between 2Ghz and 5Ghz AP, as well as join using password (no longer required to boot with NSS off first, connect the client to acquire PSK, and reboot back with nss wifi offload enabled).

Uptime has been 5 hours and so far no issues.

Be sure to enable the symbol ATH11K_MEM_PROFILE_512M in menuconfig when building.

One step closer getting a fully offloaded NSS build. Merry Christmas :smiley:

13 Likes

Thanks for your hard works!
Can you please share your build .config?

Certainly. IPQ807x dynalink dl-wrx36 build.config

Correct (more or less). You should be able to get fully offloaded 1.2-1.5Gbps for WiFi 6 clients if your ISP provides it. I can't confirm more than 1Gbps, as that's the max for my ISP. But theoretically it should work.

How is the WIFI-LAN iperf3 speed on your build?
And CPU usage?

M1 Pro -> LAN Server

iPhone 15 Pro Max -> LAN Server

CPU usage is 0% for WLAN <-> LAN (iperf3 NOT on router)

iPhone 15 Pro Max -> Router

Just providing throughput example when running iperf3 on the router. CPU usage will obviously spike since iperf3 is CPU intensive, but I've pretty consistently been able to reach 1.3-1.5gbps when before, max would be ~900-1100mbps

10 Likes