IPQ807X NSS Build

Okey. Since we are including #include <nss_api_if.h> in nss.h , and then nss_api_if.h includes the whole thing from nss-drv, we might rewrite some of the old code to make it simpler.

When we include nss_api_if.h, the whole API is exposed right?

And as far as I understand the present code. We still need to send the right data to the right place with the right format or what it's called :slight_smile:

Example is: enum ath11k_nss_vdev_cmd is already in nss-drv as enum nss_wifi_vdev_cmd.

And I'm not saying anything about bad code etc. I have a lot of respect for what is done.
But my thought was that we could aim for more use of the API in our offload patches.

and enabled by ecm (this is my understanding) ... basically nss-drv is the platform that provide the offloading via ecm to work eg. nss-drv doesn't enable offloading on its own it needs ecm

a good example is wifi offloading that at the moment is not working and the code is on ecm

ECM is the classification engine, WLAN AFAIK does not even require ECM

Yes can confirm is not needed

1 Like

you should try nss-drv without ecm ... but of course you know best what I know ...worth doing tests with/whitout ecm though ...why don't you bring nss-drv into the mainline if it is the case? just one more to ssdk

How do you notice/test that ipv6 NSS offloading doesn't work.
Since 4 weeks ago I noticed that when I use Speedtest and the qBittorrent is running at the same time I see a bit higher CPU load (5-15%, where I expect it to be almost zero).
I think that was not present on older builds but cannot be completely sure.
I keep my older builds and may try one of them.

i late join on ipv6 party but i noticed on fast.com with ipv6 connection
cpu2 bump to top when i disable ipv6 connection cpu usage goes down while testing
i also keep my older builds i may test later

1 Like

I've just tried on both fast.com and www.waveform.com/tools/bufferbloat but I cannot confirm that NSS is not working with IPv6. CPU usage is zero for me.
My build is based on latest bitthief repo (3 days old).

what /sys/kernel/debug/qca-nss-drv/stats/ipv6 shows ? i am assuming you enabled NSS_DRV_IPV6_ENABLE

root@QNAP:~# cat /sys/kernel/debug/qca-nss-drv/stats/ipv6

________________________________________________________________________________

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< IPV6 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________

        ipv6_rx_pkts           = 11912677        common
        ipv6_rx_byts           = 16156584827     common
        ipv6_tx_pkts           = 0               common
        ipv6_tx_byts           = 0               common
        ipv6_rx_queue[0]_drops = 0               drop
        ipv6_rx_queue[1]_drops = 0               drop
        ipv6_rx_queue[2]_drops = 0               drop
        ipv6_rx_queue[3]_drops = 0               drop

#ipv6 node stats

        ipv6_rx_pkts                               = 2081742         special
        ipv6_rx_bytes                              = 2725039398      special
        ipv6_tx_pkts                               = 2081742         special
        ipv6_tx_bytes                              = 2725039398      special
        ipv6_create_requests                       = 749             special
        ipv6_create_collisions                     = 0               special
        ipv6_create_invalid_interface              = 0               special
        ipv6_destroy_requests                      = 203             special
        ipv6_destroy_misses                        = 0               special
        ipv6_hash_hits                             = 523             special
        ipv6_hash_reorders                         = 3               special
        ipv6_flushes                               = 526             special
        ipv6_evictions                             = 0               special
        ipv6_fragmentations                        = 0               special
        ipv6_frag_fails                            = 0               special
        ipv6_by_rule_drops                         = 0               drop
        ipv6_mc_create_requests                    = 0               special
        ipv6_mc_update_requests                    = 0               special
        ipv6_mc_create_invalid_interface           = 0               special
        ipv6_mc_destroy_requests                   = 0               special
        ipv6_mc_destroy_misses                     = 0               special
        ipv6_mc_flushes                            = 0               special
        ipv6_mirror_invalid_ifnum_conn_create_req  = 0               special
        ipv6_mirror_invalid_iftype_conn_create_req = 0               special
        ipv6_mirror_failures                       = 0               special

#ipv6 exception stats

        ipv6_icmp_header_incomplete         = 0               exception
        ipv6_icmp_unhandled_type            = 5401            exception
        ipv6_icmp_header_incomplete         = 0               exception
        ipv6_icmp_udp_header_incomplete     = 0               exception
        ipv6_icmp_tcp_header_incomplete     = 0               exception
        ipv6_icmp_unknown_protocol          = 0               exception
        ipv6_icmp_no_icme                   = 5               exception
        ipv6_icmp_flush_to_host             = 8               exception
        ipv6_tcp_header_incomplete          = 0               exception
        ipv6_tcp_no_icme                    = 7819524         exception
        ipv6_tcp_small_hop_limit            = 0               exception
        ipv6_tcp_needs_fragmentation        = 0               exception
        ipv6_tcp_flags                      = 513             exception
        ipv6_tcp_seq_exceeds_right_edge     = 0               exception
        ipv6_tcp_small_data_offs            = 0               exception
        ipv6_tcp_bad_sack                   = 0               exception
        ipv6_tcp_big_data_offs              = 0               exception
        ipv6_tcp_seq_before_left_edge       = 5               exception
        ipv6_tcp_ack_exceeds_right_edge     = 0               exception
        ipv6_tcp_ack_before_left_edge       = 0               exception
        ipv6_udp_header_incomplete          = 0               exception
        ipv6_udp_no_icme                    = 4087057         exception
        ipv6_udp_small_hop_limit            = 0               exception
        ipv6_udp_needs_fragmentation        = 0               exception
        ipv6_wrong_target_mac               = 0               exception
        ipv6_header_incomplete              = 0               exception
        ipv6_unknown_protocol               = 164             exception
        ipv6_ingress_vid_mismatch           = 0               exception
        ipv6_ingress_vid_missing            = 0               exception
        ipv6_dscp_marking_mismatch          = 0               exception
        ipv6_vlan_marking_mismatch          = 0               exception
        ipv6_interface_mismatch             = 0               exception
        ipv6_gre_no_icme                    = 0               exception
        ipv6_gre_needs_fragmentation        = 0               exception
        ipv6_gre_small_hop_limit            = 0               exception
        ipv6_destroy                        = 0               exception
        ipv6_icmp_udplite_header_incomplete = 0               exception
        ipv6_udplite_header_incomplete      = 0               exception
        ipv6_udplite_no_icme                = 0               exception
        ipv6_udplite_small_hop_limit        = 0               exception
        ipv6_udplite_needs_fragmentation    = 0               exception
        ipv6_mc_udp_no_icme                 = 0               exception
        ipv6_mc_mem_alloc_failure           = 0               exception
        ipv6_mc_update_failure              = 0               exception
        ipv6_mc_pbuf_alloc_failure          = 0               exception
        ipv6_esp_header_incomplete          = 0               exception
        ipv6_esp_no_icme                    = 0               exception
        ipv6_esp_ip_fragment                = 0               exception
        ipv6_esp_small_hop_limit            = 0               exception
        ipv6_esp_needs_fragmentation        = 0               exception
        ipv6_tunipip6_no_icme               = 0               exception
        ipv6_tunipip6_small_hop_limit       = 0               exception
        ipv6_tunipip6_needs_fragmentation   = 0               exception
        ipv6_pppoe_bridge_no_icme           = 0               exception
        ipv6_dont_frag_set                  = 0               exception
        ipv6_reassembly_not_supported       = 0               exception
        ipv6_pppoe_no_session               = 0               drop
        ipv6_icmp_gre_header_incomplete     = 0               exception
        ipv6_icmp_esp_header_incomplete     = 0               exception
        ipv6_emesh_prio_mismatch            = 0               exception
        ipv6_mc_ucast_dmac_failure          = 0               exception
root@QNAP:~#

1 Like

Hi @Ansuel . I'm curious where things go wrong when I try to port the wifi-offload patches.
So far my changes has only been able to view wifili data, compared to none.

Example:

root@wax620:~# cat /sys/kernel/debug/qca-nss-drv/stats/wifili

________________________________________________________________________________

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< INTERNAL: UNKNOWN >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________


________________________________________________________________________________

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< PDEV 0 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
________________________________________________________________________________


#txrx

	wifili[0]_rx_msdu_error                 = 0               error
	wifili[0]_rx_inv_peer_rcv               = 0               special
	wifili[0]_rx_wds_srcport_exception      = 0               exception
	wifili[0]_rx_wds_srcport_exception_fail = 0               drop
	wifili[0]_rx_deliverd                   = 0               special
...

However, with kernel 6 and latest snapshot I can see my wireless interfaces in luci, but they are not broadcasting. I believe they are not registered or even initialized.
And no stats = interfaces are not working :slight_smile:

Anyway, am I correct if all the init, registration and setup is done in:

ath11k_nss_get_vdev_opmode = Set the correct type.
ath11k_nss_vdev_set_cmd = Commands for the Wi-Fi virtual device.
nss_wifi_vdev_msg_init = Initializes a Wi-Fi virtual device message.
ath11k_nss_vdev_configure = NSS Wi-Fi virtual device configuration message.
ath11k_nss_vdev_register = nss_register_wifi_vdev_if, Registers a Wi-Fi virtual device interface with the NSS interface.

So if I skip the stats for now and try to get the interfaces registered, the list above would be a start?

Edit: I get this in dmesg btw...

[    9.434561] ath11k c000000.wifi: htt event 48 not handled <-- I think this is ath11k_dp_htt_htc_t2h_msg_handler
[    9.436037] ath11k c000000.wifi: nss init soc nss if_num 203 <-- that is a little high?

Edit 2: Adding some more data from some messages I've set to show.

[    9.528503] ath11k c000000.wifi: ath11k_nss_setup: target_type is:24
[    9.528688] ath11k c000000.wifi: ath11k_nss_setup: Is nss.enabled:1
[    9.528694] ath11k c000000.wifi: ath11k_nss_init: target_type:24
[    9.528705] ath11k c000000.wifi: ath11k_nss_init: wim->flags:0
[    9.528710] ath11k c000000.wifi: dev_base_addr: 201326592 shadow_rdptr_mem_addr: 1146818560 shadow_wrptr_mem_addr: 1123000320 lmac_rings_start_id: 128
[    9.530086] ath11k c000000.wifi: status from init 0
[    9.545043] ath11k c000000.wifi: ath11k_nss_init: NSS Init Message TX Success 0000000029b56dea 203
[    9.545067] ath11k c000000.wifi: ath11k_nss_setup: ath11k_nss_init:0
[    9.545200] ath11k c000000.wifi: ifnum: 28 scheme_id: 0 nss_radio_priority: -1073741824
[    9.545214] ath11k c000000.wifi: ath11k_nss_pdev_init: nss_cmn_msg_init :0
[    9.548415] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.548442] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.548449] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.548456] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.548539] ath11k c000000.wifi: ifnum: 29 scheme_id: 0 nss_radio_priority: -1073741824
[    9.548550] ath11k c000000.wifi: ath11k_nss_pdev_init: nss_cmn_msg_init :0
[    9.551863] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.551890] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.551897] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.551903] ath11k c000000.wifi: ath11k_nss_radio_buf_cfg: status nss_wifili_tx_msg: 0
[    9.551962] ath11k c000000.wifi: ath11k_nss_start: nss start is a success:0
[    9.551967] ath11k c000000.wifi: ath11k_nss_setup: ath11k_nss_start:0
[    9.551977] ath11k c000000.wifi: ath11k_nss_setup: nss_wifi_vdev_base_set_next_hop:0
[   10.469422] br-lan: port 1(lan) entered blocking state
[   10.469446] br-lan: port 1(lan) entered disabled state
[   10.469674] device lan entered promiscuous mode
[   12.087029] (NULL device *): ath11k_nss_vdev_register 1058 status and if_num 0000000000000000 30
[   12.087063] ath11k c000000.wifi: ath11k_nss_vdev_configure: this is vdev_config 73732136
[   12.087068] ath11k c000000.wifi: ath11k_nss_vdev_configure: Radio interface corresponding to virtual AP: 28
[   12.087073] ath11k c000000.wifi: ath11k_nss_vdev_configure: Parent virtual device interface number: 0
[   12.087078] ath11k c000000.wifi: ath11k_nss_vdev_configure: VAP operating mode: Access-Point (AP) or Station (STA): 1
[   12.087382] ath11k c000000.wifi: ath11k_nss_vdev_create: nss vdev interface created ctx 0000000029b56dea, ifnum 30
[   12.098426] ath11k c000000.wifi: ath11k_nss_peer_create: status nss_wifili_tx_msg: 0
[   12.099623] br-lan: port 2(5GHz) entered blocking state
[   12.099643] br-lan: port 2(5GHz) entered disabled state
[   12.099851] device 5GHz entered promiscuous mode
[   12.099923] br-lan: port 2(5GHz) entered blocking state
[   12.099932] br-lan: port 2(5GHz) entered forwarding state
[   12.101913] br-lan: port 2(5GHz) entered disabled state
[   12.569182] nss-dp 3a001a00.dp6 lan: PHY Link up speed: 1000
[   12.569329] br-lan: port 1(lan) entered blocking state
[   12.569341] br-lan: port 1(lan) entered forwarding state
[   12.787249] ath11k c000000.wifi: ath11k_nss_vdev_up: nss vdev up with status 0
[   12.788164] br-lan: port 2(5GHz) entered blocking state
[   12.788185] br-lan: port 2(5GHz) entered forwarding state
[   12.867398] ath11k c000000.wifi: ath11k_nss_set_peer_sec_type: nss peer id 1 security cfg complete
[   12.867425] ath11k c000000.wifi: ath11k_nss_set_peer_authorize: status msg_cb: 13474400
[   12.867437] (NULL device *): ath11k_nss_set_peer_authorize: status nss_wifili_tx_msg: 0
[   12.867444] ath11k c000000.wifi: ath11k_nss_set_peer_authorize: nss peer id 1 auth cfg complete

Edit: After testing some more I can see that reo2host-destination-ring1,2,3,4 is missing from proc/interrupts.

Testing different modes lile frame_mode=1 outputs a lot of errors compared to frame_mode=0 and frame_mode=2.

# frame_mode=1 error log:
[   12.291667] qcom-q6v5-wcss-pil cd00000.q6v5_wcss: fatal error received:
[   12.291667] QC Image Version: QC_IMAGE_VERSION_STRING=WLAN.HK.2.9.0.1-01385-QCAHKSWPL_SILICONZ-1
[   12.291667] Image Variant : IMAGE_VARIANT_STRING=8074.wlanfw.eval_v2Q
[   12.291667]
[   12.291667] ERR_IN_PHY_OFF:0  Ucode Asserted:, PHYB    param0 :zero, param1 :zero, param2 :zero.
[   12.291667] Thread ID      : 0x00000000  Thread name    :   Process ID     : 0
[   12.291667] Register:
[   12.291667] SP : 0x00000000
[   12.291667] FP : 0x00000000
[   12.291667] PC : 0x00000000
[   12.291667] SSR : 0x00000000
[   12.291667] BADVA : 0x00000000
[   12.291667] LR : 0x00000000
[   12.291667]
[   12.291667] No stack pointer recorded (or SP is NULL)...stack data not dumped.
[   12.291667]
[   12.337323] remoteproc remoteproc0: crash detected in cd00000.q6v5_wcss: type fatal error
[   12.359597] remoteproc remoteproc0: handling crash #1 in cd00000.q6v5_wcss
[   12.367870] remoteproc remoteproc0: recovering cd00000.q6v5_wcss
[   12.400355] remoteproc remoteproc0: stopped remote processor cd00000.q6v5_wcss
[   12.687119] ath11k c000000.wifi: failed to send WMI_PDEV_SET_PARAM cmd
[   12.687164] ath11k c000000.wifi: failed to recalc txpower limit 16 using pdev param 3: -108
[   12.692556] ath11k c000000.wifi: WMI vdev 1 failed to send FILS discovery enable/disable command
[   12.700827] ath11k c000000.wifi: WMI vdev 1 failed to send FILS discovery enable/disable command
[   12.709851] ath11k c000000.wifi: WMI vdev 1 failed to send FILS discovery enable/disable command
[   12.718604] ath11k c000000.wifi: failed to send WMI_VDEV_SET_PARAM_CMDID
[   12.727365] ath11k c000000.wifi: Failed to set beacon interval for VDEV: 1
[   12.734043] ath11k c000000.wifi: failed to send WMI_PDEV_SET_PARAM cmd
[   12.740838] br-lan: port 3(2GHz) entered blocking state
[   12.747245] br-lan: port 3(2GHz) entered forwarding state
[   12.787103] ath11k c000000.wifi: ath11k_nss_set_peer_sec_type: nss peer id 3 security cfg complete
[   12.787161] ath11k c000000.wifi: ath11k_nss_set_peer_authorize: status msg_cb: 13544304
[   12.794975] (NULL device *): ath11k_nss_set_peer_authorize: status nss_wifili_tx_msg: 0
[   12.802892] ath11k c000000.wifi: ath11k_nss_set_peer_authorize: nss peer id 3 auth cfg complete
[   12.811087] 2GHz: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-108)
[   12.917199] device 2GHz left promiscuous mode
[   12.917326] br-lan: port 3(2GHz) entered disabled state
[   12.957473] ath11k c000000.wifi: ath11k_nss_vdev_down: nss vdev down with status 0
[   13.037535] ath11k c000000.wifi: failed to clear rx_filter for monitor status ring: (-108)
[   17.180916] ath11k_warn: 43 callbacks suppressed
[   17.180935] ath11k c000000.wifi: failed to send WMI_PDEV_BSS_CHAN_INFO_REQUEST cmd
[   17.184630] ath11k c000000.wifi: failed to send pdev bss chan info request
[   17.192358] ath11k c000000.wifi: failed to send WMI_PDEV_SET_PARAM cmd
[   17.198881] ath11k c000000.wifi: Failed to set beacon mode for VDEV: 0
[   17.205390] ath11k c000000.wifi: failed to send WMI_BCN_TMPL_CMDID
[   17.211898] ath11k c000000.wifi: failed to submit beacon template command: -108
[   17.218058] ath11k c000000.wifi: failed to update bcn template: -108
[   17.225253] ath11k c000000.wifi: failed to send WMI_VDEV_SET_PARAM_CMDID
[   17.231873] ath11k c000000.wifi: Failed to set dtim period for VDEV 0: -108
[   17.238548] ath11k c000000.wifi: WMI vdev 0 failed to send FILS discovery enable/disable command
[   17.687090] qcom-q6v5-wcss-pil cd00000.q6v5_wcss: start timed out
[   17.687139] remoteproc remoteproc0: can't start rproc cd00000.q6v5_wcss: -110
[   23.248228] ath11k_warn: 9 callbacks suppressed

Well, I think this is a good. I should run on frame_mode=1 right? Mode 0 and 2 give no errors...
They all share the same issue, wifi is not broadcasting, so something is going on here.

Edit: My bad about the reo2host

if (!nss_offload && ab->hw_params.ring_mask->rx[i] & BIT(j)) {
 				irq_grp->irqs[num_irq++] =
 					reo2host_destination_ring1 - j;
 			}

@robimarko , how do I enable debug mode for nss-drv? I need to see some debug for ath11k...

There are multiple levels of prints for NSS-DRV, check the Makefile in NSS-DRV source for the default and override it from OpenWrt

Thanks. So I can change a level by setting a parameter in the module?

No, it is a compile time option

1 Like

I am assuming you are trying to enable wifi offload via the nss-drv.

Did some research looking at the qca qsdk release AU_LINUX_QSDK_NHSS.QSDK.12.4.5.R2_TARGET_ALL.12.4.502.099.019

So basically the ath11k driver is patched to support nss offload (ignore ath12k)

the nss-drv works, here's a console msg telling that the code was initialised (after applying a quick patch to debug the code I am interested )

So, if you (or anyone) wants to have a go is about taking these patches and retrofitting them into the mac80211 openwrt version.

1 Like

Thanks. I'll see what I can do. Can't code C etc, so gonna be interesting :smile:

Edit:
Wiki says https://wiki.codelinaro.org/en/clo/qsdk/overview

Version  Kernel
23.05 	 6.1 	 ipq95xx, ipq53xx  	AU_LINUX_QSDK_NHSS.QSDK.12.4.5.R2_TARGET_ALL.12.4.502.099.019.xml   ATH11k+12k

Found this repo

From a quick view of the patch, it looks like (maybe edited) the original patches for kernel 6.1.
I'll give it a go and see anyway...

Are you on kernel 5 or 6 @rmandrad ? And did the wifi work?
I've managed to convert the patches so it compiles, but the wifi is not broadcasting and I can't do a channel scan... that's from the ansuel patches. I'll see if the qca patches do any good, but so far when looking at the code, they are similar. Qca include a lot more stats stuff, and then I have to remove or add the other patches...

Edit: After testing 199, 1 - 2 - 3 patches from NHSS.QSDK.12.4.5.r3 that use PKG_VERSION:=6.5-rc3, it's even worse than converting from ansuel's patches.
I'm using PKG_VERSION:=6.5. I guess there's patches all over the place to make it work for qca...

Edit: 2023-12-12, finally got some errors and I was able to scan wifi channels. But some of the errors I got below:

[   12.366983] ath11k c000000.wifi: failure to set vdev security type in nss
[   12.367031] 2GHz: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-22)
[   13.006948] ath11k c000000.wifi: failure to set vdev security type in nss
[   13.006993] 5GHz: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-22)
[  180.637744] ath11k c000000.wifi: peer not found for nss peer delete
[  180.990860] remoteproc remoteproc0: crash detected in cd00000.q6v5_wcss: type fatal error
[  181.013067] remoteproc remoteproc0: handling crash #1 in cd00000.q6v5_wcss
[  181.021177] remoteproc remoteproc0: recovering cd00000.q6v5_wcss
[  181.053967] remoteproc remoteproc0: stopped remote processor cd00000.q6v5_wcss
[  181.054257] ath11k c000000.wifi: failed to clear rx_filter for monitor status ring: (-108)
[  181.786650] ath11k c000000.wifi: failed to send WMI_PDEV_SET_PARAM cmd
[  181.786695] ath11k c000000.wifi: failed to enable PMF QOS: (-108
[  186.327723] qcom-q6v5-wcss-pil cd00000.q6v5_wcss: start timed out
[  186.327772] remoteproc remoteproc0: can't start rproc cd00000.q6v5_wcss: -110
[  211.033240] ath11k c000000.wifi: failed to send WMI_PDEV_SET_PARAM cmd
[  211.033285] ath11k c000000.wifi: failed to enable PMF QOS: (-108

I think there was some patches for some of the errors too...
This build is a mix of qca and ansuels patches. Just qca patches has too much stats stuff all over the place and I just try to get the wifi up, and take it from there.

I am using openwrt snapshot that is on 6.1

Hmm... could you share the link to the git where the patches are? Since you are able to compile and build with no issues, I'm starting to believe I have the wrong patches.

Thanks.

what do you mean ? you have bitthief repo to build ...

as I said if you want to add wifi offload you have to create the patches and setup the all thing from scratch... I haven't done it myself

also you can have a look at ansuel's repo as he tried to have it work ...