IPQ806x NSS Drivers

quarky · March 15, 2018, 6:15am

I'm starting a new thread to discuss the IPQ806x NSS subsystem drivers found on the CodeAurora's Git repository (https://source.codeaurora.org.)

Over the past weeks, I managed to cobbled together a build for my Netgear R7800 that successfully activated both NSS CPU cores of the IPQ8065 SOCs. So far, my R7800 is running the following Qualcomm's drivers:

qca-nss-gmac
qca-nss-drv

The changes I made to the lede-17.01 branch can be easily applied to IPQ8064 builds as well.

It would be great if more developers join in the fun to accelerate the development.

At the moment, the qca-nss-drv needs more work on lede as the clock drivers is incomplete. I had to disable two methods as it cannot be compiled. I obtained the clock drivers codes from the Internet. So if anyone is familiar with Linux clock drivers, your help will be much appreciated.

My Github repository for the nss components can be found here:

Netgear R7800 owners who's adventurous enough to try my build, you can get a sysupgrade image from the link below:

https://app.box.com/s/oxaht7753bskakihl5xqq0r159wt76gi

My builds have the following changes:

Includes the Shortcut Forwarding Engine
Replaced stmmac with the qca-nss-gmac driver
Includes the qca-nss-drv drivers (with the nss firmware) to activate both NSS CPU cores and process network traffic.
Increase the overlay partition by using the reserve netgear flash partition. (only for the factory.img image)
Compile OpenSSL and OpenVPN with -O3 optimisation.

Other than that it is similar to the lede-17.01 branch.

Testers and developers welcome to join the fun!

lesandie · March 16, 2018, 1:29pm

I'm in!. building your branch now. BTW i find it difficult to stumble upon documentation about the qca-nss subsystem. Reviewing codeaurora is like trying to read a book in klingon.

quarky · March 16, 2018, 3:01pm

Great! Always good to have more brains in making the IPQ806x routers better. I’m working on the crypto engine next.

lesandie · March 17, 2018, 11:04am

BTW yesterday i could not get a working image, all the build attemps failed, possibly due to my lack of expertise in the package dependencies. I can flash your image and do some tests instead.

lesandie · March 17, 2018, 11:15am

flashed it, everything went smooth but i was on a master build so i had to do a sysupgrade -F from console

Both nss cores seem to be activated:

root@OpenWrt:/etc/init.d# cat /proc/interrupts
           CPU0       CPU1       
 16:      11392      47826       GIC  18 Edge      gp_timer
 18:         33          0       GIC  51 Edge      qcom_rpm_ack
 19:          0          0       GIC  53 Edge      qcom_rpm_err
 20:          0          0       GIC  54 Edge      qcom_rpm_wakeup
 26:          0          0       GIC 241 Edge      29000000.sata
 27:      13465          0       GIC  67 Edge      qcom-pcie-msi
 28:         26          0       GIC  89 Edge      qcom-pcie-msi
 29:     198387          0       GIC 202 Edge      adm_dma
 32:          0          0       GIC 130 Level     bam_dma
 33:          0          0       GIC 128 Level     bam_dma
 34:     397415          0       GIC 245 Level     nss
 35:       1063          0       GIC 264 Level     nss
 36:       7202     389252       GIC 246 Level     nss
 37:          0          0       GIC 265 Level     nss
 44:          2          0   msmgpio   6 Edge      gpio-keys
 92:          2          0   msmgpio  54 Edge      gpio-keys
103:          2          0   msmgpio  65 Edge      gpio-keys
107:          0          0   PCI-MSI   0 Edge      aerdrv
108:      13465          0   PCI-MSI   1 Edge      ath10k_pci
140:          0          0   PCI-MSI   0 Edge      aerdrv
141:         26          0   PCI-MSI   1 Edge      ath10k_pci
173:         11          0       GIC 184 Level     msm_serial0
174:          2          0       GIC 187 Level     1a280000.spi
175:          0          0       GIC 142 Level     xhci-hcd:usb1
176:          0          0       GIC 237 Level     xhci-hcd:usb3
IPI0:          0          0  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:      11072      12506  Rescheduling interrupts
IPI3:          0          0  Function call interrupts
IPI4:        294       2426  Single function call interrupts
IPI5:          0          0  CPU stop interrupts
IPI6:          3          0  IRQ work interrupts
IPI7:          0          0  completion interrupts

lesandie · March 17, 2018, 11:20am

i also see some errors in the kernel log:

[  104.294314] nss_core_log_msg_failures[1115]:bf1f0db8: msg failure - interface: 1, type: 12, response: 4, error: 8
[  104.296744] nss_phys_if_callback[219]:phys_if Error response 4

quarky · March 18, 2018, 12:16am

I see the same error but it doesn’t seem to affect anything. Anyway need to test. I’ll probably start to do some iperf tests after I’m done struggling with the crypto engine. @dissent1 did the work on the crypto engine clocks, so I hope I can continue to make the driver loaded. Right now it’s panicking the kernel.

lesandie · March 18, 2018, 11:45am

One thing i noticed is that if i ping from my laptop to the router i get spikes, but if i ping from my laptop to another device hooked to the router, there are no spikes. Probably the nss core offloading?

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   122 MBytes   102 Mbits/sec                  sender
[  5]   0.00-10.00  sec   122 MBytes   102 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   129 MBytes   108 Mbits/sec                  sender
[  7]   0.00-10.00  sec   128 MBytes   108 Mbits/sec                  receiver
[  9]   0.00-10.00  sec  72.1 MBytes  60.4 Mbits/sec                  sender
[  9]   0.00-10.00  sec  71.7 MBytes  60.2 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  28.4 MBytes  23.9 Mbits/sec                  sender
[ 11]   0.00-10.00  sec  28.4 MBytes  23.8 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  78.5 MBytes  65.8 Mbits/sec                  sender
[ 13]   0.00-10.00  sec  78.1 MBytes  65.5 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec   105 MBytes  87.7 Mbits/sec                  sender
[ 15]   0.00-10.00  sec   104 MBytes  87.5 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec   208 MBytes   174 Mbits/sec                  sender
[ 17]   0.00-10.00  sec   207 MBytes   173 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec   130 MBytes   109 Mbits/sec                  sender
[ 19]   0.00-10.00  sec   130 MBytes   109 Mbits/sec                  receiver
[ 21]   0.00-10.00  sec   114 MBytes  95.6 Mbits/sec                  sender
[ 21]   0.00-10.00  sec   113 MBytes  95.2 Mbits/sec                  receiver
[ 23]   0.00-10.00  sec  81.2 MBytes  68.2 Mbits/sec                  sender
[ 23]   0.00-10.00  sec  81.0 MBytes  68.0 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.04 GBytes   895 Mbits/sec                  sender
[SUM]   0.00-10.00  sec  1.04 GBytes   892 Mbits/sec                  receiver

lesandie · March 18, 2018, 12:41pm

Also the nss cores seem to work, the number of irqs before and after the iperf are noticeable

before

34:   75840426          0       GIC 245 Level     nss
 35:     274865          0       GIC 264 Level     nss
 36:       7202   75486003       GIC 246 Level     nss
 37:          0          0       GIC 265 Level     nss

after

34:   75887374          0       GIC 245 Level     nss
 35:     295927          0       GIC 264 Level     nss
 36:       7202   75517331       GIC 246 Level     nss
 37:          0          0       GIC 265 Level     nss

quarky · March 18, 2018, 10:34pm

Do you see any difference in the main CPU cores utilization compared to the build without the NSS cores activated?

I think the ecm driver is needed as well to begin the offload.

philjohn · March 19, 2018, 12:08pm

From my understanding it is, because it's that which sets up the offloading onto the NSS cores, more info here: https://people.netfilter.org/pablo/netdev0.1/slides/IPQ806x-Hardware-acceleration_v2.pdf

The really fun stuff will begin once you've got that in place, as well as the nss-qdisc stuff as that could accelerate SQM ...

lesandie · March 19, 2018, 12:46pm

thanks for the info!. I was struggling to obtain pertinent info about the NSS

BTW the ECM repo at codeaurora:

https://source.codeaurora.org/quic/qsdk/oss/lklm/qca-nss-ecm/

Probably the correct branch tag could be NHSS.CC.3.0.r1-00101-P but i'm just guessing here. Probably it is better to clone it inside de package/kernel dir and try to compile it with the same tags for nss-drv and nss-gmac, as stated here https://github.com/zhangjingye03/qca-nss-drv/blob/master/README.md

What do you think? I could give it a try

philjohn · March 19, 2018, 1:58pm

Not entirely sure of the best way to progress, that would certainly be the way I'd probably go if I was still working on this though - just make sure you're using the same branch as @quarky is using for nss-drv.

quarky · March 19, 2018, 2:59pm

I'm using the codes tagged with 'NHSS.CC.3.0.r1-00258-P' from https://source.codeaurora.org/quic/qsdk/oss

Still struggling with the crypto driver ... not as straightforward as I thot ...

lesandie · March 19, 2018, 8:11pm

nice! i'll checkout those tags and see were it takes me

lesandie · March 20, 2018, 3:53pm

Trying to build qca-nss-ecm ... it expects many includes from the qca-nss-drv package. I'm copying manually all the .h files that is needing. Still a pain in the ass process. Hope i'll get a build with the ecm soon.

quarky · March 20, 2018, 9:47pm

Are you using the build Makefiles provided by the Qualcomm folks that can be found using the link below?

https://source.codeaurora.org/quic/qsdk/oss/system/feeds/nss-host/

I’m using those Makefiles with the individual driver packages. It should have all the depencies covered.

Still need to change the Makefiles a little as they are apparently using some local development repo or something.

lesandie · March 21, 2018, 7:17am

No!, i was creating the makefiles, using the template Makefile provided by

Thanks for the tip!. I'll check them out.

BTW i compiled both packages nss-drv and nss-gmac without problems.

I'll let you know my progress about the ecm driver

lesandie · March 21, 2018, 8:44am

Nice @quarky! with those Makefiles i got qca-nss-gmac and drv. Still the same errors with the ecm and the new Makefile. I'll dive deep in the code, but it is a 12K lines .c

lesandie · March 21, 2018, 9:17am

Well found the problem why the compiler was always yelling the implicit declaration function. That's because with the new C99 compiler you need to declare your function before referencing it, either directly or in a header

I'll try to cleanup the code and recompile.

Fingers crossed!