The benchmark kernel modules are cryptsetup and cryptsetup-openssl?
I'd like to join this effort although I'm not sure what would be the best use of time.
Is it too early to think of porting this to 4.9? Does it make sense to wait for 4.14?
What else needs to be done?
IMHO, the top priority is to get both NSS CPU cores working. It doesn't seem to work at the moment, although the driver and the firmware seems to be loaded successfully.
It looks like we're missing the msm-bus support or DMA bus support for the NSS sub-systems. When I tried to test the qca-nss-crypto driver, it is complaining that the NSS cores are not ready and kernel panicked.
At this point, I'm kind of stuck ... need to understand the existing 3.4 kernel workings and hopefully can figure out whether it is feasible to port the msm-bus drivers or re-use one of the existing 4.x kernel bus drivers.
I may be way off tangent tho ...
1 Like
Still working! though i have limited time to invest!
Hi @quarky!,
I've investigated a little bit more and i wanted your opinion if i'm off-piste. Probably you stumbled across the same errors or warnings but i wanted to get your confirmation and knowledge about this so that i can continue ruling out errors and understanding a little bit more what the code does. On the other hand if you have any task or investigation that you're willing to "offload" and treat me as a nss core 
please tell me.
Using dmesg i've traced all the errors the gmac and drv are spitting out from your build.
First , as you previously said, it seems that the nss cores are loading the firmware correctly
**********************************************************
[ 13.168929] * Driver :NSS GMAC Driver for RTL v(3.72a)
[ 13.175340] * Version :1.0
[ 13.180903] * Copyright :Copyright (c) 2013-2016 The Linux Foundation. All rights reserved.
[ 13.183765] **********************************************************
[ 13.212827] ipip: IPv4 over IPv4 tunneling driver
[ 13.220712] nss-driver: fabric 0 handler
[ 13.220777] nss-driver: fabric 1 handler
[ 13.271374] nss_driver - fw of size 434240 bytes copied to load addr: 40000000, nss_id : 0
[ 13.271838] nss_driver - Turbo Support 1
[ 13.278673] Supported Frequencies - 110Mhz 600Mhz 800Mhz
[ 13.291371] nss_driver - fw of size 189248 bytes copied to load addr: 40800000, nss_id : 1
Later i get the errors:
[ 18.252895] nss_data_plane_work_function[140]:bf1f0db8: Register data plane failed for gmac:0
[ 18.252933] nss_data_plane_work_function[140]:bf1f0db8: Register data plane failed for gmac:3
[ 18.264042] nss_core_handle_nss_status_pkt[237]:bf1f1dd0: Callback not registered for interface 100
[ 18.299560] nss_ipv4_update_conn_count_cb[506]:IPv4 connection count update success: 0
[ 18.299595] nss_ipv6_update_conn_count_cb[506]:IPv6 connection count update success: 0
[ 18.306536] nss_core_handle_nss_status_pkt[237]:bf1f0db8: Callback not registered for interface 100
The first one is related to the data plane work function that gets queued to "install" the gmac overlays, in the nss_data_plane.c file of the nss-drv:
Inspecting this file i can see the work function
/*
* nss_data_plane_work_function()
* Work function that gets queued to "install" the gmac overlays
*/
static void nss_data_plane_work_function(struct work_struct *work)
{
int i;
struct nss_ctx_instance *nss_ctx = &nss_top_main.nss[NSS_CORE_0];
for (i = 0; i < NSS_MAX_PHYSICAL_INTERFACES; i++) {
if (!nss_data_plane_register_to_nss_gmac(nss_ctx, i)) {
nss_warning("%p: Register data plane failed for gmac:%d\n", nss_ctx, i);
} else {
nss_info("%p: Register data plan to gmac:%d success\n", nss_ctx, i);
}
}
}
So it seems that it fails to install the hal overlays for the gmac. Reading more core i've stumbled upon a Lkml list 2015 email talking about the nss-drv and the nss-gmac drivers, explaining a little bit more how they work, not in-deph but a general view.
https://lkml.org/lkml/2015/1/8/534
They din't upstream it and they constructively criticize the code, though it seems to be old.
So my perception is that if they still haven't upstreamed it to the kernel, maybe there are many things to change that they do not plan to or is not worth their time. What do you think?. I'll investigate further a couple of more errors:
[ 22.161501] nss_core_log_msg_failures[1115]:bf1f0db8: msg failure - interface: 93, type: 5, response: 4, error: 7
[ 22.161536] nss_ipv4_conn_cfg_callback[409]:IPv4 connection configuration failed with error: 7
[ 25.299111] nss_core_log_msg_failures[1115]:bf1f0db8: msg failure - interface: 2, type: 12, response: 4, error: 8
[ 25.299140] nss_phys_if_callback[219]:phys_if Error response 4
Best!
@lesandie the errors you saw for the data plane should be expected, from what I understand. gmac0 and gmac3 are not enabled, so the the nss driver will not be able to associate anything to both interfaces.
I’m still struggling in getting the NSS cores to run the firmware that’s uploaded, which at the moment I suspect it’s due to apps fabric clock/bus not properly initialized. Kind of stuck at the moment.
I'm not a developer but I do have a great deal of experience operating unix OSes. Please let me know if I'm in the wrong section or breaking a forum rule.
I compiled your repo and it went through that process without issue. Upon flashing it, though, my router bootloops (Netgear Nighthawk X4 R7500v2). I currently do not own any hardware to read the boot status so I can't really submit any logs for you to see. All I compiled as extra was the NSS crypto + drv + gmac, LuCI + openssl and replaced dnsmasq + wpad-mini with their full versions.
I also noticed board-2.bin and firmare-5.bin links are borked when building ath10k firmware. Easy to fix but it's more of a hack than a real way of getting the firmware built and I'm quite positive this is the source of my bootlooping issue. Perhaps I should buy a RPi and make sure myself!
At the moment, my repo is geared specifically for the R7800 as only the R7800 DTS is complete to support the NSS drivers. You may want to try tweaking the R7500 DTS files similar to what I’ve done for the R7800 and see if it fixes your bootloop problem. It’s best tho to have serial console access to see what’s wrong.
can i copy the patches from 4.4 to 4.9 with trunk,and compile it?
You can try, but I suspect you will probably encounter a lot of compile errors.
That won't work, it's not that easy, sadly.
Additional info: Quarky's repo is geared towards IPQ8065, so if you have hardware with IPQ8064 or other SoC, you will bootloop with the resulting compiled image. I'm missing a lot of info on the IPQ8064 SoC (like clocks, I have no idea where Quarky got 8065 clocks nor the NSS clocks but they don't match: 8064 is 1.4ghz and 8065 1.7ghz, same for NSS: 730mhz vs 800mhz respectively) but what I do know is that 8064 does NOT have GMAC, so it would make sense to only compile NSS and Crypto, and optionally qrfs-nohnat for software offload. (unless that's not what qrfs does and I completely misunderstood)
1 Like
yes,for kernel,the patch failed.
That's correct.
BTW the qrfs module is here:
https://source.codeaurora.org/quic/qrdk/oss/lklm/qca-rfs/
but it is useless without the nss-drv module and also useless without the nss-ecm module. I cannot progress because of my lack of knowledge in the kernel internals. Although i'm reading a couple of books about linux device drivers and kernel the progress is non-existant. The guys at qualcomm just released their code for the 3.x line of kernels and that's it.
Wish I could help with this effort, would be amazing to see the NSS cores enabled on the R7800. Since the recent commit below set the ipq806x to 2 cores instead of 4, wouldn't this stop NSS driver from working?
https://github.com/openwrt/openwrt/commit/fff65dbe2436351ea1feee6c79110971ec4d5881
Why would it? This is an artifact from when ipq4 and 8 were the same build target.
1 Like
Linux, the kernel (and that's what this setting is all about), only runs on the two ARMv7 (KRAIT300, roughly similar to cortex A15) cores. The two NSS cores are based on a ubicom32 IP core and need their own (non-linux) firmware and are totally unrelated to this setting (which is rather cosmetic (o.k., limiting it to the correct value saves a tiny amount of RAM), but correct for all known ipq806x SOCs so far).
anyway i think i found the problem with crypto...
checking the code in r7800 a struct is missing...
/*
* NSS crypto for IPQ806x (version 1.0)
*/
#define NSS_CRYPTO_PBASE_SZ 0x20000 /* Crypto Register space size */
#define NSS_CRYPTO_BAM_PBASE_SZ 0x22000 /* Crypto BAM Register space size */
#define NSS_CRYPTO_PBASE_OFST 0x400000 /* Crypto Register offset for each engine */
#define NSS_CRYPTO_PBASE_ENG(engine) (0x38000000 + (NSS_CRYPTO_PBASE_OFST * engine))
#define NSS_CRYPTO_BAM_PBASE_ENG(engine) (NSS_CRYPTO_PBASE_ENG(engine) + 0x4000)
#define NSS_CRYPTO_INIT_PLATFORM_DATA_ENG(engine) { \
.crypto_pbase = NSS_CRYPTO_PBASE_ENG(engine), \
.crypto_pbase_sz = NSS_CRYPTO_PBASE_SZ, \
.bam_pbase = NSS_CRYPTO_BAM_PBASE_ENG(engine), \
.bam_pbase_sz = NSS_CRYPTO_BAM_PBASE_SZ, \
.bam_ee = 0, \
}
/*
* instantiate platform data for nss crypto
*/
static struct nss_crypto_platform_data nss_crypto_eng[] = {
NSS_CRYPTO_INIT_PLATFORM_DATA_ENG(0),/* Engine - 0 */
NSS_CRYPTO_INIT_PLATFORM_DATA_ENG(1),/* Engine - 1 */
NSS_CRYPTO_INIT_PLATFORM_DATA_ENG(2),/* Engine - 2 */
NSS_CRYPTO_INIT_PLATFORM_DATA_ENG(3),/* Engine - 3 */
};
static u64 nss_crypto_dma_mask = DMA_BIT_MASK(32);
#define NSS_CRYPTO_INIT_PLATFORM_DEVICE_ENG(engine) { \
.name = "nss-crypto", \
.id = (engine), \
.dev = { \
.dma_mask = &nss_crypto_dma_mask, \
.coherent_dma_mask = DMA_BIT_MASK(32), \
.platform_data = &nss_crypto_eng[(engine)], \
} \
}
/*
* instantiate platform devices for nss crypto
*/
struct platform_device ipq806x_device_nss_crypto[] = {
NSS_CRYPTO_INIT_PLATFORM_DEVICE_ENG(0),/* Engine - 0 */
NSS_CRYPTO_INIT_PLATFORM_DEVICE_ENG(1),/* Engine - 1 */
NSS_CRYPTO_INIT_PLATFORM_DEVICE_ENG(2),/* Engine - 2 */
NSS_CRYPTO_INIT_PLATFORM_DEVICE_ENG(3),/* Engine - 3 */
};
This struct is present in r7800 stock firmware march-msm / devices-ip at the very end of the file...
as the crypto platform file say...
/*
* nss_crypto_probe()
* probe routine called per engine from MACH-MSM
*/
We need to first init the platform engine from the msm driver and then we can use the crypto
Anyway one question... Someone said that the nss core are not working... I think they are working as from the log they get initialized and also we have traffic... am i wrong or the nss core needs to work to actually have packet traffic?
Hi @Ansuel, I’m kind of tied up with work lately. Had to push this aside for the time being. Hoping to restart soon.
As for the missing pieces of codes, what you’re reading are those bits from the platform files sources which are not using the device tree method. In kernel 4.4, I’ve replaced those bits with devicetree definitions.
From the logs generated when I tested my builds, it appears the NSS cores are initialize. Unfortunately when I tried to use the NSS cores using the crypto functions, it always sends a panic signal back to the Linux kernel driver. I’m stuck at this point.
As I understand it, having the NSS cores initialized is the first part. In order for the NSS cores to be used for traffic offload we need to load the ecm driver as well. It looks like ecm is the one that orchestrated the entire offload operations.