IPQ806x NSS Drivers

Two things:

  1. That is the folder where I keep my diffconfig on my build system. If you clone my directory the diffconfig is included so you can delete that part of the command.

  2. don’t run make dirclean after all the prior commands. That thoroughly cleans out the directory and deletes everything you just did.

I’d edit the diffconfig to your liking - after you’ve edited it to your own liking your build process would look like this (the 4 after the make command represent a 3 CPU system, feel free to use more CPUs if you have more CPUs):


git clone -b kernel5.4-nss-qsdk10.0  https://github.com/ACwifidude/openwrt.git
cd openwrt
git remote add upstream https://git.openwrt.org/openwrt/openwrt.git 
git fetch upstream && git rebase upstream/master && ./scripts/feeds update -a && ./scripts/feeds install -a && cp diffconfig .config && make defconfig && ./scripts/getver.sh
make -j4

1 Like

Woo Hoo. I finally built an image. Now to see if it works and if I can add the features I want.

Thanks for the help!

1 Like

Still running @ACwifidude latest and had 2 router crashes this morning. I am guessing it was the router and not the service (ATT fiber) because I could no longer get to the router UI.
Wifi load minimal. 3 computers connected 2 kids in 'school' and me at 'work'. It has been very stable since I loaded it up till this AM. I could not get to the router to see the log. Pretty low load.
68% Memory available, Load Average: 0.02, 0.04, 0.05...
The version was 20201029. I just flashed my version that I built (20201108) and will see hos it goes. It is strange that it has been stable for a number of days then just spontaneously rebooted. 3x today.

Does anyone know if any additional fixes have been made to the NSS codebase?
The last change on @Ansuel's repo is 3 weeks old
Just wondering if it makes sense to do a new build.

It has just been master rebasing since then. I like to keep up with the latest kernel to get updates to software packages and bug fixes. I push rebases every couple days. :sunglasses:

As @ACwifidude said new build = rebase with new kernel version and patch.

Some people above were capturing some logs. To make further improvements the developers need logs of issues / crashes specific to NSS drivers.

NSS builds have been the equivalent of normal master builds for me so I’ve been only building with NSS drivers included to get maximum performance on my three r7800’s.

Thank you @ACwifidude and @Ansuel, bugfixes and better stability are always welcome.
It seems that the kernel mainline also had a few updates, so it is time to run a rebuild :slight_smile:
Regarding logs, if you have some specific steps we should include to collect them, please, feel free.

Pls don't give me too much credit... all of this is possible thx to @quarky so remember who started and make this thing real!

3 Likes

I'm getting these warnings:

WARNING: Makefile 'package/qca/qca-nss-drv/Makefile' has a dependency on 'kmod-qca-nss-dp', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-ovsmgr', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-ovsmgr', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-ovsmgr', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-hyfi-bridge', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-mcs', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-ovsmgr', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-hyfi-bridge', which does not exist
WARNING: Makefile 'package/qca/qca-nss-ecm/Makefile' has a dependency on 'kmod-qca-mcs', which does not exist

Is this critical?

Targeting to the Linksys EA7500 V1.

Not critical - those warnings are present every time I build. Let us know if it doesn’t build or doesn’t boot.

Well, it builds:

But there are some bin files for the R7800, still.

How can I get sure the EA7500 V1 binaries are with NSS drivers turned on?

Bad news: with a sysupgrade binary, upgrading my current version, it does not boot.

After 3 failed reboots, reverted to the previous flashed versions.

Any ability to capture some logs from your ea7500 v1 for @Ansuel to see if there is a modification to get it to work?

Additionally can you post your config.buildinfo from your bin folder to see if there are any config issues?

I can't get any logs, since the router doesn't boot correctly, I guess.

The config.buildinfo:

CONFIG_TARGET_ipq806x=y
CONFIG_TARGET_ipq806x_generic=y
CONFIG_TARGET_ipq806x_generic_DEVICE_linksys_ea7500-v1=y
CONFIG_BUSYBOX_CUSTOM=y
CONFIG_BUILD_PATENTED=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVEHISTORY=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVE_ON_EXIT=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_FLAGS=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_REGEXP=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_WINCH=y
CONFIG_DROPBEAR_ECC=y
CONFIG_IMAGEOPT=y
CONFIG_OPENSSL_ENGINE=y
CONFIG_OPENSSL_PREFER_CHACHA_OVER_GCM=y
CONFIG_OPENSSL_WITH_ASM=y
CONFIG_OPENSSL_WITH_CHACHA_POLY1305=y
CONFIG_OPENSSL_WITH_CMS=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_ERROR_MESSAGES=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_WITH_SRP=y
CONFIG_OPENSSL_WITH_TLS13=y
CONFIG_PACKAGE_6in4=y
CONFIG_PACKAGE_6rd=y
CONFIG_PACKAGE_6to4=y
CONFIG_PACKAGE_MAC80211_NSS_SUPPORT=y
CONFIG_PACKAGE_NTFS-3G_HAS_PROBE=y
CONFIG_PACKAGE_ath10k-firmware-qca9984-ct=y
# CONFIG_PACKAGE_ath10k-firmware-qca99x0-ct is not set
CONFIG_PACKAGE_block-mount=y
CONFIG_PACKAGE_ca-certificates=y
CONFIG_PACKAGE_cgi-io=y
CONFIG_PACKAGE_collectd=y
CONFIG_PACKAGE_collectd-mod-conntrack=y
CONFIG_PACKAGE_collectd-mod-cpu=y
CONFIG_PACKAGE_collectd-mod-cpufreq=y
CONFIG_PACKAGE_collectd-mod-entropy=y
CONFIG_PACKAGE_collectd-mod-exec=y
CONFIG_PACKAGE_collectd-mod-interface=y
CONFIG_PACKAGE_collectd-mod-iwinfo=y
CONFIG_PACKAGE_collectd-mod-load=y
CONFIG_PACKAGE_collectd-mod-memory=y
CONFIG_PACKAGE_collectd-mod-network=y
CONFIG_PACKAGE_collectd-mod-ping=y
CONFIG_PACKAGE_collectd-mod-rrdtool=y
CONFIG_PACKAGE_collectd-mod-sqm=y
CONFIG_PACKAGE_collectd-mod-thermal=y
CONFIG_PACKAGE_collectd-mod-uptime=y
CONFIG_PACKAGE_dawn=y
CONFIG_PACKAGE_hostapd-utils=y
CONFIG_PACKAGE_ip6tables-mod-nat=y
CONFIG_PACKAGE_iptables-mod-conntrack-extra=y
CONFIG_PACKAGE_iptables-mod-extra=y
CONFIG_PACKAGE_iptables-mod-ipopt=y
CONFIG_PACKAGE_iptables-mod-physdev=y
CONFIG_PACKAGE_irqbalance=y
CONFIG_PACKAGE_kmod-br-netfilter=y
CONFIG_PACKAGE_kmod-crypto-acompress=y
CONFIG_PACKAGE_kmod-crypto-aead=y
CONFIG_PACKAGE_kmod-crypto-arc4=y
CONFIG_PACKAGE_kmod-crypto-authenc=y
CONFIG_PACKAGE_kmod-crypto-cbc=y
CONFIG_PACKAGE_kmod-crypto-crc32c=y
CONFIG_PACKAGE_kmod-crypto-deflate=y
CONFIG_PACKAGE_kmod-crypto-des=y
CONFIG_PACKAGE_kmod-crypto-ecb=y
CONFIG_PACKAGE_kmod-crypto-echainiv=y
CONFIG_PACKAGE_kmod-crypto-hash=y
CONFIG_PACKAGE_kmod-crypto-hmac=y
CONFIG_PACKAGE_kmod-crypto-manager=y
CONFIG_PACKAGE_kmod-crypto-md4=y
CONFIG_PACKAGE_kmod-crypto-md5=y
CONFIG_PACKAGE_kmod-crypto-null=y
CONFIG_PACKAGE_kmod-crypto-pcompress=y
CONFIG_PACKAGE_kmod-crypto-sha1=y
CONFIG_PACKAGE_kmod-crypto-sha256=y
CONFIG_PACKAGE_kmod-fs-cifs=y
CONFIG_PACKAGE_kmod-fs-exfat=y
CONFIG_PACKAGE_kmod-fs-ext4=y
CONFIG_PACKAGE_kmod-fs-hfsplus=y
CONFIG_PACKAGE_kmod-fs-msdos=y
CONFIG_PACKAGE_kmod-fs-vfat=y
CONFIG_PACKAGE_kmod-fuse=y
CONFIG_PACKAGE_kmod-ifb=y
CONFIG_PACKAGE_kmod-ipsec=y
CONFIG_PACKAGE_kmod-ipt-conntrack-extra=y
CONFIG_PACKAGE_kmod-ipt-extra=y
CONFIG_PACKAGE_kmod-ipt-ipopt=y
CONFIG_PACKAGE_kmod-ipt-nat6=y
CONFIG_PACKAGE_kmod-ipt-physdev=y
CONFIG_PACKAGE_kmod-ipt-raw=y
CONFIG_PACKAGE_kmod-iptunnel=y
CONFIG_PACKAGE_kmod-iptunnel4=y
CONFIG_PACKAGE_kmod-ledtrig-default-on=y
CONFIG_PACKAGE_kmod-ledtrig-heartbeat=y
CONFIG_PACKAGE_kmod-ledtrig-netdev=y
CONFIG_PACKAGE_kmod-ledtrig-timer=y
CONFIG_PACKAGE_kmod-lib-crc16=y
CONFIG_PACKAGE_kmod-lib-zlib-deflate=y
CONFIG_PACKAGE_kmod-lib-zlib-inflate=y
CONFIG_PACKAGE_kmod-nf-conntrack-netlink=y
CONFIG_PACKAGE_kmod-nf-nat6=y
CONFIG_PACKAGE_kmod-nfnetlink=y
CONFIG_PACKAGE_kmod-nls-cp1250=y
CONFIG_PACKAGE_kmod-nls-cp437=y
CONFIG_PACKAGE_kmod-nls-cp850=y
CONFIG_PACKAGE_kmod-nls-iso8859-1=y
CONFIG_PACKAGE_kmod-nls-iso8859-15=y
CONFIG_PACKAGE_kmod-nls-utf8=y
CONFIG_PACKAGE_kmod-nss-ifb=y
CONFIG_PACKAGE_kmod-qca-nss-drv=y
CONFIG_PACKAGE_kmod-qca-nss-drv-qdisc=y
CONFIG_PACKAGE_kmod-qca-nss-ecm-standard=y
CONFIG_PACKAGE_kmod-qca-nss-gmac=y
CONFIG_PACKAGE_kmod-sched-cake=y
CONFIG_PACKAGE_kmod-sched-core=y
CONFIG_PACKAGE_kmod-sit=y
CONFIG_PACKAGE_kmod-usb-storage=y
CONFIG_PACKAGE_libblkid=y
CONFIG_PACKAGE_libelf=y
CONFIG_PACKAGE_libgcrypt=y
CONFIG_PACKAGE_libgpg-error=y
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_libltdl=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
CONFIG_PACKAGE_libopenssl=y
CONFIG_PACKAGE_libopenssl-conf=y
CONFIG_PACKAGE_liboping=y
CONFIG_PACKAGE_librrd1=y
CONFIG_PACKAGE_librt=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libustream-openssl=y
# CONFIG_PACKAGE_libustream-wolfssl is not set
CONFIG_PACKAGE_libuuid=y
# CONFIG_PACKAGE_libwolfssl is not set
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-app-commands=y
CONFIG_PACKAGE_luci-app-dawn=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-app-nlbwmon=y
CONFIG_PACKAGE_luci-app-opkg=y
CONFIG_PACKAGE_luci-app-sqm=y
CONFIG_PACKAGE_luci-app-statistics=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-compat=y
CONFIG_PACKAGE_luci-lib-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-json=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-network=y
CONFIG_PACKAGE_luci-mod-status=y
CONFIG_PACKAGE_luci-mod-system=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-ssl-openssl=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_nlbwmon=y
CONFIG_PACKAGE_ntfs-3g=y
CONFIG_PACKAGE_openssl-util=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-luci=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_rrdtool1=y
CONFIG_PACKAGE_sqm-scripts=y
CONFIG_PACKAGE_tc=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
CONFIG_PACKAGE_umdns=y
# CONFIG_PACKAGE_wpad-basic-wolfssl is not set
CONFIG_PACKAGE_wpad-openssl=y
CONFIG_PACKAGE_zlib=y
CONFIG_PREINITOPT=y
CONFIG_TARGET_PREINIT_TIMEOUT=5
CONFIG_WPA_MSG_MIN_PRIORITY=4
# CONFIG_WPA_WOLFSSL is not set

PS: This router is only supported on the snapshots.

PS2: I don't know if this is relevant or not, but with the November 12th snapshops, I can't get the WAN to work. Not even with static IP address. Had to return to the November 9th one:

https://pastebin.com/yN7F97Yz

I guess it'll probably be much easier waiting for the OpenVPN kernel module and hack that to use the NSS crypto engine.

Just let you know the OpenVPN kernel module is out and under in heavy development. There are below limiations as below according to README.

== Limitations ==
This is a list of current limitations which are planned to be removed as we move forward:

  • Only client mode supported
  • Only AES-256-GCM and 'none' (with no auth) supported

https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg21065.html

Thanks for the update @Tony.He.

Good to know that this finally gets some traction. The ipq806x (and I believe for ipx807x) NSS firmware currently doesn't support GCM mode tho. I guess the developers concentrated on GCM due to it being the recommended mode currently. Hopefully adding AES-CBC-SHA-HMAC mode should be straightforward.

At the moment I don't really have much time to hack router firmware due to work. Hopefully I can restart this sometime in future soon.

1 Like

Do you think the openvpn kmod will give better perf as it will skip the userspace problem?
Also I wonder if wireguard performance can benefit of the crypto core.

Hi @Tyco89,

To follow up and to raise awareness for others on the spontaneous restarts, there seem to be an underlying bug not related to NSS that may cause router restarts.

I had 17d uptime with my NSS build before a spontaneous restart yesterday.
The crash logs points directly to the already created bug report below.

The workaround mentioned in that bug report is to use performance governor.

Your restarts may be from a different cause but it might be worth a try to change to performance governor if you are still facing frequent restart problem and not already using the performance governor.

As for the fix for that reported bug, it may take quite some time to get fixed...

Definitely. From what I’ve tested with the NSS crypto core, moving from user space to kernel space for the data plane results in 4X improvement in thruput. With further optimization, it should go higher (I hope) with better integration with the NSS firmware.

I briefly looked at the ovpn-dco code. It looks like they are creating a separate network interface and not using the Linux tun interface, which makes sense since the tun interface is mainly for user space processes to send network packets. I think using the new approach should make integration with the NSS a lot easier using the NSS virtual interface APIs.

Unfortunately the NSS firmware doesn’t support the crypto algorithm or the authentication mode that WireGuard is using. Good news is that I could get about 200mbps (if my memory serves) thruput between a mt7621 and a bcm4708 SoC router over a WAN link between two ISPs across two countries. Using ipq806x SoC routers for both ends would definitely results in higher WireGuard thruput.

1 Like