IPQ806x NSS Drivers

@darksky Have you tried to simply run

make world

before the full build? If this runs successfully, your toolchain should be ok. If it fails, you need to look into your setup

I have been building my own duplicate of @ACwifidude repo based on the commands shown. I just updated it and re-built and the date is 20201115. and the resulting image appears to be the same as what is in Chris' git. When I install it, though, there are missing packages and stuff doesn't work. I have removed pretty much everything from the router to run this firmware. The only thing I add is ddns. The problem is that I cannot install 'drill' (It does not even show up as a package to install) and curl seems broken so ddns doesn't work.

I tried to add these packages to the my build by editing the diffconfig and rebuilding (ddns packages and wireguard) so I could bring things back onto the router, (ddns, VPN) but when I built it, it failed and bricked the router. I had to tftp back to the start. So I loaded the 20201115 image and tried to add the packages and ran into this.

Is there any possible way we can base the nss packages on 19.07.4? Stuff just doesn't work in the bleeding edge. Firewall time based settings do not work, and DDNS are the 2 things I'd like to work.

If you want specific packages on the image, change the =m to =y in your diffconfig.
As for using the stable code tree, I would like the same, the family is not happy when the router crashes mid-day. I think the main question is will the NSS code work with the older kernel

1 Like

i rebuilt my image yesterday evening updating both master and ansuel's stuff (and with hnyman's patches also!) and it went all ok, so it seems there is nothing strange.. i'd check your build environment @darksky

Btw, i'm not updating my system, i'm still waiting for the first reboot :slight_smile:

Uptime	13d 14h 2m 2s

I ran make -j9 world and it errored out at the same point as when I simply ran make -j9:

...
 make[3] -C package/utils/busybox compile
 make[3] -C feeds/luci/applications/luci-app-opkg compile
 make[3] -C feeds/luci/collections/luci compile
 make[3] -C package/kernel/mac80211 compile
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
make: *** [/scratch/union/include/toplevel.mk:242: world] Error 1

Complete build log.

I should say that I can successfully build master with this setup/toolchain. As I sanity check, I will start from scratch (empty dir) and post back.

git clone -b kernel5.4-nss-qsdk10.0  https://github.com/ACwifidude/openwrt.git nss
cd nss
git remote add upstream https://git.openwrt.org/openwrt/openwrt.git\ngit fetch upstream && git rebase upstream/master
cp /scratch/union/defconfig .config
# edit .config adding [these 7 lines](https://github.com/ACwifidude/openwrt/blob/kernel5.4-nss-qsdk10.0/diffconfig#L9-L16)
make -j9

Will update the thread when it finishes... since it's rebuilding the toolchain might take some time.

@darksky Try from scratch, but I would also try this

make -j1 V=s world

This should give you more details about the error

1 Like

Thanks for the helpful advice, all. I think I understand the problem. It is that I accidentally ran diffconfig.sh on a version of my options lacking the 7 lines needed for use the NSS code :blush: I did it repeatedly since it was a typo in my build script.

When I pointed the script to the correct target, the build with my existing toolchain completed successfully.

EDIT: I also wanted to share that cloning Ansuel's branch and trying to apply master to it can be problematic as others have mentioned above. I found that I am able to rebase his branch onto a branch of master more cleanly and without errors.

Someone new to trying NSS might try this procedure instead:

# clone openwrt
git clone https://github.com/openwrt/openwrt.git
cd openwrt

# add and get Ansuel's fork with nss code
git remote add nss https://github.com/Ansuel/openwrt.git
git fetch nss

# make a branch of master
git checkout -b my-nss-branch

# rebase  Ansuel's code into the branch of master you just made
git rebase nss/kernel5.4-nss-qsdk10.0

Note that you will need to manually add a few files to get the rebase to complete. Just run git status to see the unmerged files. As I type this reply, there are just three. I show the output for illustrative purposes.

% git status
interactive rebase in progress; onto b27dbbdf1c
Last commands done (196 commands done):
   pick a7afeb3142 ipq40xx: remove support for kernel 4.19
...
Unmerged paths:
  (use "git restore --staged <file>..." to unstage)
  (use "git add <file>..." to mark resolution)
	added by us:     target/linux/ipq806x/files/include/net/netfilter/nf_conntrack_dscpremark_ext.h
	added by us:     target/linux/ipq806x/files/include/uapi/linux/tc_act/tc_nss_mirred.h
	added by us:     target/linux/ipq806x/files/net/netfilter/nf_conntrack_dscpremark_ext.c

So as I said, add these and continue the rebase:

git add target/linux/ipq806x/files/include/net/netfilter/nf_conntrack_dscpremark_ext.h target/linux/ipq806x/files/include/uapi/linux/tc_act/tc_nss_mirred.h target/linux/ipq806x/files/net/netfilter/nf_conntrack_dscpremark_ext.c

git rebase --continue
2 Likes

I think there are too many changes in master for a safe proper nss build.
I got this conflict after adding all files.

Could not apply ff7f880c96... Use pppoe offload 5.4 patch from codeaurora

Here's my command when I do the rebase.

# Revert back to Ansuel's original commit because I already did a rebase few days ago
git reset --hard ORIG_HEAD

# Rebase
git fetch upstream && git rebase upstream/master

Here's the output

$ git reset --hard ORIG_HEAD
HEAD is now at b27dbbdf1c Try to support more ipq8064 device
$ git fetch upstream && git rebase upstream/master
First, rewinding head to replay your work on top of it...
Applying: Add qca package for qsdk10.0
Applying: Add needed patch for nss driver
Applying: Update target patches and config
Using index info to reconstruct a base tree...
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8065-r7800.dts
Falling back to patching base and 3-way merge...
Auto-merging target/linux/ipq806x/files/arch/arm/boot/dts/qcom-ipq8065-r7800.dts
Applying: Use qsdk10.0 for crypto driver
Applying: Add qca-nss-cfi support
Applying: Update target patches
Applying: Fix broken cfi driver
Applying: Fix broken nf_netlink module
Applying: Add missing patch for nss-drv
Applying: Fix sfe simulated-driver with tc config enabled
Applying: Refresh patch for nss-cfi
Applying: Make ECM SFE support optional
Applying: Fix kernel panic on UDP CONNTRACK
Applying: Split SFE support patch to specific patch
Applying: mac80211: add support for ipq806x NSS firmware
Applying: Fix more kernel panic with nss crashdump
Applying: Add nss clients packages
Applying: iproute2: tc: add support for NSS qdisc
Applying: shortcut-fe: cm: bug fix
Applying: shortcut-fe: rework netfilter conntrack notification
Applying: qca-nss-ecm: rework netfilter conntrack notification
Applying: qca-rfs: rework netfilter conntrack notification
Applying: Add missing include file from clients commit
Applying: Fix compilation errors
Applying: ipq806x: qca-nss: fix nss core clock frequency (#1)
Using index info to reconstruct a base tree...
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8065.dtsi
Falling back to patching base and 3-way merge...
Applying: Add missing bonding changes
Applying: Fix compilation problem with netlink
Applying: Fix warn on gmac unload
Applying: Test backport fixes from qsdk nss brach
Applying: Add patch for ifb custom driver support
Applying: nss-ifb: NSS Physical Interface Ingress Driver
Applying: Adapt changes in ab966da455 nss qdisk with qsdk 10
Applying: mac80211: improve NSS virtual interface init
Applying: lock frequency on max to fix stability problems
Using index info to reconstruct a base tree...
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8065.dtsi
Falling back to patching base and 3-way merge...
Applying: Use newer nss bin
Applying: ipq806x: add standalone krait cache scaling driver
Applying: Rework fab scaling driver to work with krait cache
Applying: Patch upstream dtsi with nss bindings
Applying: Sync new config flag
Applying: Update mac80211 nss support patch
Applying: mac80211: ipq806x-nss: add tasklet_kill step during interface take down
Applying: Update experimental L2 scaling driver
Applying: update ecm support patch with recent kernel changes
Applying: Fix error in L2 scaling driver
Applying: Try to support r7500
Using index info to reconstruct a base tree...
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8064-r7500.dts
Falling back to patching base and 3-way merge...
Auto-merging target/linux/ipq806x/files/arch/arm/boot/dts/qcom-ipq8064-r7500.dts
Applying: Refresh patch with new iproute version
Applying: Use pppoe offload 5.4 patch from codeaurora
Applying: Try another implementation for wifi offload
Applying: Fix compilation error with pppoe offload
Applying: Fix missing function ppp_dev_index
Applying: Fix broken patch
Applying: Fix broken cfi patch
Applying: Add missing patch for mac80211 nss offload
Applying: Enable tx encap offload for ath10k
Applying: Test workaround patch for mu-mimo
Applying: Migrate ath10k-ct custom patch to normal ath10k-ct
Applying: Enable vht on 2.4Ghz
Applying: Fix patch for upstream ath10k
Applying: Try to support ea7500 v1
Using index info to reconstruct a base tree...
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8064-eax500.dtsi
Falling back to patching base and 3-way merge...
Applying: Experimental workqueue rx wifi offload
Applying: Simplify nss patch
Applying: Try to support more ipq8064 device
Using index info to reconstruct a base tree...
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8064-c2600.dts
A       target/linux/ipq806x/files-5.4/arch/arm/boot/dts/qcom-ipq8064-eax500.dtsi
Falling back to patching base and 3-way merge...
Auto-merging target/linux/ipq806x/files/arch/arm/boot/dts/qcom-ipq8064-c2600.dts

It builds fine

1 Like

I have been running the image with the NSS drivers for several hours and am noticing periods where my ssh connection freezes up. If I leave htop run on the OW device, I am seeing 100% CPU usage on both cores during the freeze ups. Dmesg is showing the following. Any thoughts?

[ 4347.033244] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4347.033324] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4347.039500] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4347.046705] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4347.054043] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4347.061352] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4347.068642] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4347.075857] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4347.083213] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4347.090487] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4354.088201] ath10k_warn: 24 callbacks suppressed
[ 4354.088218] ath10k_pci 0000:01:00.0: failed to wait for created wmi peer 40:74:e0:09:ae:02 on vdev 1: -110
[ 4354.091976] ath10k_pci 0000:01:00.0: failed to add peer 40:74:e0:09:ae:02 for vdev 1 when adding a new sta: -110
[ 4363.386367] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4363.386421] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4363.392600] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4363.399854] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4363.407139] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4363.414451] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4363.421773] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4363.429016] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped new beacon
[ 4363.436300] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4363.443631] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4372.568147] ath10k_warn: 17 callbacks suppressed
[ 4372.568160] ath10k_pci 0000:01:00.0: bss channel survey timed out
[ 4375.608162] ath10k_pci 0000:01:00.0: wmi command 36892 timeout, restarting hardware
[ 4375.608215] ath10k_pci 0000:01:00.0: failed to set dtim period for vdev 0: -11
[ 4378.648289] ath10k_pci 0000:01:00.0: failed to recalculate rts/cts prot for vdev 0: -108
[ 4378.648337] ath10k_pci 0000:01:00.0: failed to set cts protection for vdev 0: -108
[ 4378.655756] ath10k_pci 0000:01:00.0: failed to set preamble for vdev 0: -108
[ 4378.662950] ath10k_pci 0000:01:00.0: failed to set mgmt tx rate -108
[ 4378.670776] ath10k_pci 0000:01:00.0: failed to send pdev bss chan info request
[ 4378.676726] ath10k_pci 0000:01:00.0: failed to set beacon mode for vdev 1: -108
[ 4378.683507] ath10k_pci 0000:01:00.0: failed to set dtim period for vdev 1: -108
[ 4378.691374] ath10k_pci 0000:01:00.0: failed to recalculate rts/cts prot for vdev 1: -108
[ 4378.697968] ath10k_pci 0000:01:00.0: failed to set cts protection for vdev 1: -108
[ 4378.706331] ath10k_pci 0000:01:00.0: failed to set preamble for vdev 1: -108
[ 4381.723341] ieee80211 phy2: Hardware restart was requested
[ 4387.908470] ath10k_warn: 1 callbacks suppressed
[ 4387.908520] ath10k_pci 0000:01:00.0: Unknown eventid: 36933
[ 4388.269874] ath10k_pci 0000:01:00.0: device successfully recovered
[ 4390.034339] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4390.034375] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4390.040581] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4390.047786] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4390.055184] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4390.062396] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4390.069712] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4390.076930] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4390.084311] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4393.034624] ath10k_warn: 6 callbacks suppressed
[ 4393.034632] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4393.038008] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4393.045415] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4393.052643] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4393.059981] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4393.067158] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4393.074587] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4393.081800] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 4393.089142] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 1, skipped old beacon
[ 4393.096307] ath10k_pci 0000:01:00.0: SWBA overrun on vdev 0, skipped old beacon

I retried your commands, it rebased fine.

1 Like

I'd like to play a bit more with this... is there a dependency I need?

# tc qdisc add dev nssifb root handle 1: nsstbl rate 650Mbit burst 1Mb
Cannot find device "nssifb"

EDIT: I see now that a module is required (modprobe nss-ifb). This readme was helpful: https://github.com/quarkysg/openwrt/blob/master-ipq806x-nss-qsdk11/package/qca/nss-ifb/README.md

When I do the following, I am bufferbloat results comparable to when I was running non-NSS image with SQM per the dsl reports test.

modprobe nss-ifb
ip link set up nssifb
tc qdisc add dev nssifb root handle 1: nsstbl rate 650Mbit burst 1Mb
tc qdisc add dev nssifb parent 1: handle 10: nssfq_codel limit 10240 flows 1024 quantum 1514 target 5ms interval 100ms set_default
tc qdisc add dev eth0 root handle 1: nsstbl rate 20Mbit burst 1Mb
tc qdisc add dev eth0 parent 1: handle 10: nssfq_codel limit 10240 flows 1024 quantum 1514 target 5ms interval 100ms set_default
1 Like

my results are not as good, spikes to 150+ms once or twice. on cake it is rock-solid on < 10ms.

How are you measuring it (details please)? I would be interested to try it on mine.

oh, this is just on dslreports speedtest

I am not seeing a plot of bufferbloat vs time. Can you share a result and point me to the section of page? The histograms at the top can be really misleading. What are the error bars? StdDev? StdErr? Just Min/max values? We really need to see all individual samples plotted in a distribution with some corresponding statistics.

Still FW issues here, devices on wifi can hit my forward to internal website fine.
But devices on ethernet can't.
Traffic should be forwarded like this:
Internal device > external ip on router > Internal webserver

@Ansuel Wifi drops seems to be gone tho', so I think you fixed that.

Lots to follow here. After I build nss code openwrt, I have to modprobe and run tc commands and then ip link command? Can this all just go into a startup script?

I just have it in a startup script. Makes it much easier. :sunglasses:

1 Like

What is your script? Is the order of commands right like @darksky posted?

What values to use for the tc commands? Do I take my max values from speed test and multiple by 0.95 like the wiki says and use them? How can I set a different value once I run the command?

My connection is 250 down so I try:

tc qdisc add dev nssifb root handle 1: nsstbl rate 250Mbit burst 1Mb

But I want to adjust to a lower number like 225 and see if I get better bufferbloat but I get file exists error:

tc qdisc add dev nssifb root handle 1: nsstbl rate 225Mbit burst 1Mb
RTNETLINK answers: File exists