Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500)

ACwifidude · December 17, 2020, 11:38pm

Have Master and OpenWrt 22.03 builds for ipq806x devices with hardware offloading using the two NSS cores. All Master R7800 builds are tested before positing (router boots + wifi works). I don't own a C2600, EA7500v1, EA8500, G10, NBG6817, R7500v1, R7500v2, or d7800 (builds for these are untested - make sure you known how to tftp if you get in trouble)

Build Goals:

Maximum ipq806x performance for Gigabit WAN connections.
Simple and compatible with master (able to flash back and forth if you like testing different builds)
Optimize support for 802.11 v/k
Simple set of packages focused on statistics / graphs / monitoring

Features:

NSS drivers for NAT offloading & SQM offloading (NSS Fq_codel)
USB / LEDs / Drive mounting works
Longer wait for failsafe button
Wpad-OpenSSL
OpenSSL Luci + Luci stats
PPPoE
Ath10k-ct or ath10k drivers builds. The builds with the “ath10k” in the name are running mainline firmware and drivers. The builds with no additional description are running the OpenWRT default ath10k-ct firmware and drivers.
Extras: IPv6 support, adblock, ddns, htop, openvpn-openssl, and wireguard.

Known Issues:

Luci SQM and normal SQM config doesn’t work. Have to use custom config.

Ground rules:

Appreciate feedback and improvements relative to the goals of the build
Appreciate config recommendations to optimize performance and 802.11 v/k
Appreciate developer support to fix bugs and maximize the potential of ipq806x with NSS hardware offloading support.

Master + NSS Hardware Offloading Download: (sysupgrade and factory image in bin folder)

Ipq806x Targets (most devices):

Chromium Targets:

OpenWrt 23.05 (Stable) + NSS Hardware Offloading Download: (sysupgrade and factory image in bin folder)

OpenWrt 22.03 (Stable) + NSS Hardware Offloading Download: (sysupgrade and factory image in bin folder)

Wireless Performance: (with the below settings on a R7800 using an iphone 2x2 client)

ath10k-ct, 5.10 Kernel with NSS Hardware Offloading
[SUM]   0.00-30.01  sec  2.30 GBytes   659 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.99 GBytes   569 Mbits/sec  189             sender

ath10k (OpenWrt with no offloading)
[SUM]   0.00-30.01  sec  1.60 GBytes   459 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.14 GBytes   326 Mbits/sec  699             sender

ath10k-ct (OpenWrt with no offloading)
[SUM]   0.00-30.01  sec  1.53 GBytes   437 Mbits/sec                  receiver
[SUM]   0.00-30.01  sec  1.21 GBytes   347 Mbits/sec  763             sender

Wired Performance: (No NSS SQM, using build default settings)

ACwifidude · December 17, 2020, 11:39pm

Replicating my build from scratch:

Master:

git clone -b kernel5.15-nss-qsdk11 https://github.com/ACwifidude/openwrt.git

Openwrt 22.03:

git clone -b openwrt-22.03-nss-qsdk10.0 https://github.com/ACwifidude/openwrt.git

If you are building for one type of router you can change the first section of the diffconfig on github to just one device if you are building for a particular model (delete all the routers you don’t want to build for, follow the instructions below after the # sign):


# delete the “multi profile” line
# delete the line ending in ""
CONFIG_TARGET_MULTI_PROFILE=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_netgear_r7800=""

My diffconfig file is set up for ath10k-ct driver/firmware. If you want to use ath10k driver/firmware instead - use the diffconfig-ath10k file.

When your diffconfig is to your liking - this is how to prep and build (I have a 4 CPU system, change the last line to the number of CPUs in your system):


./scripts/feeds update -a && ./scripts/feeds install -a && cp diffconfig .config && make defconfig && ./scripts/getver.sh

make -j5

Rebasing with Master or OpenWrt 22.03 (make sure you are in the openwrt folder, rebase with the latest updates - then you can build as per above)

#Remove “rebase” commit (this gives you a clean build environment - it deletes the final bin content and diffconfig files, I’d copy the diffconfig to a separate folder before running this command)
git reset --hard HEAD~1

git remote add upstream https://git.openwrt.org/openwrt/openwrt.git

#Master Rebase:
git fetch upstream && git rebase upstream/master 

#OpenWrt 22.03 Rebase:
git fetch upstream && git rebase upstream/openwrt-22.03

If you need further help the OpenWrt wiki is well written and has further build system usage instructions:

Recommended configuration (build is the master defaults (CPU has been further optimized as per below), you’ll have to do all the rest yourself):

Firewall Software & Hardware offloading disabled
Use the default ondemand optimized settings:


    echo 600000 > /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
	echo 600000 > /sys/devices/system/cpu/cpu1/cpufreq/scaling_min_freq
	echo 25 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold
	echo 10 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor

(or switch to the performance governor)

echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor; echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor

Irqbalance disabled, packet steering disabled

uci set irqbalance.irqbalance.enabled=0; uci set network.globals.packet_steering=0; uci commit

Custom NSS fq_codel startup script if SQM is desired (below is set to 900/900). Recommend setting to 5% below your ISP provided speed and then adjusting from there as per your testing. Maximum is ~900Mbit.

modprobe nss-ifb

ip link set up nssifb

# Shape ingress traffic to 900 Mbit with chained NSSFQ_CODEL
tc qdisc add dev nssifb root handle 1: nsstbl rate 900Mbit burst 1Mb
tc qdisc add dev nssifb parent 1: handle 10: nssfq_codel limit 10240 flows 1024 quantum 1514 target 5ms interval 100ms set_default

# Shape egress traffic to 900 Mbit with chained NSSFQ_CODEL
tc qdisc add dev eth0 root handle 1: nsstbl rate 900Mbit burst 1Mb
tc qdisc add dev eth0 parent 1: handle 10: nssfq_codel limit 10240 flows 1024 quantum 1514 target 5ms interval 100ms set_default

802.11v,k enabled on 5ghz radio (change to your timezone)

wireless settings


uci set wireless.default_radio0.ieee80211v=1; uci set wireless.default_radio0.ieee80211k=1; uci set wireless.default_radio0.bss_transition=1; uci set wireless.default_radio0.wnm_sleep_mode=1; uci set wireless.default_radio0.time_advertisement=2; uci set wireless.default_radio0.time_zone=CST6CDT,M3.2.0,M11.1.0; uci commit

Make custom DNS internal sites work (if you have custom sites):


ifconfig br-lan promisc

turn custom internal sites off:


ifconfig br-lan -promisc

kmods don’t install like normal on this build. If you want a custom set of kmods the best solution is to build from scratch. Here is the full listing of packages that are built when I build:

MasterNSS builds:
https://github.com/ACwifidude/openwrt/tree/kernel5.15-nss-qsdk11bin/targets/ipq806x/generic/packages-MasterNSS

MasterNSS-ath10k builds:

Example install of a package (doesn't always work)-


opkg install https://github.com/ACwifidude/openwrt/raw/kernel5.15-nss-qsdk11bin/targets/ipq806x/generic/packages-MasterNSS/kmod-bluetooth_5.10.107-1_arm_cortex-a15_neon-vfpv4.ipk

Want Ath10k instead of Ath10k-ct?

Load the ath10k version to get both the driver and firmware. If you want to just change the firmware you can switch like this:


opkg update && opkg remove ath10k-firmware-qca9984-ct && opkg install ath10k-firmware-qca9984

Looking for more of a minimalist / no frills build with the minimum active services running?

Add this script to your startup script on your router to disable “extra” services:


# these services do not run on the router
for i in adblock nlbwmon openvpn sqm; do
  if /etc/init.d/"$i" enabled; then
    /etc/init.d/"$i" disable
    /etc/init.d/"$i" stop
  fi
done

Add this script to your startup script on your dumb Access Point(s), NOT YOUR MAIN ROUTER - this disables the firewall!!!! to disable “extra” services:


# these services do not run on dumb APs
for i in firewall dnsmasq odhcpd adblock nlbwmon openvpn ddns sqm; do
  if /etc/init.d/"$i" enabled; then
    /etc/init.d/"$i" disable
    /etc/init.d/"$i" stop
  fi
done

darksky · December 19, 2020, 2:57pm

I read some people suspect this is needed for stability but in my case, the R7800 is perfectly stable running the schedutil governor with the NSS patches enabled.

ACwifidude · December 19, 2020, 3:07pm

I haven’t had any stability problems either. I’ve found all the governors work fine. I’ve been rocking the performance governor recently and it is going well.

eems01 · December 19, 2020, 5:10pm

Been a Kong fan forever but with a long holiday break I figured I give your minimal build a try since we have similar setups with multiple R7800s as APs and I'll have time to troubleshoot. Fast transitioning was a reason I briefly tried other builds over the summer to use DAWN but never felt it improved transitioning. Can you please post what you settled on now for a reliable DAWN configuration with the latest builds?

Note: One issue I've always had on my network setup is that I'm not able to use FT over DS and must use FT over the Air or else I get a long pause on my devices when they switch APs. If that is required for DAWN to work properly then that maybe part of the issues I've seen in the past.

ACwifidude · December 19, 2020, 5:31pm

802.11 v and k are still not a 100% in OpenWRT - I’ve included and activated as much as is out there and my transitions are pretty smooth. My APs all are hardwired to the original router and I’ve found ft over ds offers the best setup.

This is what I’ve added beyond turning on the majority of the dawn features (I don’t think the broadcast command does anything):


uci set dawn.@network[0].broadcast_ip='192.168.1.255'; uci commit dawn

uci add_list umdns.@umdns[0].network='wan'; uci commit umdns

This is my config- I’ll defer to @PolynomialDivision to see if he has any more tips on optimizing 802.11 k,v with the recent master commits + his dawn package


root@OpenWrt:~# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option hwmode '11a'
        option path 'soc/1b500000.pci/pci0000:00/0000:00:00.0/0000:01:00.0'
        option htmode 'VHT80'
        option txpower '20'
        option country 'US'
        option legacy_rates '0'
        option beacon_int '101'
        option channel '161'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option ssid ''
        option encryption 'psk2+ccmp'
        option key ''
        option ieee80211r '1'
        option ft_over_ds '1'
        option ft_psk_generate_local '1'
        option wpa_disable_eapol_key_retries '1'
        option ieee80211v '1'
        option ieee80211k '1'
        option bss_transition '1'
        option wnm_sleep_mode '1'
        option time_advertisement '2'
        option time_zone 'CST6CDT,M3.2.0,M11.1.0'

config wifi-device 'radio1'
        option type 'mac80211'
        option channel '11'
        option hwmode '11g'
        option path 'soc/1b700000.pci/pci0001:00/0001:00:00.0/0001:01:00.0'
        option htmode 'HT20'
        option txpower '20'
        option country 'US'
        option legacy_rates '0'
        option beacon_int '191'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option ssid ''
        option encryption 'psk2+ccmp'
        option key ''

root@OpenWrt:~# cat /etc/config/dawn

config network
        option broadcast_port '1025'
        option tcp_port '1026'
        option network_option '2'
        option shared_key 'Niiiiiiiiiiiiiik'
        option iv 'Niiiiiiiiiiiiiik'
        option use_symm_enc '1'
        option collision_domain '-1'
        option bandwidth '-1'
        option broadcast_ip '192.168.1.255'

config ordering
        option sort_order 'cbfs'

config hostapd
        option hostapd_dir '/var/run/hostapd'

config times
        option update_client '10'
        option denied_req_threshold '30'
        option remove_client '15'
        option remove_probe '30'
        option remove_ap '460'
        option update_hostapd '10'
        option update_tcp_con '10'
        option update_chan_util '5'
        option update_beacon_reports '20'

config metric
        option ap_weight '0'
        option no_ht_support '0'
        option no_vht_support '0'
        option rssi '10'
        option low_rssi '-500'
        option freq '100'
        option chan_util '0'
        option max_chan_util '-500'
        option rssi_val '-60'
        option low_rssi_val '-80'
        option chan_util_val '140'
        option max_chan_util_val '170'
        option bandwidth_threshold '6'
        option use_station_count '1'
        option max_station_diff '1'
        option deny_auth_reason '1'
        option deny_assoc_reason '17'
        option use_driver_recog '1'
        option min_number_to_kick '3'
        option chan_util_avg_period '3'
        option set_hostapd_nr '1'
        option op_class '0'
        option duration '0'
        option mode '0'
        option scan_channel '0'
        option ht_support '10'
        option vht_support '100'
        option min_probe_count '2'
        option eval_probe_req '1'
        option eval_auth_req '1'
        option eval_assoc_req '1'

root@OpenWrt:~# cat /etc/config/umdns

config umdns
        option jail '1'
        list network 'lan'
        list network 'wan'

PolynomialDivision · December 19, 2020, 5:40pm

Sry, that I'm so unresponsive. I currently have so much to do.
This config

        option eval_probe_req '1'
        option eval_auth_req '1'
        option eval_assoc_req '1'

can be a bit problematic, because it does aggressive steering of clients. But if you have no issues, then it is fine.

I'm currently alone with DAWN and right now I am doing different IPv6 project. There is now even another project https://github.com/blogic/usteer I hope we can somehow combine our work.

If I finally have that IPv6 working I will go further with dawn.

PolynomialDivision · December 19, 2020, 5:42pm

The broadcast only does something, if u use it. The standard option is tcp with umdns.

ACwifidude · December 19, 2020, 5:53pm

I have all Apple clients on 5 ghz - they seem to like the settings and provide predictable roaming.

Master has been getting several BSS commits and several more are in staging. Hope you get some developer friends to synchronize on getting 802.11 k,v mainstream 100% working.

PolynomialDivision · December 19, 2020, 6:31pm

There was a talk on battle mesh: "NEWS FROM THE HOSTAPD LAND BATTLEMESH 2020" that provides information about all the changes.

Gilgamesh · December 20, 2020, 1:09am

@ACwifidude would love if you could add adblock and upnp, also why not use the Luci 2020 theme as it's much more mobile friendly? will WiFi offloading (NSS) be enabled at some point? thank you!

ACwifidude · December 20, 2020, 1:41am

Wifi offloading was working until master was updated to 5.8 ath10k-ct drivers. The patch just needs to be updated from 5.4 to 5.8. When the developers have time it’ll be back.

I don’t personally use Adblock or upnp but feel free to build off my GitHub and add it in.

Gilgamesh · December 20, 2020, 1:59am

ok thank you!

I was thinking of doing that but never done it before as I just used to flash Kongs build that already have those enabled, but if you can guide me on how to do that I would appreciate it, either here or drop me a PM. thanks.

ACwifidude · December 20, 2020, 4:09am

Easy enough. If you want to replicate it I have detailed instructions on the second post, just add the packages you desire to the diffconfig

keithspg · December 20, 2020, 11:51pm

@ACwifidude The last image I grabbed from you (20201208 OpenWrt SNAPSHOT r15149+64-28a9ac74cc) seemed to be the most stable so far. It ran over a week before rebooting. At least 10 days...
I logged in this afternoon and noticed that it rebooted ~ 3 days ago and decided to grab this 'latest' image (R7800-20201219-MasterNSS-sysupgrade.bin) but this did not load on my router. I had to do a tftp reinstall after I did a force install When I try to load this image over the one from 20201208, I get this:

Sun Dec 20 17:46:57 CST 2020 upgrade: Image metadata not present Sun Dec 20 17:46:57 CST 2020 upgrade: Use sysupgrade -F to override this check when downgrading or flashing to vendor firmware Image check failed.

The uploaded image file does not contain a supported format. Make sure that you choose the generic image format for your platform.

Did I do something wrong? I have images from your archive from 20201022 through the 20201208 and all 'just flashed' and rebooted.

ACwifidude · December 21, 2020, 12:10am

Hmm. Don’t know - I have that build loaded on three r7800s currently. Any other errors when you try to load it?

I’ll do a clean new build and post it up later tonight.

ACwifidude · December 21, 2020, 3:43am

Try the new build and see if it resolves your issue.

Kostja_V · December 21, 2020, 7:12am

May be, somebody has a build for ZYXEL Armor Z2 (NBG6817)?
I am very interested to test it.

ACwifidude · December 21, 2020, 1:11pm

I only have r7800s - the devices are similar enough and should work if you build with zyxel_nbg6817 as the device type.

Feel free to build off my base. I could give it a whirl and get you an untested version if you don’t have a build system.

keithspg · December 21, 2020, 2:23pm

That worked. I must have had a bad download or 'did it wrong' somehow. I'm now on OpenWrt SNAPSHOT r15241+72-3ab695368a.
The only issue I have had is the spontaneous reboot. It reboots pretty quickly, so it is not a huge deal, but, still... To me it feels like a stack issue or some memory issue because it 'just happens'. Does not seem to be dependent on wifi or 'heavy use'. Is there something I could be doing to diagnose the spontaneous reboot?