Ipq806x NSS build (Netgear R7800 / TP-Link C2600 / Linksys EA8500)

ACwifidude · March 8, 2021, 2:51am

Have both Master and OpenWrt 21.02 builds with hardware offloading using the two NSS cores. All Master R7800 builds are tested before positing (router boots + wifi works). I don't own a C2600, EA7500v1, EA8500, G10, NBG6817, R7500v1, or R7500v2 (builds for these are untested - make sure you known how to tftp if you get in trouble)

Build Goals:

Maximum ipq806x performance for Gigabit WAN connections.
Simple and compatible with master (able to flash back and forth if you like testing different builds)
Optimize support for 802.11 v/k (Dawn)
Simple set of packages focused on statistics / graphs / monitoring

Features:

NSS drivers for NAT offloading & SQM offloading (NSS Fq_codel)
USB / LEDs / Drive mounting works
Longer wait for failsafe button
Wpad-OpenSSL
OpenSSL Luci + Luci stats
Dawn decentralized network controller
Irqbalance
Ath10k-ct drivers
Extras: IPv6 support, adblock, banip, bcp38, ddns, htop, openvpn-openssl, and wireguard.

Known Issues:

Luci SQM and normal SQM config doesn’t work. Have to use custom config.

Ground rules:

Appreciate feedback and improvements relative to the goals of the build
Appreciate config recommendations to optimize performance and 802.11 v/k
Appreciate developer support to fix bugs and maximize the potential of ipq806x with NSS hardware offloading support.

Master + NSS Hardware Offloading Download: (sysupgrade and factory image in bin folder)

OpenWrt 21.02 (Stable) + NSS Hardware Offloading Download: (sysupgrade and factory image in bin folder)

ACwifidude · April 11, 2021, 1:13am

Replicating my build from scratch:

Master:

git clone -b kernel5.4-nss-qsdk10.0 https://github.com/ACwifidude/openwrt.git

Openwrt 21.02:

git clone -b openwrt-21.02-nss-qsdk10.0 https://github.com/ACwifidude/openwrt.git

This is my diffconfig. Feel free to edit my diffconfig to your needs:

# Use "make defconfig" to expand this to a full .config
CONFIG_TARGET_ipq806x=y
CONFIG_TARGET_ipq806x_generic=y
CONFIG_TARGET_MULTI_PROFILE=y
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_netgear_r7800=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_netgear_r7800=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_linksys_ea8500=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_linksys_ea8500=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_linksys_ea7500-v1=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_linksys_ea7500-v1=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_tplink_c2600=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_tplink_c2600=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_netgear_r7500=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_netgear_r7500=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_netgear_r7500v2=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_netgear_r7500v2=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_zyxel_nbg6817=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_zyxek_nbg6817=""
CONFIG_TARGET_DEVICE_ipq806x_generic_DEVICE_asrock_g10=y
CONFIG_TARGET_DEVICE_PACKAGES_ipq806x_generic_DEVICE_asrock_g10=""
CONFIG_TARGET_PER_DEVICE_ROOTFS=y

# exfat is patented
CONFIG_BUILD_PATENTED=y

# NSS Drivers
CONFIG_PACKAGE_kmod-qca-nss-drv=y
CONFIG_PACKAGE_kmod-qca-nss-drv-qdisc=y
CONFIG_PACKAGE_kmod-qca-nss-ecm-standard=y
CONFIG_PACKAGE_kmod-qca-nss-gmac=y
CONFIG_PACKAGE_kmod-nss-ifb=y
CONFIG_PACKAGE_iptables-mod-physdev=y
CONFIG_PACKAGE_kmod-ipt-physdev=y
CONFIG_PACKAGE_kmod-qca-nss-drv-pppoe=y
CONFIG_PACKAGE_MAC80211_NSS_SUPPORT=y

# Longer waiting for failsafe button push
CONFIG_IMAGEOPT=y
CONFIG_PREINITOPT=y
CONFIG_TARGET_PREINIT_TIMEOUT=5

# Busybox tweaks
CONFIG_BUSYBOX_CUSTOM=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVEHISTORY=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVE_ON_EXIT=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_FLAGS=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_REGEXP=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_WINCH=y

# Add-on programs
CONFIG_PACKAGE_irqbalance=y
CONFIG_DROPBEAR_ECC=y
CONFIG_PACKAGE_openvpn-openssl=y
CONFIG_PACKAGE_htop=y
CONFIG_PACKAGE_kmod-cryptodev=y
CONFIG_PACKAGE_libopenssl-devcrypto=y

# USB device mount & file systems support
CONFIG_PACKAGE_block-mount=y
CONFIG_PACKAGE_kmod-usb-storage=y
CONFIG_PACKAGE_kmod-fs-cifs=y
CONFIG_PACKAGE_kmod-fs-exfat=y
CONFIG_PACKAGE_libblkid=y
CONFIG_PACKAGE_kmod-fs-ext4=y
CONFIG_PACKAGE_kmod-fs-hfsplus=y
CONFIG_PACKAGE_kmod-fs-msdos=y
CONFIG_PACKAGE_kmod-fs-vfat=y
CONFIG_PACKAGE_ntfs-3g=y
CONFIG_PACKAGE_kmod-nls-cp1250=y
CONFIG_PACKAGE_kmod-nls-cp437=y
CONFIG_PACKAGE_kmod-nls-cp850=y
CONFIG_PACKAGE_kmod-nls-iso8859-1=y
CONFIG_PACKAGE_kmod-nls-iso8859-15=y
CONFIG_PACKAGE_kmod-nls-utf8=y

# IPv6 support
CONFIG_PACKAGE_6in4=y
CONFIG_PACKAGE_6to4=y
CONFIG_PACKAGE_6rd=y

# IPv6 NAT support (ip6tables NAT extensions, ipt-nat6 and nf-nat6 kmods)
CONFIG_PACKAGE_ip6tables-mod-nat=y

# WLAN/WPS support
CONFIG_PACKAGE_hostapd-utils=y
CONFIG_WPA_MSG_MIN_PRIORITY=4
CONFIG_PACKAGE_wpad-openssl=y
# CONFIG_PACKAGE_wpad-basic-wolfssl is not set
# CONFIG_PACKAGE_libustream-wolfssl is not set

# SSL certificates
CONFIG_PACKAGE_ca-certificates=y

# Luci (SSL from OpenSSL)
CONFIG_PACKAGE_luci-ssl-openssl=y
CONFIG_PACKAGE_luci-app-commands=y
CONFIG_PACKAGE_luci-app-sqm=y
CONFIG_PACKAGE_luci-app-dawn=y
CONFIG_PACKAGE_luci-app-adblock=y
CONFIG_PACKAGE_luci-app-openvpn=y
CONFIG_PACKAGE_luci-app-ddns=y
CONFIG_PACKAGE_luci-app-wireguard=y
CONFIG_PACKAGE_luci-theme-openwrt-2020=y
CONFIG_PACKAGE_luci-app-bcp38=y
CONFIG_PACKAGE_luci-app-banip=y

# Luci statistics
CONFIG_PACKAGE_luci-app-statistics=y
CONFIG_PACKAGE_collectd-mod-conntrack=y
CONFIG_PACKAGE_collectd-mod-cpufreq=y
CONFIG_PACKAGE_collectd-mod-dhcpleases=y
CONFIG_PACKAGE_collectd-mod-entropy=y
CONFIG_PACKAGE_collectd-mod-exec=y
CONFIG_PACKAGE_collectd-mod-interface=y
CONFIG_PACKAGE_collectd-mod-iwinfo=y
CONFIG_PACKAGE_collectd-mod-load=y
CONFIG_PACKAGE_collectd-mod-memory=y
CONFIG_PACKAGE_collectd-mod-network=y
CONFIG_PACKAGE_collectd-mod-ping=y
CONFIG_PACKAGE_collectd-mod-sqm=y
CONFIG_PACKAGE_collectd-mod-thermal=y
CONFIG_PACKAGE_collectd-mod-wireless=y
CONFIG_PACKAGE_collectd-mod-uptime=y

# nlbwmon app
CONFIG_PACKAGE_luci-app-nlbwmon=y

Change the first section of the diffconfig to just one device if you are building for a particular model (r7800 as the example):


# Use "make defconfig" to expand this to a full .config
CONFIG_TARGET_ipq806x=y
CONFIG_TARGET_ipq806x_generic=y
CONFIG_TARGET_ipq806x_generic_DEVICE_netgear_r7800=y

When your diffconfig is to your liking - this is how to prep and build (I have a 4 CPU system, change the last line to the number of CPUs in your system):


./scripts/feeds update -a && ./scripts/feeds install -a && cp diffconfig .config && make defconfig && ./scripts/getver.sh

make -j5

Rebasing with Master or OpenWrt 21.02 (make sure you are in the openwrt folder)

git remote add upstream https://git.openwrt.org/openwrt/openwrt.git

#Master Rebase:
git fetch upstream && git rebase upstream/master 

#OpenWrt 21.02 Rebase:
git fetch upstream && git rebase upstream/openwrt-21.02

Recommended configuration (build is the master defaults, you’ll have to do this yourself):

Firewall Software & Hardware offloading disabled
Performance Governor

echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor; echo performance > /sys/devices/system/cpu/cpufreq/policy1/scaling_governor

Irqbalance enabled

uci set irqbalance.irqbalance.enabled=1; uci commit

Custom NSS fq_codel startup script if SQM is desired (below is set to 900/900). Recommend setting to 5% below your ISP provided speed and then adjusting from there as per your testing. Maximum is ~900Mbit.

modprobe nss-ifb

ip link set up nssifb

# Shape ingress traffic to 900 Mbit with chained NSSFQ_CODEL
tc qdisc add dev nssifb root handle 1: nsstbl rate 900Mbit burst 1Mb
tc qdisc add dev nssifb parent 1: handle 10: nssfq_codel limit 10240 flows 1024 quantum 1514 target 5ms interval 100ms set_default

# Shape egress traffic to 900 Mbit with chained NSSFQ_CODEL
tc qdisc add dev eth0 root handle 1: nsstbl rate 900Mbit burst 1Mb
tc qdisc add dev eth0 parent 1: handle 10: nssfq_codel limit 10240 flows 1024 quantum 1514 target 5ms interval 100ms set_default

802.11v,k enabled on 5ghz radio (change to your timezone)


uci add_list umdns.@umdns[0].network='wan'; uci set wireless.default_radio0.ieee80211v=1; uci set wireless.default_radio0.ieee80211k=1; uci set wireless.default_radio0.bss_transition=1; uci set wireless.default_radio0.wnm_sleep_mode=1; uci set wireless.default_radio0.time_advertisement=2; uci set wireless.default_radio0.time_zone=CST6CDT,M3.2.0,M11.1.0; uci commit


uci set dawn.@metric[0].ht_support='10'; uci set dawn.@metric[0].vht_support='100'; uci set dawn.@metric[0].min_probe_count='2'; uci set dawn.@metric[0].eval_probe_req='1'; uci set dawn.@metric[0].eval_auth_req='1'; uci set dawn.@metric[0].eval_assoc_req='1'; uci commit

Make custom DNS internal sites work (if you have custom sites):


ifconfig br-lan promisc

turn custom internal sites off:


ifconfig br-lan -promisc

Want Ath10k instead of Ath10k-ct?


opkg update && opkg remove ath10k-firmware-qca9984-ct && opkg install ath10k-firmware-qca9984

darksky · December 19, 2020, 3:01pm

I read some people suspect this is needed for stability but in my case, the R7800 is perfectly stable running the schedutil governor with the NSS patches enabled.

ACwifidude · December 19, 2020, 3:07pm

I haven’t had any stability problems either. I’ve found all the governors work fine. I’ve been rocking the performance governor recently and it is going well.

eems01 · December 19, 2020, 5:10pm

Been a Kong fan forever but with a long holiday break I figured I give your minimal build a try since we have similar setups with multiple R7800s as APs and I'll have time to troubleshoot. Fast transitioning was a reason I briefly tried other builds over the summer to use DAWN but never felt it improved transitioning. Can you please post what you settled on now for a reliable DAWN configuration with the latest builds?

Note: One issue I've always had on my network setup is that I'm not able to use FT over DS and must use FT over the Air or else I get a long pause on my devices when they switch APs. If that is required for DAWN to work properly then that maybe part of the issues I've seen in the past.

ACwifidude · December 19, 2020, 5:33pm

802.11 v and k are still not a 100% in OpenWRT - I’ve included and activated as much as is out there and my transitions are pretty smooth. My APs all are hardwired to the original router and I’ve found ft over ds offers the best setup.

This is what I’ve added beyond turning on the majority of the dawn features (I don’t think the broadcast command does anything):


uci set dawn.@network[0].broadcast_ip='192.168.1.255'; uci commit dawn

uci add_list umdns.@umdns[0].network='wan'; uci commit umdns

This is my config- I’ll defer to @PolynomialDivision to see if he has any more tips on optimizing 802.11 k,v with the recent master commits + his dawn package


root@OpenWrt:~# cat /etc/config/wireless

config wifi-device 'radio0'
        option type 'mac80211'
        option hwmode '11a'
        option path 'soc/1b500000.pci/pci0000:00/0000:00:00.0/0000:01:00.0'
        option htmode 'VHT80'
        option txpower '20'
        option country 'US'
        option legacy_rates '0'
        option beacon_int '101'
        option channel '161'

config wifi-iface 'default_radio0'
        option device 'radio0'
        option network 'lan'
        option mode 'ap'
        option ssid ''
        option encryption 'psk2+ccmp'
        option key ''
        option ieee80211r '1'
        option ft_over_ds '1'
        option ft_psk_generate_local '1'
        option wpa_disable_eapol_key_retries '1'
        option ieee80211v '1'
        option ieee80211k '1'
        option bss_transition '1'
        option wnm_sleep_mode '1'
        option time_advertisement '2'
        option time_zone 'CST6CDT,M3.2.0,M11.1.0'

config wifi-device 'radio1'
        option type 'mac80211'
        option channel '11'
        option hwmode '11g'
        option path 'soc/1b700000.pci/pci0001:00/0001:00:00.0/0001:01:00.0'
        option htmode 'HT20'
        option txpower '20'
        option country 'US'
        option legacy_rates '0'
        option beacon_int '191'

config wifi-iface 'default_radio1'
        option device 'radio1'
        option network 'lan'
        option mode 'ap'
        option ssid ''
        option encryption 'psk2+ccmp'
        option key ''

root@OpenWrt:~# cat /etc/config/dawn

config network
        option broadcast_port '1025'
        option tcp_port '1026'
        option network_option '2'
        option shared_key 'Niiiiiiiiiiiiiik'
        option iv 'Niiiiiiiiiiiiiik'
        option use_symm_enc '1'
        option collision_domain '-1'
        option bandwidth '-1'
        option broadcast_ip '192.168.1.255'

config ordering
        option sort_order 'cbfs'

config hostapd
        option hostapd_dir '/var/run/hostapd'

config times
        option update_client '10'
        option denied_req_threshold '30'
        option remove_client '15'
        option remove_probe '30'
        option remove_ap '460'
        option update_hostapd '10'
        option update_tcp_con '10'
        option update_chan_util '5'
        option update_beacon_reports '20'

config metric
        option ap_weight '0'
        option no_ht_support '0'
        option no_vht_support '0'
        option rssi '10'
        option low_rssi '-500'
        option freq '100'
        option chan_util '0'
        option max_chan_util '-500'
        option rssi_val '-60'
        option low_rssi_val '-80'
        option chan_util_val '140'
        option max_chan_util_val '170'
        option bandwidth_threshold '6'
        option use_station_count '1'
        option max_station_diff '1'
        option deny_auth_reason '1'
        option deny_assoc_reason '17'
        option use_driver_recog '1'
        option min_number_to_kick '3'
        option chan_util_avg_period '3'
        option set_hostapd_nr '1'
        option op_class '0'
        option duration '0'
        option mode '0'
        option scan_channel '0'
        option ht_support '10'
        option vht_support '100'
        option min_probe_count '2'
        option eval_probe_req '1'
        option eval_auth_req '1'
        option eval_assoc_req '1'

root@OpenWrt:~# cat /etc/config/umdns

config umdns
        option jail '1'
        list network 'lan'
        list network 'wan'

PolynomialDivision · December 19, 2020, 5:40pm

Sry, that I'm so unresponsive. I currently have so much to do.
This config

        option eval_probe_req '1'
        option eval_auth_req '1'
        option eval_assoc_req '1'

can be a bit problematic, because it does aggressive steering of clients. But if you have no issues, then it is fine.

I'm currently alone with DAWN and right now I am doing different IPv6 project. There is now even another project https://github.com/blogic/usteer I hope we can somehow combine our work.

If I finally have that IPv6 working I will go further with dawn.

PolynomialDivision · December 19, 2020, 5:42pm

The broadcast only does something, if u use it. The standard option is tcp with umdns.

ACwifidude · December 19, 2020, 5:54pm

I have all Apple clients on 5 ghz - they seem to like the settings and provide predictable roaming.

Master has been getting several BSS commits and several more are in staging. Hope you get some developer friends to synchronize on getting 802.11 k,v mainstream 100% working.

PolynomialDivision · December 19, 2020, 6:31pm

There was a talk on battle mesh: "NEWS FROM THE HOSTAPD LAND BATTLEMESH 2020" that provides information about all the changes.

Gilgamesh · December 20, 2020, 1:09am

@ACwifidude would love if you could add adblock and upnp, also why not use the Luci 2020 theme as it's much more mobile friendly? will WiFi offloading (NSS) be enabled at some point? thank you!

ACwifidude · December 20, 2020, 1:41am

Wifi offloading was working until master was updated to 5.8 ath10k-ct drivers. The patch just needs to be updated from 5.4 to 5.8. When the developers have time it’ll be back.

I don’t personally use Adblock or upnp but feel free to build off my GitHub and add it in.

Gilgamesh · December 20, 2020, 1:59am

ok thank you!

I was thinking of doing that but never done it before as I just used to flash Kongs build that already have those enabled, but if you can guide me on how to do that I would appreciate it, either here or drop me a PM. thanks.

ACwifidude · December 20, 2020, 4:09am

Easy enough. If you want to replicate it I have detailed instructions on the second post, just add the packages you desire to the diffconfig

keithspg · December 20, 2020, 11:51pm

@ACwifidude The last image I grabbed from you (20201208 OpenWrt SNAPSHOT r15149+64-28a9ac74cc) seemed to be the most stable so far. It ran over a week before rebooting. At least 10 days...
I logged in this afternoon and noticed that it rebooted ~ 3 days ago and decided to grab this 'latest' image (R7800-20201219-MasterNSS-sysupgrade.bin) but this did not load on my router. I had to do a tftp reinstall after I did a force install When I try to load this image over the one from 20201208, I get this:

Sun Dec 20 17:46:57 CST 2020 upgrade: Image metadata not present Sun Dec 20 17:46:57 CST 2020 upgrade: Use sysupgrade -F to override this check when downgrading or flashing to vendor firmware Image check failed.

The uploaded image file does not contain a supported format. Make sure that you choose the generic image format for your platform.

Did I do something wrong? I have images from your archive from 20201022 through the 20201208 and all 'just flashed' and rebooted.

ACwifidude · December 21, 2020, 12:10am

Hmm. Don’t know - I have that build loaded on three r7800s currently. Any other errors when you try to load it?

I’ll do a clean new build and post it up later tonight.

ACwifidude · December 21, 2020, 3:43am

Try the new build and see if it resolves your issue.

Kostja_V · December 21, 2020, 7:12am

May be, somebody has a build for ZYXEL Armor Z2 (NBG6817)?
I am very interested to test it.

ACwifidude · December 21, 2020, 1:12pm

I only have r7800s - the devices are similar enough and should work if you build with zyxel_nbg6817 as the device type.

Feel free to build off my base. I could give it a whirl and get you an untested version if you don’t have a build system.

keithspg · December 21, 2020, 2:23pm

That worked. I must have had a bad download or 'did it wrong' somehow. I'm now on OpenWrt SNAPSHOT r15241+72-3ab695368a.
The only issue I have had is the spontaneous reboot. It reboots pretty quickly, so it is not a huge deal, but, still... To me it feels like a stack issue or some memory issue because it 'just happens'. Does not seem to be dependent on wifi or 'heavy use'. Is there something I could be doing to diagnose the spontaneous reboot?