I managed to narrow down the issue to the feeds. I used to pull in all the feeds and hope the build process will pick out what it needs. That lead to the error above which is an expanded version of the one below:
make[3] -C feeds/packages/libs/nss compile
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
make: *** [world] Error 1
$
It seems that this was caused by the libreswan package.
The bad news is that if I pulled in this package from the feed, the only way I could get rid of the issue was by starting from scratch (probably for the better)
The next good news is that strongswan (default setup) compiled without issues.
Unfortunately once I added quagga for BGP routing into the mix, I ended up again with an error:
make[3] -C package/qca/qca-nss-ecm compile
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
make: *** [world] Error 1
$
I'll play with this a bit more in the coming days, but if anyone has any suggestions, please let me know.
As for questions:
Has anyone used OpenWRT as an ipsec site to site solution? If yes, can you point me to any documentation? I'm not looking for some high performance with ipsec, just trying to avoid an extra device on my network.
Additionally, does anyone have good steps to build an image with extra packages (like the previously mentioned ipsec, luci etc). Somehow I can't seem to figure this part out
Starting from a clone or a merge of my git repo or Ansuel’s:
make dirclean
./scripts/feeds update -a
./scripts/feeds install -a
./scripts/diffconfig.sh > diffconfig
Copy the below in to your new diffconfig file (the below compiles). The below gets you luci, working usb, NSS qdisc/wifi/routing offload, and nice statistics graphs. The # are intentional (ex: # in front of wolfssl makes sure that only OpenSSL is loaded). You can add custom programs here.
# Use "make defconfig" to expand this to a full .config
CONFIG_TARGET_ipq806x=y
CONFIG_TARGET_ipq806x_generic=y
CONFIG_TARGET_ipq806x_generic_DEVICE_netgear_r7800=y
# exfat is patented
CONFIG_BUILD_PATENTED=y
# NSS Drivers
CONFIG_PACKAGE_kmod-qca-nss-drv=y
CONFIG_PACKAGE_kmod-qca-nss-drv-qdisc=y
CONFIG_PACKAGE_kmod-qca-nss-ecm-standard=y
CONFIG_PACKAGE_kmod-qca-nss-gmac=y
CONFIG_PACKAGE_kmod-nss-ifb=y
CONFIG_PACKAGE_iptables-mod-physdev=y
CONFIG_PACKAGE_kmod-ipt-physdev=y
# Longer waiting for failsafe button push
CONFIG_IMAGEOPT=y
CONFIG_PREINITOPT=y
CONFIG_TARGET_PREINIT_TIMEOUT=5
# Busybox tweaks
CONFIG_BUSYBOX_CUSTOM=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVEHISTORY=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVE_ON_EXIT=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_FLAGS=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_REGEXP=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_WINCH=y
# Add-on programs
CONFIG_PACKAGE_irqbalance=y
CONFIG_DROPBEAR_ECC=y
# USB device mount & file systems support
CONFIG_PACKAGE_block-mount=y
CONFIG_PACKAGE_kmod-usb-storage=y
CONFIG_PACKAGE_kmod-fs-cifs=y
CONFIG_PACKAGE_kmod-fs-exfat=y
CONFIG_PACKAGE_libblkid=y
CONFIG_PACKAGE_kmod-fs-ext4=y
CONFIG_PACKAGE_kmod-fs-hfsplus=y
CONFIG_PACKAGE_kmod-fs-msdos=y
CONFIG_PACKAGE_kmod-fs-vfat=y
CONFIG_PACKAGE_ntfs-3g=y
CONFIG_PACKAGE_kmod-nls-cp1250=y
CONFIG_PACKAGE_kmod-nls-cp437=y
CONFIG_PACKAGE_kmod-nls-cp850=y
CONFIG_PACKAGE_kmod-nls-iso8859-1=y
CONFIG_PACKAGE_kmod-nls-iso8859-15=y
CONFIG_PACKAGE_kmod-nls-utf8=y
# IPv6 support
CONFIG_PACKAGE_6in4=y
CONFIG_PACKAGE_6to4=y
CONFIG_PACKAGE_6rd=y
# IPv6 NAT support (ip6tables NAT extensions, ipt-nat6 and nf-nat6 kmods)
CONFIG_PACKAGE_ip6tables-mod-nat=y
# WLAN/WPS support
CONFIG_PACKAGE_hostapd-utils=y
CONFIG_WPA_MSG_MIN_PRIORITY=4
CONFIG_PACKAGE_wpad-openssl=y
# CONFIG_PACKAGE_wpad-basic-wolfssl is not set
# CONFIG_PACKAGE_libustream-wolfssl is not set
# SSL certificates
CONFIG_PACKAGE_ca-certificates=y
# Luci (SSL from OpenSSL)
CONFIG_PACKAGE_luci-ssl-openssl=y
CONFIG_PACKAGE_luci-app-commands=y
CONFIG_PACKAGE_luci-app-sqm=y
CONFIG_PACKAGE_luci-app-dawn=y
# Luci statistics
CONFIG_PACKAGE_luci-app-statistics=y
CONFIG_PACKAGE_collectd-mod-conntrack=y
CONFIG_PACKAGE_collectd-mod-cpufreq=y
CONFIG_PACKAGE_collectd-mod-entropy=y
CONFIG_PACKAGE_collectd-mod-ping=y
CONFIG_PACKAGE_collectd-mod-sqm=y
CONFIG_PACKAGE_collectd-mod-thermal=y
CONFIG_PACKAGE_collectd-mod-uptime=y
# nlbwmon app
CONFIG_PACKAGE_luci-app-nlbwmon=y
Lastly (feel free to use make with more processors, ex for a 4 processor system: make -j5)
@Ansuel: I tested your wifi offload patch and it seems solid. I only tested it in my office with a stand alone R7800 and a file server. Everything works alright and speed test was better than before, in line with @ACwifidude results.
I would like to test it in my house, where I have a R7800 as main AP, a C2600 as wired AP, and two C7 as WDS repeaters. I also use the 80211r/k/v protocols and I want to test if everything works.
As for the C2600, is there someone here that successfully compiled the NSS version? If so, would it be possible to share the dts with @Ansuel so that he could upload it to his repo?
I can try to add the needed node for c2600. Also with wifi offload testing it would be good to have a top command while speedtesting. (to verify the cpu load during a high traffic)
@quarky in your opinion can the tasklet implementation cause some packet drop? Also in theory the tasklet implementation should give better perf... Was thinking of implementing it only for normal wifi operation and keep the fast_rx with the current implementation of directly passing the packet to nss. (my stupid theory is if something must be handled fast, it can't be handled using a tasklet and a queue process)
Also in an old qsdk 8.0 patch i notice that if nss offload is enabled, I notice they align the packet. This was absent from your old patch. Can this cause problems and cause pk drop?
As the NSS driver to NSS firmware are done via DMA and interrupts, I thot it may cause a bottleneck. From what I remember, the tasklet queue grew to a backlog of more than 100 SKB packets during my iperf3 tests, so I thot using the tasklet would help in making the rx side better.
Packets alignment should have been done when the SKBs are created by ath10k (for the rx path), at least that's what I remember, and by the NSS drivers when it received network packets from the firmware. I think the Linux kernel also creates page aligned SKBs as well as it will make DMA more efficient.
How did you check and confirm that packets are dropped? Which path sees dropped packets? rx or tx or both?
@ACwifidude, thanks for a detailed write up about your build process! That is exactly what I was doing.
I changed the feed part to get only individual packages from the feeds once I found out that libreswan (for ipsec) and quagga (for bgp routing on a top of it) break the build.
To be more specific on the question about the image build process:
How do you get the R7800-20201012-MasterNSS-sysupgrade.bin file ?
I'm just wondering how do I get all the ipks into one file for upgrade, so I don't need to pull stuff from the repo?
@ACwifidude and @Ansuel, I measured similar numbers on my laptop with an Intel 9260 card.
The actual file transfer speed over wifi seems a bit faster as well
Now, what is more impressive for me, is the fact that with the NSS build I get around 940-960Mb/s up/down (pretty much line speed) on my wired desktop for a 1Gb/s fiber connection with 2-3% CPU utilization.
I was unable to go over 850-880Mb/s on the standard build, even with all the tweaks I could think of.
That said the last build from @ACwifidude is pretty stable, it was up for 2+days. Just noticed today that uptime got back down to a few hrs, but I don't remember doing any reboot.
Whatever you have in your OpenWRT folder + .config file will be loaded in to your bin file after the make command. I have a 3 processor x86 Linux computer so once everything is set - this creates the bin folder and all the files in ~30minutes (never timed it):
make -j4
Definitely keep this thread posted on any patch issues or particular package issues so that the developers can troubleshoot. They have done great work to add features and keep abreast of changes to master.
This is where the .bin file is when make is done:
cd bin/targets/ipq806x/generic
I test that my firmware boots - once everything appears normal I rename it from the default name and push it to GitHub. To rename the file you can manually use a command like mv *squashfs-sysupgrade.bin R7800-20201012-MasterNSS-sysupgrade.bin or you can use a nice build script like hynman that builds the filename:
@rog@ACwifidude i pushed some new patch... do you want to try them?
Anyway https://www.right.com.cn/forum/thread-258118-1-1.html this is a forum where I found some time ago a newer nss firmware... for some reason they also have qsdk12 source (no nss firmware)
Would be very good if someone that know Chinese could ask if they can give us some firmware
very nice wifi performance (i mean, low cpu) with your files @Ansuel !
today i'll build with the new patches, what did you change? what have we to try?
is the firmware you found newer that the one we are using today?
it would be nice to have a "version number" of your development, just to be sure what we are working on
with my 2x2 client i could not see any relevant impact on performance or cpu usage
with my new r14715+56 (lol for the +56) or the "old" driver, over 500Mbps and around 30% cpu usage by softirqd
