That would be great if you could 
Maybe they could suggest at least where to look?
Maybe it is something trivial to fix?
I am trying to be realistic,
I can't believe we would be the only one having these issues if it doesn't work at all with multi radios
OEM (from my logs):
dev: size erasesize name
mtd0: 00100000 00020000 "0:SBL1"
mtd1: 00100000 00020000 "0:MIBIB"
mtd2: 00300000 00020000 "0:QSEE"
mtd3: 00080000 00020000 "0:DEVCFG"
mtd4: 00080000 00020000 "0:RPM"
mtd5: 00080000 00020000 "0:CDT"
mtd6: 00080000 00020000 "0:APPSBLENV"
mtd7: 00100000 00020000 "0:APPSBL"
mtd8: 00080000 00020000 "0:ART"
mtd9: 00080000 00020000 "bdata"
mtd10: 00080000 00020000 "crash"
mtd11: 00080000 00020000 "crash_syslog"
mtd12: 023c0000 00020000 "rootfs"
mtd13: 023c0000 00020000 "rootfs_1"
mtd14: 01ec0000 00020000 "overlay"
mtd15: 00080000 00020000 "rsvd0"
mtd16: 0041e000 0001f000 "kernel"
mtd17: 015cc000 0001f000 "ubi_rootfs"
mtd18: 01876000 0001f000 "data"
OpenWrt:
dev: size erasesize name
mtd0: 00100000 00020000 "0:sbl1"
mtd1: 00100000 00020000 "0:mibib"
mtd2: 00300000 00020000 "0:qsee"
mtd3: 00080000 00020000 "0:devcfg"
mtd4: 00080000 00020000 "0:rpm"
mtd5: 00080000 00020000 "0:cdt"
mtd6: 00080000 00020000 "0:appsblenv"
mtd7: 00100000 00020000 "0:appsbl"
mtd8: 00080000 00020000 "0:art"
mtd9: 00080000 00020000 "bdata"
mtd10: 00080000 00020000 "crash"
mtd11: 00080000 00020000 "crash_syslog"
mtd12: 023c0000 00020000 "rootfs"
mtd13: 023c0000 00020000 "rootfs_1"
mtd14: 01ec0000 00020000 "overlay"
mtd15: 00080000 00020000 "rsvd0"
@bitthief Any chance of sharing the NSS stuff?
I finally have some time this weekend and would really love to sort out the clocks as well so that I can send the whole series upstream next week.
Finally figured out what was missing for a working bridge-mgr
CONFIG_REGULATOR_FIXED_VOLTAGE=y
So as well as the following patch to add back the nss nodes, then either removing the crypto or adding a patch to accommodate them, it just works.
https://github.com/robimarko/openwrt/commit/5e0cee43e289c1a47dd80532e3370228e14b296b
I see in Bittheif's last commit this isn't in config-5.15, so do the other NSS features work without this and only bridge-mgr needs it?
Well, fixed regulator is needed for anything NSS-DRV related as it does voltage scaling internally, but its implemented using a fixed regulator on almost all boards.
You forgot the \n in the end of each echoed string, maybe that makes some difference?
And kmod-wireguard is duplicated, although that's not the problem.
All the lines you added should have the new-line char, to look like the already existing lines:
echo "CONFIG_PACKAGE_luci-app-wireguard=y\n" >> .config
edit:
Remmembered something: did you confirm in config.buildinfo if the wireguard packages are effectively selected?
I am looking at utilizing the Cortex-A53 crypto extensions and thus the NEON SIMD optimized ChaCha20 and Poly1305 in the kernel.
Wireguard should be the easiest way of benchmarking as it heavily utilizes those, but I need a quick and easy setup with it.
Any ideas?
Actually, the NEON version should be automatically pulled in when Wireguard is selected as its the optimised version.
I've been running WireGuard on my AX6 for a while, and it does use the NEON optimized versions already:
root@OpenWrt:~# lsmod | grep neon
chacha_neon 20480 1 libchacha20poly1305
libchacha 16384 1 chacha_neon
poly1305_neon 20480 1 libchacha20poly1305
Yeah, I have updated my reply after I saw that OpenWrt already packages the NEON version for ARM64.
Thanks for checking.
I am doing OpenSSL benchmarks to see what QCE options work best for it as QCE engine seems slower then the CPU most of the time and IPQ40xx for example only enables it for skciphers.
Hey, yes, I'm merging your latest changes now and I'll push the repos in a bit.
That's great to hear, let's get it done!
how is possible that the qce is slower?
There are more changes needed, on top of that. I'll push everything in a bit.
If you have time, I'm pretty sure I found the root cause for the interrupt issues during nf_conntrack notifier registrations very early in the boot (triggered by ECM).
The problems appear to be triggered with the following default sysctls:
11-br-netfilter.conf
# Do not edit, changes to this file will be lost on upgrades
# /etc/sysctl.conf can be used to customize sysctl settings
# disable bridge firewalling by default
net.bridge.bridge-nf-call-arptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-iptables=0
Even if you enable or disable bridging in ECM, literally anything that touches the nf_ct_bridge_register/unregister functions via those sysctls during early boot, will cause the issues I was seeing (kernel oops in interrupt handler). I know you did those patches, and I saw you added BLOCKING notifiers and ATOMIC notifiers etc. My guess is that something needs to be reworked there, I've added mutexes as a precaution in ECM, but it's not enough. It'd be great if you had a look and we also 'fix' this in the clean way.
And yes, that voltage regulator was missing in my config also, I'll add it:
That's easy to setup, I use 3 WireGuard tunnels all the time, from Njalla and Mullvad. I can give you a key/pair for Mullvad and the relevant config bits if you want to test easily. Otherwise, I can spin up a wireguard server on a test VPS etc. and give you keys to benchmark in like 20m.
It pulls in those modules from the kernel, but I'm not sure if they're SIMD optimized, my dream was to offload them to NSS also, in a bright future when we finished everything else. But to be honest, not sure it's worth the effort, at least for now, I can reach 400-450mbps (maximum that Mullvad offers) with maybe 1-2 cores pinned to around 60-70-% at peak.
I remember a guy who was fixing the QCE on IPQ40xx, he did a lot of testing and upstreaming.
Some details here:
You can even see it in the kernel Kconfig for the enabled algs:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/crypto/Kconfig?h=v5.15.28#n670
IPQ4019 and IPQ8074 use the same QCE IP revision so they share the same restrictions.
Just a quick test:
QCE via devcrypto:
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-256-cbc 15491.20k 82219.73k 173709.96k 752082.49k 8762709.33k 8279244.80k
Software implementation:
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
aes-256-cbc 10490.93k 49450.24k 552000.00k 985088.00k 26389708.80k 16639590.40k
Thanks for the offer, but since the optimized modules are already used there is no point.
They are ARM64 optimized modules that use NEON instructions so they are SIMD, since they are already hand-tuned NEON ASM there is not much you can do there.
Ping me here once you push the NSS stuff.
Code's up, I've synced all my changes so far.
I'll be around, I'm looking at some other parts that I wanna clean-up/fix, so let me know if I can help.
I will take a better look tomorrow, it's kind of hard to see what's required and what's not.
These are the main requirements, clocks-wise:
FWIW, just building @bitthief's latest works fine on my AX6. Performance seems similar, but my entire uplink goes through Wireguard for reasons, so the hardware accelerated path is not taken often.
I can't take much credit for patches 600-604, the original versions are in IPQ807x-5.10-backports and I've just updated those for k5.15. 605 is mine though.
I've built your repo and included ecm, but my config is just an AP so it didn't get loaded. I added ecm to /etc/modules.d and it loaded on next boot without any errors, but I suspect that's down to no routing happening? I'll reconfigure it as a router and see if I can get it misbehaving...