Qualcommax NSS Build

qosmio · June 7, 2024, 1:48pm

Yea, uncheck the devcrypto option in OpenSSL. Since you won't have any working crypto devices available. Same with cryptodev-linux.

You can also uncheck everything that isn't enforced in the NSS drv section. Let the makefile sort out the dependencies as you enable specific nss client drivers.

Great! I'll check it out. Thanks!

dentillozie · June 7, 2024, 5:08pm

Any new insights into mesh functionality by any chance?

I have had lots op issues with NSS and mesh builds. Mainly the node (AX3600) becoming unreachable and locking up after some hours. Its only reachable again after a reboot.
It works very well until it suddenly doesn't.

I have reverted back to a non NSS build and its been stable for over a week.

qosmio · June 7, 2024, 8:30pm

NSS offloaded mesh was a nightmare to get setup. I briefly switched to WDS based offloading, since it's actually working on NSS 12.1. It was less painful to get setup, but was even more unstable compared to mesh. I would get random kernel panics. This was on both 11.4 and 12.1 firmwares.

I switched back to mesh and experimented with a few options and I've finally gotten it pretty stable.

You'll need the mesh11sd package as some other mesh related settings can't be set in /etc/config/wireless alone.

I use the following on my 2 mx5300.

/etc/config/wireless

config wifi-device 'radio1'
	option type 'mac80211'
	option path 'platform/soc@0/c000000.wifi'
	option band '5g'
	option txpower '30'
	option country 'US'
	option htmode 'HE80'
	option cell_density '3'
	option channel '48'
	option noscan '1'

config wifi-iface 'mesh_radio1'
	option device 'radio1'
	option mode 'mesh'
	option encryption 'sae'
	option mesh_id 'example-mesh'
	option key '*******'
	option network 'lan'

/etc/config/mesh11sd

On MX5300-1 which is hardwired to DL-WRX36

config mesh11sd 'setup'
	option enabled '1'
	option auto_config '0'
	option mesh_gate_enable '1'
	option mesh_leechmode_enable '0'

config mesh11sd 'mesh_params'
	option mesh_fwding '1'
	option mesh_hwmp_rootmode '2'
	option mesh_connected_to_gate '1'
	option mesh_gate_announcements '1'

MX5300-2 only connecting by WiFi to MX5300-1

config mesh11sd 'setup'
	option enabled '1'
	option auto_config '0'
	option mesh_gate_enable '1'
	option mesh_leechmode_enable '1'

config mesh11sd 'mesh_params'
	option mesh_fwding '0'
	option mesh_hwmp_rootmode '0'
	option mesh_connected_to_gate '1'
	option mesh_gate_announcements '0'

Not sure how well it'd work for you as I have dedicated radios for the backhaul. I've also noticed that NSS meshing doesn't like operating on any DFS channels, or on 160Mhz.

I tried to see if I could also enable mesh on my main router but everything goes nuts as there's all sorts of bridge-loop warnings, even though I have nftables filtering out duplicate src<->dst packets.

jkool702 · June 7, 2024, 10:29pm

Update: this seems to be related to the disable_gro UCI option for ecm.

Strictly based on observing CPU usage, it seems that:

when I have disable_gro='1' set, network traffic originating from / terminating at the router (not a LAN-side device) seemingly doesnt go through the NSS but hits the CPU instead.
when I have disable_gro='0' set, all network traffic (including traffic originating from / terminating at the router) seems to go through the NSS

Im not sure if this is a bug or if this is the expected behavior when disabling generic receive offload

qosmio · June 7, 2024, 10:50pm

Technically enabling GRO conflicts with NSS' ability to properly offload, as the kernel is trying to generically offload received traffic. QSDK even specifically disables it. Especially for mac80211.

qca/feeds/nss-host/qca-nss-ppe/files/nss_perf_config.sh:			ethtool -K $dev gro off
qca/feeds/nss-host/qca-edma/files/qca-edma:	ethtool -K eth0 gro off
qca/feeds/nss-host/qca-edma/files/qca-edma:	ethtool -K eth1 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:					ethtool -K eth1 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:					ethtool -K eth0 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:					ethtool -K wlan0 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:					ethtool -K wlan1 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:					ethtool -K wlan2 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth4 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth4 gso off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth5 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth5 gso off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth4 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth4 gso off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth5 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth5 gso off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth4 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth4 gso off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth5 gro off
openwrt-patches/package/kernel/mac80211/mac80211/files/lib/boost_performance.sh:				ethtool -K eth5 gso off

I've actually noticed it improved mesh offloading on MX5300.

jkool702 · June 8, 2024, 5:45am

well then, that makes this extra weird.

I did a bit more testing and it seems that the combination of having GRO disabled and SQM enabled is causing traffic whose source or destination is the router itself to go through the CPU not the NSS. I'll reiterate that this isnt a problem for LAN-side devices, just traffic directly to/from the router.

My "test" for whether traffic is hitting the CPU or going through the NSS is to run speedtest-netperf.sh and compare the average CPU usage to the netperf overhead. If they are roughly equal (typically avg cpu is slightly lower) then it means that traffic is going through the NSS. If the average CPU utilization is much higher than the netperf overhead (meaning that running the netperf test is eating up more CPU time than what is required to run netperf) then the traffic is hitting the CPU.

`speedtest-netperf.sh` results

GRO: yes     SQM: no
 Download: 885.05 Mbps
  Latency: [in msec, 31 pings, 0.00% packet loss]
      Min:  17.500
    10pct:  17.600
   Median:  20.400
      Avg:  22.216
    90pct:  28.400
      Max:  38.000
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 28 samples]
     cpu0:  15.9 +/-  1.5  @ 2208 MHz
     cpu1:  12.2 +/-  1.2  @ 2208 MHz
     cpu2:  13.8 +/-  2.0  @ 2208 MHz
     cpu3:  19.6 +/-  2.5  @ 2208 MHz
 Overhead: [in % used of total CPU available]
  netperf:  16.5
  avg cpu:  15.4

GRO: yes    SQM: yes

 Download: 869.39 Mbps
  Latency: [in msec, 31 pings, 0.00% packet loss]
      Min:  17.600
    10pct:  17.700
   Median:  18.600
      Avg:  19.435
    90pct:  20.600
      Max:  28.700
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 28 samples]
     cpu0:  23.1 +/-  5.3  @ 2208 MHz
     cpu1:  11.5 +/-  3.6  @ 2208 MHz
     cpu2:  12.2 +/-  3.9  @ 2208 MHz
     cpu3:  35.4 +/-  0.0  @ 2208 MHz
 Overhead: [in % used of total CPU available]
  netperf:  23.5
  avg cpu:  20.6

GRO: no    SQM: no

 Download: 897.08 Mbps
  Latency: [in msec, 31 pings, 0.00% packet loss]
      Min:  17.700
    10pct:  18.400
   Median:  22.100
      Avg:  23.342
    90pct:  26.700
      Max:  44.200
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 28 samples]
     cpu0:  37.1 +/-  7.9  @ 2208 MHz
     cpu1:  22.1 +/-  2.2  @ 2208 MHz
     cpu2:  20.2 +/-  2.4  @ 2208 MHz
     cpu3:  29.8 +/-  2.4  @ 2208 MHz
 Overhead: [in % used of total CPU available]
  netperf:  28.0
  avg cpu:  27.3

GRO: no    SQM: yes

 Download: 835.79 Mbps
  Latency: [in msec, 31 pings, 0.00% packet loss]
      Min:  17.500
    10pct:  17.700
   Median:  18.900
      Avg:  20.416
    90pct:  21.600
      Max:  49.100
 CPU Load: [in % busy (avg +/- std dev) @ avg frequency, 28 samples]
     cpu0:  78.8 +/-  0.0  @ 2208 MHz
     cpu1:  39.3 +/-  6.5  @ 2208 MHz
     cpu2:  32.1 +/-  6.4  @ 2208 MHz
     cpu3:  46.1 +/-  8.8  @ 2208 MHz
 Overhead: [in % used of total CPU available]
  netperf:  33.3
  avg cpu: 49.1

In the first 3 the netperf overhead is accounting for all the CPU usage. Granted, enabling SQM increases the netperf overhead by ~5% and disabling GRO increases it by 15%, but cpu usage increases basically the same amount. Its only with GRO disabled and SQM enabled that the traffic itself is causing cpu usage to increase.

Note that i see this with the much more lightweight speedtest (from speedtest-cpp)...the first 3 cases cpu usage is basically 0, the last one i see significant cpu usage spikes. It is just harder to quantify that test.

Also note that that issue is minor enough that I could definitely live with it...Im mostly reporting it since it seems like whatever is causing this might be causing other undiscovered and/or hard-to-diagnose issues (e.g., losing WAN access after several hours when GRO is enabled).

zxlhhyccc · June 8, 2024, 7:40am

After the update, routes are disconnected from the network and the PC cannot obtain the IP address. You are advised to roll back.

glassdoor · June 8, 2024, 7:56am

rollback??!! this is from maintree. so a fix will be posted soon enough.
just revert to last working build in the mean time.

fyi your quote is misleading.

hingbong · June 8, 2024, 9:44am

Faced issues but didn't know why, and I know now. Thanks.

glassdoor · June 8, 2024, 11:01am

https://github.com/openwrt/openwrt/commit/80d1c353b79e6c216dcb2534420470e3e6ed5d60

already reverted on maintree

sppmaster · June 8, 2024, 1:10pm

sppmaster:

root@QNAP:~# openssl engine -t -c
FATAL: Startup failure (dev note: apps_startup()) for openssl
302D258A7F000000:error:12800067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:118:filename(libengines.so): Error loading shared library libengines.so: No such file or directory
302D258A7F000000:error:12800067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:152:
302D258A7F000000:error:0700006E:configuration file routines:module_load_dso:error loading dso:crypto/conf/conf_mod.c:321:module=engines, path=engines
302D258A7F000000:error:07000071:configuration file routines:module_run:unknown module name:crypto/conf/conf_mod.c:266:module=engines

Guys can you run this openssl engine -t -c and show the output.
I tried to get rid of the above error.
I've compiled absolutely from scratch using just a minimal config but same error. I didn't have the error in the past when I used all that devcrypto engine settings.
@jkool702 I remember you had issues in the past that we discussed

How did you sort them out?

@rmandrad How is it on your QNAP?

Primus · June 8, 2024, 4:50pm

Hi! I'm working on setting up dynamic VLANs via wpa2 enterprise and I'm trying to figure out if this is an NSS issue or an issue with my config (the vlans work on ethernet but not on wlan), it continuously sends EAP-RETRANSMIT so no auth works (even on wpa2-personal & wpa2-enterprise) -- nothing is ever sent to the radius server.

Sat Jun  8 17:18:10 2024 daemon.info hostapd: phy0-ap0: STA 16:f1:01:97:3a:72 IEEE 802.11: authenticated
Sat Jun  8 17:18:10 2024 daemon.notice hostapd: phy0-ap0: STA-OPMODE-N_SS-CHANGED 16:f1:01:97:3a:72 2
Sat Jun  8 17:18:10 2024 daemon.info hostapd: phy0-ap0: STA 16:f1:01:97:3a:72 IEEE 802.11: associated (aid 1)
Sat Jun  8 17:18:10 2024 daemon.notice hostapd: phy0-ap0: CTRL-EVENT-EAP-STARTED 16:f1:01:97:3a:72
Sat Jun  8 17:18:10 2024 daemon.notice hostapd: phy0-ap0: CTRL-EVENT-EAP-PROPOSED-METHOD vendor=0 method=1
Sat Jun  8 17:18:13 2024 daemon.notice hostapd: phy0-ap0: CTRL-EVENT-EAP-RETRANSMIT 16:f1:01:97:3a:72
Sat Jun  8 17:18:19 2024 daemon.notice hostapd: phy0-ap0: CTRL-EVENT-EAP-RETRANSMIT 16:f1:01:97:3a:72
Sat Jun  8 17:18:31 2024 daemon.notice hostapd: phy0-ap0: CTRL-EVENT-EAP-RETRANSMIT 16:f1:01:97:3a:72

Here are my config files:
wireless:

config wifi-device 'radio0'
	option type 'mac80211'
	option path 'platform/soc@0/c000000.wifi'
	option channel '44'
	option band '5g'
	option htmode 'HE40'
	option cell_density '0'
	option country 'GB'

config wifi-device 'radio1'
	option type 'mac80211'
	option path 'platform/soc@0/c000000.wifi+1'
	option channel '6'
	option band '2g'
	option htmode 'HE40'
	option cell_density '0'
	option country 'GB'

config wifi-iface 'wifinet0'
	option device 'radio1'
	option mode 'ap'
	option ssid ‘wpa2_psk_24’
	option encryption 'psk2'
	option hidden '1'
	option key ‘password’
	option network 'lan'
	option ieee80211w '1'

config wifi-iface 'wifinet1'
	option device 'radio0'
	option mode 'ap'
	option ssid ‘wpa_2_psk_5’
	option encryption 'psk2'
	option hidden '1'
	option key ‘password’
	option network 'lan'
	option ieee80211w '1'
	option disabled '1'

config wifi-iface 'wifinet2'
	option device 'radio0'
	option mode 'ap'
	option ssid ‘SK’
	option encryption 'wpa2'
	option hidden '1'
	option auth_server ‘asv’
	option auth_port '1812'
	option auth_secret ‘asec’
	option acct_server ‘asv’
	option acct_port '1813'
	option acct_secret ‘acsec’
	option ieee80211w '1'
	option network 'lan'
	option vlan_naming '1'
	option dynamic_vlan '1'
	option vlan_tagged_interface 'br-lan'

config wifi-iface 'wifinet3'
	option device 'radio0'
	option mode 'sta'
	option network 'wwan'
	option ssid ‘sw_bh’
	option bssid ’01:02:03:04:05:06’
	option encryption 'wpa2'
	option macaddr ’01:01:01:01:01:01’
	option eap_type 'peap'
	option subject_match '/CN=ipam’
	option auth 'EAP-MSCHAPV2'
	option identity ‘backhaul_username’
	option password ‘backhaul_password’
	option ieee80211w '1'

network:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd35:7e4e:da58::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1'
	list ports 'eth2'

config interface 'lan'
	option device 'br-lan.1'
	option proto 'static'
	option ipaddr '192.168.8.1'
	option netmask '255.255.255.0'
	option ip6assign '60'

config interface 'wwan'
	option proto 'dhcp'
	option hostname '*'

config bridge-vlan
	option device 'br-lan'
	option vlan '1'
	list ports 'eth1:t'
	list ports 'eth2:u*'

config bridge-vlan
	option device 'br-lan'
	option vlan '3'
	list ports 'eth1:t'

firewall:

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'
	option flow_offloading '1'
	option flow_offloading_hw '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'outboundvpn'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	list network 'windscribe'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wwan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
root@OpenWrt:~# cat /etc/config/firewall

config defaults
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'
	option flow_offloading '1'
	option flow_offloading_hw '1'

config zone
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	list network 'lan'

config zone
	option name 'outboundvpn'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	list network 'windscribe'

config zone
	option name 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	list network 'wwan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Support-UDP-Traceroute'
	option src 'wan'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option enabled 'false'

config include
	option path '/etc/firewall.user'

config include 'miniupnpd'
	option type 'script'
	option path '/usr/share/miniupnpd/firewall.include'
	option family 'any'
	option reload '1'

config include 'bcp38'
	option type 'script'
	option path '/usr/lib/bcp38/run.sh'

config forwarding
	option src 'lan'
	option dest 'outboundvpn'

config forwarding
	option src 'outboundvpn'
	option dest 'wan'

config forwarding
	option src 'lan'
	option dest 'wan'

Does anyone have any ideas?

jkool702 · June 8, 2024, 6:55pm

 # openssl engine -t -c

(dynamic) Dynamic engine loading support
     [ unavailable ]

In my build config I disable openssl engine support entirely, and de-select the kmod-devcrypto package. I also deslect all the NSS crypto packages.

Not including anything devcrypto related or NSS-crypto related is also the answer to

To be honest, you dont really lose anything from not having devcrypto nor NSS-crypto "acceleration". I ran some speedtests on the parts that were working and in 95% of situations the standard openssl routines were faster (often considerably faster) than their "accelerated" counterparts.

EDIT: I found my speedtest records, in case you are interested.

The only places where the accelerated versions did better was for long length (>1k) DES and AES-192-CTR. Everywhere else they were either identical (meaning they were actually being accellerated) or worse...often by orders of magnitude.

openssl speed results

# NOTE: For each test, the 1st and 2nd rows are the "accelerated" results 
#       (one is devcrypto, one is afalg. I forget which is which though...) 
#       The last row is the standard openssl non-accelerated result.

type             16 bytes     64 bytes    256 bytes    1024 bytes   8192 bytes  16384 bytes
-------------   ----------   ----------   ----------   ----------   ----------   ----------

DES-CBC            492.10k     2041.92k     7410.43k    21988.35k    49397.76k    54165.50k
DES-CBC            489.60k     2029.31k     7366.66k    21616.64k    49119.23k    54214.66k
DES-CBC          33679.54k    39508.16k    41354.24k    41874.43k    42008.58k    41959.42k

DES-EDE3-CBC       464.14k     1897.73k     6042.11k    13562.88k    21241.86k    22167.55k
DES-EDE3-CBC       466.40k     1904.00k     6220.54k    13793.28k    21323.78k    22233.09k
DES-EDE3-CBC     13386.94k    14230.14k    14473.47k    14497.79k    14524.42k    14467.07k

AES-128-CBC        466.98k     2026.37k     7559.42k    26929.15k   103849.98k   130088.96k
AES-128-CBC        462.98k     1956.16k     7472.90k    26740.74k   104062.98k   130465.79k
AES-128-CBC     207966.43k   605846.46k  1138940.67k  1497862.14k  1642643.46k  1646919.68k

AES-128-CTR        452.21k     1931.71k     6908.16k    25218.05k   101105.66k   128352.26k
AES-128-CTR        447.66k     1796.10k     7003.14k    25029.63k   100696.06k   127795.20k
AES-128-CTR     167569.97k   491499.46k   983621.89k  1378906.11k  1571274.75k  1584005.12k

AES-128-ECB     229572.98k   562328.45k  1174422.27k  1653328.90k  1867628.54k  1874149.38k
AES-128-ECB     225006.77k   561217.09k  1174845.18k  1653063.68k  1868881.92k  1882996.74k
AES-128-ECB     224236.30k   558541.31k  1170733.82k  1645973.50k  1858215.94k  1876279.30k

AES-192-CBC       3050.69k    11989.25k    45946.11k   163026.94k   522772.48k   619888.64k
AES-192-CBC       3056.88k    11988.54k    46199.04k   162593.79k   537026.56k   619266.05k
AES-192-CBC     196602.06k   534467.78k   921363.97k  1147779.07k  1233027.07k  1240399.87k

AES-192-CTR       2599.58k    10344.13k    40442.37k   148062.21k   573333.50k   692043.78k
AES-192-CTR      13589.14k    54023.75k   235224.85k   920499.20k  5745704.96k 14021427.20k
AES-192-CTR     161507.97k   462660.48k   892081.66k  1244072.96k  1396793.34k  1406468.10k

AES-192-ECB     212877.36k   523058.94k  1064294.66k  1458062.34k  1623048.19k  1638449.15k
AES-192-ECB     215302.51k   523016.77k  1064192.51k  1457836.03k  1625300.99k  1636990.98k
AES-192-ECB     215601.38k   523195.07k  1065053.70k  1450206.21k  1616257.02k  1624555.52k

AES-256-CBC        443.97k     1852.61k     7077.38k    24635.39k    88129.54k   108675.07k
AES-256-CBC        452.90k     1853.31k     7115.01k    24779.78k    87916.54k   107593.73k
AES-256-CBC     189977.02k   488694.46k   795642.11k   959036.42k  1018257.41k  1017233.41k

AES-256-CTR        424.34k     1731.52k     6627.33k    23412.74k    85909.50k   106004.48k
AES-256-CTR        415.97k     1727.68k     6653.67k    23399.95k    85893.12k   106037.25k
AES-256-CTR     155736.93k   436405.95k   817223.17k  1127587.84k  1256398.85k  1263337.47k

AES-256-ECB     206368.22k   490831.94k   957814.27k  1301154.82k  1435451.39k  1447297.02k
AES-256-ECB     206523.44k   491080.06k   959379.46k  1303701.50k  1436786.69k  1437794.30k
AES-256-ECB     203096.75k   489807.30k   959630.08k  1304099.84k  1438883.84k  1432666.11k

sppmaster · June 8, 2024, 7:19pm

I have all of them disabled.

Do you mean the last one on the screenshot below? I cannot find kmod-devcrypto in menuconfig. I disabled it weeks ago.

I know this but I don't understand why openssl throws those errors. Even compiling from scratch with minimal config and then flashing without keeping any settings didn't help.
I thought I finally setup my config but now this error confused me because it seems that encryption somehow still works. I have curl that uses https and I dont see any other broken things. I don't have mbedtls nor wolfssl on my system. On the same build system I compile for other routers (R7800 being one of them) using openssl on all of them and it works.
Any suggestions how to test if openssl actually works?

qosmio · June 8, 2024, 8:01pm

Never enable these when using NSS offloading. It defeats the whole purpose of the NSS cores.

JuliusBairaktaris · June 8, 2024, 8:51pm

@qosmio as your repo is currently affected by this it would be great if you could rebase it. Thanks in advance.

qosmio · June 8, 2024, 9:16pm

The thing about GRO and running the tests locally on the router is that, while yes you're offloading some tasks in the kernel, you end up risking packets getting mishandled in the long run with NSS offloading too. There has been some changes in ECM to better handle skb that use GRO vs. standard list, but it's still not ideal way.

As for running any speed tests locally on the router, remember you are creating those packets to send. The job of the router isn't really to "create", buffer AND move them back and forth from the endpoint, it's strictly supposed to be the later. Moving back and forth. So there is going to be some expectation of CPU spikes. GRO would alleviate some of those tasks locally, but again, not good for the long run.

Now, if you ran that speed test on a local client and it caused CPU spike on the router... then that's a whole different story, and something seriously broken NSS wise.

qosmio · June 8, 2024, 9:33pm

@robimarko reverted commit https://github.com/openwrt/openwrt/commit/80d1c353b79e6c216dcb2534420470e3e6ed5d60

issue was using (..) (sub shell) vs. {..} (simple grouping) when trying to source /lib/config/uci.sh

( [ -f /lib/config/uci.sh ] && . /lib/config/uci.sh ) || true

Sub shells don't preserve functions/variables so none of the UCI functions were defined.

Curly braces though group multiple commands into one big logic, within the same shell, which was the intention.

qosmio · June 8, 2024, 9:54pm

Not sure why you're using the "engine" option here. You would only see that if enabling the "ENGINE" option during build and using the kmod-devcrypto. Neither of which you should be enabled on NSS builds.

Are you trying to run benchmark tests?

If curl is working with https, then everything should as well.

You can check other packages using that library and verify individually too.

opkg whatdepends libopenssl3

sppmaster · June 8, 2024, 9:56pm

I don't want to spam this thread with more messages about openssl.
I think I am on the right track, almost.
Last question. What are your settings here?
I had engine support disabled but faced that startup failure.

I tried openssl speed but it gave the same error.
openssl version gives the same error that I'm concerned for.
Actually I compiled the newest openssl (v3.0.14 that is already merged) version into my build and decided to check if it is OK.
But it turned out that previous one v3.0.13 gave the same startup failure.
It works OK only if I enable devcrypto.

root@QNAP:~# opkg whatdepends libopenssl3
Root set:
  libopenssl3
What depends on root set
        openssl-util 3.0.14-r1  depends on libopenssl3
        libustream-openssl20201210 2024.04.19~524a76e5-r1       depends on libopenssl3
        libopenssl-legacy 3.0.14-r1     depends on libopenssl3
        luci-ssl-openssl 24.158.03388~a6f8361   depends on libustream-openssl20201210
        wpad-openssl 2024.03.09~695277a5-r1     depends on libopenssl3
        libopenssl-conf 3.0.14-r1       depends on libopenssl3
        libcurl4 8.8.0-r1       depends on libopenssl3
        wget-ssl 1.24.5-r1      depends on libopenssl3