Nss-utils + i386 = Illegal instruction

I believe it is related to compilation options of nss-utils package (and those it depends on).

Steps to reproduce:

Board: Alix Board
CPU: Geode(TM) Integrated Processor by AMD PCS (family: 0x5, model: 0xa, stepping: 0x2)
OpenWRT: 22.03.05
Package: nss-utils

  1. mkdir ${HOME}/temp
  2. certutil -N -d sql:${HOME}/temp
  3. certutil -S -x -n "Router CA" -s "O=Router,CN=Router CA" -k rsa -g 4096 -v 12 -d sql:${HOME}/temp -t "CT,," -2

During CA generation it will generate "Illegal instruction" with following lines in dmesg:

traps: certutil[8135] trap invalid opcode ip:b7b2c5a4 sp:bf8e49d0 error:0 in libfreebl3.so[b7acb000+62000]

P.S. I was unable to reproduce the bug on more modern APU Board.

You didn't specify what subtarget image You used. If You are using generic image then it's self-inflicted, if legacy or geode is used, then it's a bug in the package and should be reported in https://github.com/openwrt/packages/issues.

This can't be compared if You are using different image for both boards. And if You tried to use same image, very likely the newer CPU on APU board simply supports that opcode.

If there's a "leak" from host environment during package build, testing on different arch like arm or mips should reveal if that's the case.

I used geode target as it was specifically made for this board. Utility itself starts well, it generates exception when it starts generating certificate after collecting entropy.

I can't report the bug to github because I write from a country which was sanctioned by the US and I'm too lazy to use VPN, because openwrt's VPN solutions are kinda buggy (like this) :laughing:

cc: @lucize ^^^

He's maintainer of that package, I taged him here. Unless he doesn't reply in this topic, send him a mail.

BTW, You told that it works on APU, so can't You set VPN there?

@tmn505 that was just an excuse. I'm having hard time setting up reliable VPN solution for OpenWRT. Actually I was trying to set up CA for libreswan. But since it depends on nss... well I failed a bit again (like with WireGuard). I could have generated those on another machine but libreswan depends on libnss (it stores certificates in database)... so I thought it may generate similar problems during regular libreswan operation :slight_smile:

Thanks for tagging the guy.

sorry, never actually used the nss utils and certificates, only the psk options, so don't know if I can be of help, it was a real pain to port the nss anyway but I really wanted libreswan

Well, I quick-checked certool in Debian on i386 (on same Alix board) and it worked fine. Not the same version I must admit, but it works. So, clearly something is wrong with compilation environment. Or a specific bug which was already addressed in Debian.

The freebl3.so library was likely built with AES instructions which are unsupported by the Geode CPU.

Just veried with the x86/geode SDK:

i486-openwrt-linux-musl-gcc -o build_dir/Linux_SINGLE_SHLIB/aes-x86.o -c -std=c99 ... -mpclmul -maes aes-x86.c

So it is indeed built with -maes. Couldn't figure out to disable this and could not find debian specific patches either.


That could explain it. Shouldn't be hard to fix it too.

@lucize any chance to fix the bug? My concern is that it can not only prevent certutil operation but normal libreswan+certs operation too.

@jow seems to be right on this, the only i386 device that I have is a via cpu where I also use the ipsec vpn based on it's AES cpu acceleration and seems that it's working fine

I guess I could try something

root@remser2net:~# certutil -N -d sql:${HOME}/temp
Enter a password which will be used to encrypt your keys.
The password should be at least 8 characters long,
and should contain at least one non-alphabetic character.

Enter new password:
Re-enter password:
root@remser2net:~# certutil -S -x -n "Router CA" -s "O=Router,CN=Router CA" -k rsa -g 4096 -v 12 -d sql:${HOME}/temp -t "CT,," -2
Enter Password or Pin for "NSS Certificate DB":

A random seed must be generated that will be used in the
creation of your key.  One of the easiest ways to create a
random seed is to use the timing of keystrokes on a keyboard.

To begin, type keys on the keyboard until this progress meter

Continue typing until the progress meter is full:


Finished.  Press enter to continue:

Generating key.  This may take a few moments...

Is this a CA certificate [y/N]?
Enter the path length constraint, enter to skip [<0 for unlimited path]: >
Is this a critical extension [y/N]?

root@remser2net:~# cat /proc/cpuinfo | more
processor       : 0
vendor_id       : CentaurHauls
cpu family      : 6
model           : 13
model name      : VIA Eden Processor 1000MHz

Well, it doesn't work on AMD Geode, though it comes with AES128 acceleration :frowning:

VIA has specific CPU instruction set in their processors called Padlock, which supports AES, so no wonder it works. AMD Geode has separate AES accelerator that's why it is not working.
According to this Debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=875694, AES detection looks to be fixed upstream long ago, so something in OpenWrt build is not properly configured.

1 Like

Maybe it was at some point but the nss-3.76 release does not appear to have any sophisticated detection logic. As soon as the target uname is i386 or x86_64, the gcm-x86.c and aes-x86.c objects are unconditionally added and built with -mpclmul and -maes which also implies usage of SSE2 instructions.

From a cursory look, I neither could find a more fine grained detection logic, nor any option to turn off this behavior...

Maybe passing a CPU_ARCH of i486 to bypass feature detection would be a way, but who knows what else gets disabled then...

Indeed, it hadn't changed much since then. I still wonder how does that work in Debian case, since I don't see any logic for that in the package recipe tarball.

That would be good option, but we need to patch that Makefile for our explicit case (geode) and mark package as nonshared, since legacy target is built with same toolchain. And/Or maybe extending coreconf/Linux.mk with our case. There are also files like build.sh, nss.gyp and exports.gyp. I'm not familiar with that build system, so I leave it to @lucize.

Why i486 if it supports cmov and mmx? It's just a regular pentium mmx clone.

I just suggested a random value which does not match the Makefile conditional.

Am I right, that this condition should be false if CPU_ARCH is i586? Because uname -m says i586. I am a bit confused, because some packages for geode come from i386_pentium_mmx and other from x86_geode platform. Maybe that package (see very bottom) should be compiled for x86_geode (NSS and linked packages)?

opkg --download-only install libreswan
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/libreswan_4.11-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-iptunnel_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-iptunnel6_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ip6-tunnel_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-crypto-authenc_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-lib-zlib-inflate_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-lib-zlib-deflate_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-crypto-deflate_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-crypto-des_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-crypto-echainiv_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-crypto-md5_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-crypto-sha1_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ipsec_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ipsec6_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ip6-vti_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/zlib_1.2.13-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/libelf1_0.189-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/libbpf1_1.2.2-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/ip-full_6.3.0-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ipt-ipsec_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/iptables-mod-ipsec_1.8.8-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-iptunnel4_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ipsec4_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-ip-vti_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/targets/x86/geode/packages/kmod-xfrm-interface_5.15.127-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/libevent2-7_2.1.12-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/libevent2-core7_2.1.12-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/base/libevent2-pthreads7_2.1.12-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/libldns_1.8.3-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/libunbound_1.17.1-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/libsqlite3-0_3410200-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/nspr_4.35-2_i386_pentium-mmx.ipk
-------> Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/libnss_3.89.1-1_i386_pentium-mmx.ipk
Downloading https://downloads.openwrt.org/releases/23.05.0-rc3/packages/i386_pentium-mmx/packages/nss-utils_3.89.1-1_i386_pentium-mmx.ipk

the thing is that these are compile options, not runtime options, so maybe I'll have to see what debian is doing at compile time

1 Like