Getrandom error 0x2015016) sshd version OpenSSH_8.0, OpenSSL 1.1.1b

mvebu / wrt3200acm with sshd version OpenSSH_8.0, OpenSSL 1.1.1b

self-build image from git

since upgrading to the latest ssh 8.0 I am unable to login with openssh (dropbear works)

I get the following on sshd

debug3: ssh_sandbox_parent_finish: finished
User child is on pid 24105
Couldn't obtain random bytes (error 0x2015016)

strace shows

26107 getrandom("\x99\x34\x18\xfe\xd8\x3e\x4d\xb8\x9e\x5b\xe8\x21\x02\x42\x9d\xff\x20\xa5\x5f\xfe\x34\xff\x04\xf6\x91\x41\x32\xec\xc3\xf4\x2c\x73", 32, 0) = 32
26107 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x68, 0x1c), 0xbea99344) = -1 EINVAL (Invalid argument)
26107 clock_gettime(CLOCK_BOOTTIME, {tv_sec=8051, tv_nsec=574440396}) = 0
26107 clock_gettime(CLOCK_REALTIME, {tv_sec=1556603134, tv_nsec=832788800}) = 0
26107 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x68, 0x1c), 0xbea992e4) = -1 EINVAL (Invalid argument)
26107 write(2, "Couldn't obtain random bytes (er"..., 48) = 48
26107 write(2, "debug1: do_cleanup\r\n", 20) = 20
26107 exit_group(255)                   = ?
25981 <... poll resumed> )              = 1 ([{fd=7, revents=POLLIN|POLLHUP}])
25981 write(2, "debug3: mm_request_receive enter"..., 37) = 37
25981 read(7, "", 4)                    = 0

I have urandom & random installed with rng running but it seems the getrandom function is not working - I haven't checked the code but it was working on the previous version of openssh ... any clues on how to solve this issue ?

Thank you

getrandom 512 > random.txt

is your error from bootup? or manual instantiation?

the getrandom command works fine ... entropy is fine also ... the issue is with ssh

the sshd process drops everytime a client attempts to connect - the strace output is from running the process with debug enabled.

as noted above sshd fails with the following message - I need to restart manually sshd

Couldn't obtain random bytes (error 0x2015016)

on the sshd process the getrandom function fails with

26107 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x68, 0x1c), 0xbea992e4) = -1 EINVAL (Invalid argument)
26107 write(2, "Couldn't obtain random bytes (er"..., 48) = 48
1 Like
> cat .config | grep -v '^#' | grep -i opens

to confirm you had and image made with the buildroot > then later you upgraded the package on the router?

Hi,
so I build the image using buildroot every week or so... I don't upgrade any packages on the router. This issue started to happen when the new openssh 8 was released.

CONFIG_PACKAGE_python-openssl=y
CONFIG_PACKAGE_python3-openssl=y
CONFIG_PACKAGE_python3-pyopenssl=y
CONFIG_PACKAGE_libopenssl=y
CONFIG_OPENSSL_OPTIMIZE_SPEED=y
CONFIG_OPENSSL_WITH_ASM=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_ERROR_MESSAGES=y
CONFIG_OPENSSL_WITH_TLS13=y
CONFIG_OPENSSL_WITH_DTLS=y
CONFIG_OPENSSL_WITH_NPN=y
CONFIG_OPENSSL_WITH_SRP=y
CONFIG_OPENSSL_WITH_CMS=y
CONFIG_OPENSSL_WITH_EC=y
CONFIG_OPENSSL_WITH_EC2M=y
CONFIG_OPENSSL_WITH_CHACHA_POLY1305=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_WITH_COMPRESSION=y
CONFIG_OPENSSL_WITH_RFC3779=y
CONFIG_OPENSSL_ENGINE=y
CONFIG_OPENSSL_ENGINE_BUILTIN=y
CONFIG_OPENSSL_ENGINE_BUILTIN_AFALG=y
CONFIG_OPENSSL_ENGINE_BUILTIN_DEVCRYPTO=y
CONFIG_PACKAGE_libopenssl-conf=y
CONFIG_PACKAGE_libustream-openssl=y
CONFIG_PACKAGE_luci-lib-nixio_openssl=y
CONFIG_PACKAGE_openssh-client=y
CONFIG_PACKAGE_openssh-client-utils=y
CONFIG_PACKAGE_openssh-keygen=y
CONFIG_PACKAGE_openssh-moduli=y
CONFIG_PACKAGE_openssh-server=y
CONFIG_PACKAGE_openvpn-openssl=y
CONFIG_OPENVPN_openssl_ENABLE_LZO=y
CONFIG_OPENVPN_openssl_ENABLE_LZ4=y
CONFIG_OPENVPN_openssl_ENABLE_SERVER=y
CONFIG_OPENVPN_openssl_ENABLE_FRAGMENT=y
CONFIG_OPENVPN_openssl_ENABLE_MULTIHOME=y
CONFIG_OPENVPN_openssl_ENABLE_PORT_SHARE=y
CONFIG_OPENVPN_openssl_ENABLE_DEF_AUTH=y
CONFIG_OPENVPN_openssl_ENABLE_PF=y
CONFIG_OPENVPN_openssl_ENABLE_IPROUTE2=y
CONFIG_OPENVPN_openssl_ENABLE_SMALL=y
CONFIG_PACKAGE_hostapd-openssl=y
CONFIG_PACKAGE_openssl-util=y

I've asked on github - https://github.com/openwrt/packages/pull/8272
got the reply that there is not enough entropy ... which is weird as I am using rngd

the fix suggested was to unset getrandom from openssh

1 Like

hmmmmm..... see

cat build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/openssl-1.1.1b/CHANGES | grep -C10 RAND_seed

some sort of format panic on the getrandom call....

see here for DEFINES
build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/openssl-1.1.1b/crypto/rand/rand_unix.c

OPENSSL_API_COMPAT < 0x10100000L etc.

grain of salt though i'm a novice... fwiw rndg didn't do too much on my system. maybe try without it?

it is failing on entropy.c of the openssh build ...

void
rexec_send_rng_seed(struct sshbuf *m)
{
        u_char buf[RANDOM_SEED_SIZE];
        size_t len = sizeof(buf);
        int r;

        if (RAND_bytes(buf, sizeof(buf)) <= 0) {
                error("Couldn't obtain random bytes (error %ld)",
                    ERR_get_error());
                len = 0;
        }
        if ((r = sshbuf_put_string(m, buf, len)) != 0)
                fatal("%s: buffer error: %s", __func__, ssh_err(r));
        explicit_bzero(buf, sizeof(buf));
}
1 Like

I suspect somehow the code is not detecting the SSL_ENGINE and tries to get directly from the CPU

Check right before and after you try to log in:

cat /proc/sys/kernel/random/entropy_avail

3006 before; 2914 after

1 Like

one other interesting thing is that with openssh

getrandom("\x77\x7f\xb8\x63\x0c\x95\x5d\x39\x6e\xa9\x15\xfa\x14\xda\x70\xfa\x33\xc0\xe6\xc2\x82\x53\xf7\xe1\x0c\xa2\x8d\xe5\x97\x50\x99\xdb", 32, 0) = 32
18395 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x68, 0x1c), 0xbeebb344) = -1 EINVAL (Invalid argument)
18395 clock_gettime(CLOCK_BOOTTIME, {tv_sec=6935, tv_nsec=2888242}) = 0
18395 clock_gettime(CLOCK_REALTIME, {tv_sec=1556636168, tv_nsec=237552129}) = 0
18395 ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x68, 0x1c), 0xbeebb2e4) = -1 EINVAL (Invalid argument)

on the other hand openssl rand 10

ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x66, 0x1c), 0xb68bb0) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1556636495, tv_nsec=233188351}) = 0
clock_gettime(CLOCK_BOOTTIME, {tv_sec=7262, tv_nsec=204352769}) = 0
clock_gettime(CLOCK_REALTIME, {tv_sec=1556636495, tv_nsec=233406091}) = 0
ioctl(3, _IOC(_IOC_READ|_IOC_WRITE, 0x63, 0x68, 0x1c), 0xbeff773c) = 0

compiled openssh without-ssl-engine and it now works... definetely something to do with openssl.1.1 or/and openssh 8

Wonder if haveged might offer an assist, or if possibly AF_ALG changes behaviour.