I had this problem before and I forgot the solution
Using OpenVPN with OpenSSL and the Cryptodev engine always fails the OpenVPN crypto test at byte 560 with a RAND_bytes() fail assertion. This is regardless of any hardware crypto driver loaded.
I had this working before I did full clean install of the build environment so it must be some selection I missed.
The obvious:
select the hardware acceleration in the OpenSSL lib.
have some (p)rng show up under /proc/crypto
check dmesg log that the CRNG is initialized.
Low entropy could be a reason, but if I run the test again immediately it runs until the same 560. I have the same problem on different SoC / builds so it must be a selection / module I’m missing.
The OpenVPN --test-crypto test runs fine as long as I don't select Use Hardware / Accelerated Digests.
This doesn't make a lot of sense (glancing over both the cryptodev and openssl code). Both are not (p)rng related?? For now I keep Hardware Digest enabled to do benchmarking using Openssl-utils.
Thanks for pointing me at those pull requests. However this problem doesnt seem to be device specific. Without any hardware driver loaded it generates the same problem.
I will try again upgrading to the OpenSSL 1.1.1 version (we are still using 1.0.1). There are some patches already but it didn’t seem to make a big difference a few months ago when we did some testing. Admittedly Cipher tests only, no digests which is what I’m trying to implement now.
Edit:
I did a quick modification to my code to see what is being called. It seems that during every cycle of the OpenVPN --test-crypto it is somehow calling SHA1. This seems be be OpenSSL related, but its strange that the cryptodev doesn't fallback to software-only as it does with ciphers.