Package build locates "wrong" OpenSSL library - how to force the correct one?

For Developers
1

I am currently attempting to update the softethervpn5 package, since the existing version stopped working on newer kernels (from the 22.x series, as far as I can tell). The fix involves updating the underlying SoftetherVPN libraries to a newer build. While I can cobble a working build together with some manual intervention, I am unfortunately stumped how to achieve a reliable build system build.

The issue:
The softethervpn5/host library compilation fails in a check for two OpenSSL functions. Specifically, those library builds include the following in a CMakeLists:

find_package(OpenSSL REQUIRED)
set(CMAKE_REQUIRED_INCLUDES ${OPENSSL_INCLUDE_DIR})
set(CMAKE_REQUIRED_LIBRARIES OpenSSL::Crypto)
set(CMAKE_REQUIRED_LIBRARIES OpenSSL::SSL)
check_symbol_exists(EVP_PKEY_get_raw_public_key "openssl/evp.h" HAVE_EVP_PKEY_GET_RAW_PUBLIC_KEY)
check_symbol_exists(SSL_CTX_set_num_tickets "openssl/ssl.h" HAVE_SSL_CTX_SET_NUM_TICKETS)

But, those aren't included in the LibreSSL (?) that it locates:

-- Found OpenSSL: /home/tew/openwrt-sdk/staging_dir/host/lib/libcrypto.a (found version "2.0.0")
-- Looking for EVP_PKEY_get_raw_public_key
-- Looking for EVP_PKEY_get_raw_public_key - not found
-- Looking for SSL_CTX_set_num_tickets
-- Looking for SSL_CTX_set_num_tickets - not found
CMake Error at src/Mayaqua/CMakeLists.txt:34 (message):
  Required EVP_PKEY_get_raw_public_key() not found in OpenSSL library!

Now, the package Makefile does specify DEPENDS:= +libopenssl, and the proper OpenSSL libraries are in fact being compiled -- only they are under /staging_dir/target-mips_24kc_musl/usr/ where they are either not found or overridden by those in /staging_dir/host/.

The question then: How to force-include the "correct" OpenSSL version?

Any help is much appreciated!

Attempted Solution:
I did try adding openssl to HOST_BUILD_DEPENDS and/or PKG_BUILD_DEPENDS, but that did not change anything at all as far as I could ascertain.
I also tried openssl/host for HOST_BUILD_DEPENDS, but that yields the Warning: build dependency on 'openssl/host', which does not exist

How I was able to force the build:
I eliminated the offending symbol checks for the host library build. But that doesn't seem like the smartest solution.

2

yeah, there is no host build for openssl

I have a PR to add an openssl host build, but for another reason that didn't get a lot of attention and agreement on. maybe it is needed after all

The idea there is, it can coexist with our host openssl (which is actually libressl in the tools directory) by installing into staging_dir/hostpkg where libressl installs into staging_dir/host

I would first grep into the libressl source to see for sure whether those functions are actually there or not, otherwise it could just be a compilation issue. That would be in a directory like
build_dir/host/libressl-2.4.6
or whatever the version is now

3

Yeah I did confirm that the functions are in fact missing in the LibreSSL version included with the 22.03 toolchain.

I also tested compiling the package for master branch with the snapshot SDK today, which includes an updated LibreSSL - and that actually works fine. Unfortunately I can't really test snapshot builds on my router...

And good to know - an openssl/host build option could probably fix the issues in 22.03 (assuming the correct libraries get precedence), but then again since this affects only one currently unmaintained and not super popular package in one specific branch it may not be a good enough reason to create that option?

Lastly, I suppose that a "dirtier" workaround as I originally had in mind, like messing with CFLAGS/LDFLAGS, or copying of the relevant libraries from $target/usr into /hostpkg, could run into unexpected issues?