wolfSSL, OpenSSL and the risk of running both

WPA3 and HTTPS (Luci) will be available by default in OpenWRT 21.02 using the wolfSSL library. Despite both features having been available since 19.07 I think adding this by default is a great feature for most users :slight_smile:

I understand the choice for wolfSSL as default SSL library, and for users who simply use the default builds this will have no impact at all. However, the power of OpenWRT lies in its modularity, and many users install additional packages to fulfill their needs. Either 'on top of' with OPKG, or with custom builds via the Imagebuilder. That is where picking the right SSL library becomes more tedious. I foresee a significant number of deployments running both wolfSSL and OpenSSL in parallel due to package-dependencies and unclarity thereof. Especially on systems at the lower end of the spectrum regarding memory and compute, this will result in a performance hit.

For example, installing OpenVPN requires OpenSSL, the same goes for Stubby (DoT). So to avoid running both wolfSSL and OpenSSL in parallel, anyone running OpenVPN or Stubby is better off building a custom image with something like:

-luci-ssl -wpad-basic-wolfssl luci-ssl-openvpn wpad-basic-openssl

This is in principle perfectly doable for anyone who has some basic skills in Linux and running Linux VMs, as long as they invest some time in investigating the dependencies of their packages.

There are also packages which require compiling from source to be compatible with a specific SSL library. For example, the curl package is by default compiled against wolfSSL, but can be compiled from source for OpenSSL manually. IMHO compiling from source is a significant step up from the imagebuilder in terms of complexity and skill. Not that not all packages can be compiled this way, OpenVPN and NextDNS (for Stubby) explicitly require the OpenSSL library.

So in short:

  • Advanced users can likely avoid running both wolfSSL and OpenSSL in parallel with either the image builder or compiling from source.
  • Unaware or less-advanced users face the risk of (unknowingly) running both wolfSSL and OpenSSL in parallel due to package dependencies.

My question to the community and the devs is this:

  • Will effort be invested in switching packages to use wolfSSL by default, knowing this is mostly a dependency on external package maintainers?
  • Will more packages (e.g. curl) be offered for BOTH wolfSSL and OpenSSL libraries, allowing them to be selected via the Imagebuilder instead of compiling them from source?
  • What else can we do to try to prevent users from (unknowingly) running both wolfSSL and OpenSSL?
2 Likes

There is no 'risk' involved with running two ssl libraries.

OpenSSL needs around 1 MB flash storage, wolfSSL around half of that - that is the only reason to prefer wolfSSL over OpenSSL by default. If both are installed, both need to fit into the flash and leave their footprint in RAM, but that's about it.

Yes, depending on the packages using either of those SSL libraries, you may be suspectible to potential security issue in both of them (with differing attack vectors depending on which package uses which SSL provider), but that's not really a concern as all security bugs need to be fixed anyways.

Don't expect OpenWrt to drive porting external packages from one SSL provider to another, this involves serious work and familiarity with the upstream code or you'd end up with more problems than answers. If you care about this for an individual package, approach upstream and work with them, present a patch (both upstream and later to the OpenWrt packaging).

2 Likes

Is there some reason why the packaging of wolfssl has some versioning information directly in the package names? I am trying to automate upgrades of additional packages (ones I installed that did not come with the image) and having this info in the package names interferes with being able to write a clean script to do this.

As an example, there is a package named libwolfssl4.8.1.66253b90 in openwrt 21.02.1, but the same package is named libwolfssl5.1.1.99a5b54a in openwrt 21.02.2. An automated process would have to be smart enough to know that these are just different versions of the same package.

By contrast, packages like dropbear do not carry any version info in their package name; version info is available by inspecting the package contents of course (or using luci which does that for you).

I totally get that this is done for convenience of developers. But I think for people who add a lot of extra packages (that's conceivable I think), trying to restore packages after an update might present some trouble, trying to remember which packages they had before and after upgrades.

I also get that scripting this process might be overkill for other users. I was just trying to streamline the upgrading process a little bit. I find that I sometimes need to attempt an upgrade a few times on a few routers, and this would make things easier, that's all. Please don't shoot me for trying/asking; thanks.

NB I note that openssl is named openssl1.1, but that does not change between these two openwrt images.

Libraries are special, you somehow need to guarantee ABI compatibility - beyond sonames and in the absence of symbol versioning (as opkg isn't smart enough to do the later) - or define ABI breakage. If you don't, packages would crash, although the dependencies are (at least from a superficial look) are fullfilled.

1 Like

Any possibility that sysupgrade could handle this for us? And by that, I mean, not only install equivalent packages after upgrade, but also remove packages that had been removed by us before the upgrade.

As an example, not only do I like to have tcpdump, htop, strace, and a few others on board, I also like to remove odhcp6c, as well as set dropbear to disabled (I start it when I need it only and then stop it when I no longer need it, or reboot the router).

No, I've seen similar approaches to automatically filter package lists in order to restore them after an upgrade elsewhere two decades ago (with much smarter package managers involved), it's bound to fail - and it usually fails in an ugly way.

Package names (especially those indirectly pulled in via dependencies) are volatile, that's a fact of life,

The leaf package vim might always be called vim (or vim-full, or vim-fuller, …), but other packages come and go - we've just seen it in 21.02 with swconfig --> dsa, we will see it again with 22.xy.z and fw3 --> fw4 - and those are just the big elephant in the room. Library ABIs will change by definition, even within maintenance releases of a single stable release (and OpenWrt doesn't even package/ version kernel or libc, that would be real fun to deal with).

Not meaning to disagree, but I note that rpm and apt don't seem to suffer this way. Virtually every package can be upgraded and compatibility is mostly ensured. (There are a few exceptions to this, such as systems that do not run systemd.) From your previous remarks, it sounds like opkg badly needs that symbol versioning facility (or something). If opkg had that support, would this discussion be largely obviated?

  1. they are much, much larger (over 10 MB for dpkg+apt) than opkg
  2. they'd fail in exactly the same way, if you'd take a package list from an older distribution release and now want to reinstall that list on a newer distribution release, what they could do, is in-place dist-upgrades (and there are reasons why this can't work with OpenWrt, unless devices gets much larger flash and more flexible ways of booting).
  3. they spend a lot more effort on packaging, ABI versioning and symbol versioning, to allow co-installable ABI versions and do much tighter dependency tracking.
    If you wanted to do that on OpenWrt, you'd have to
    • replace opkg with apt+dpkg or rpm (or portage, pacman, etc. maybe apk)
      • this would raise the minimum system requirements in regards to the flash size by at least 30-50 MB, both to accomodate for a more capable (larger-) package manager and allow at least some basic co-installability of libraries.
    • spend a lot more developer time on ABI tracking, you'd need 2-3 times the manpower at least
    • work on symbol tracking (infrastructure), to help developers taking care of it.
1 Like

slh: Thank you for all of your responses. Now I understand the limitations.

Rosen Penev said:

the reason OpenWrt switched to wolfssl is b cause hostapd has no support for mbedTLS

In the same thread author of Lighttpd said the he prefer MbedTLS.
So I guess in future OpenWrt may switch back to MbedTLS instead of WolfSSL. Please confirm.

The OpenVPN has openvpn-mbedtls version. But it looks like there is a licensing problem

But the OpenSSL is not only a library but also a tool from openssl-util package. Many packages like acme.sh depends on it. So ideally to keep a disk space it would be great to have something in replace. There is mbedtls-util package that contains a gen_key program. Maybe it can be used in acme.sh instead of openssl.
wget-ssl depends on libopenssl and looks like doesn't support the mbedtls.
From sources it looks like curl can be compiled with mbedtls but a curl package version again depends on libopenssl.

The primary reason for switching to wolfssl is indeed hostapd, which requires either openssl or wolfssl to support wpa3/ SAE, but not mbedtls. Among these two options, wolfssl is the smaller one, while openssl would be larger, but functionally more reliable (we've seen quite a few ABI surprises and functional bugs with wolfssl and its hostapd integration already). But, as there is no mbedtls support in hostapd so far (nor any efforts to develop such), your question about switching back to mbedtls is hypothetical at best.

4 Likes

Thank you for the clarification. There is a patch to hostapd with mbedtls support so hope this will be merged.
I created a wiki page with list of the crypto libraries

it looks like redundant but having the small article might save my a lot of time when I tried to understand what's the difference. From both user and developer perspective. If you have anything to add there please do.

as there is no mbedtls support in hostapd so far (nor any efforts to develop such), your question about switching back to mbedtls is hypothetical at best.

Glenn Strauss, the author of Lighttpd started working on this

Please create some issue ticket so that all interested people may find and contribute

Since this commit, a mbedtls-variant for hostapd exists. I've been running it for a while and I'm not noticing any performance or stability issues. The only 'problem' is that there isn't a mbedtls-variant for luci-ssl anymore.

Will OpenWRT switch back from wolfssl to mbedtls as default in the future?

2 Likes