21.02.0 Cannot connect to any letsencrypt site using ssl (Letsencrypt global root cert expiration)

I thought I was doing that by preventing the wolfssl packages from being installed.
As a reminder, I'm using IB so I guess that's why I cannot get past this.

I agree, it would have been great to have curl variants depending on different SSL libraries, but I haven't tried as both champtar and dangowrt concur it's not easy due to the library namespace collisions.

Since there are still issues specifically with WolfSSL, I would still advocate for switching curl dependency to mbedtls by default if wpad can also be made dependent on mbedtls without any sacrifices.

I don't know why I've taken a liking to your problem. Guess I need a hobby.

I have a potential workaround for you, but requires some work outside IB. Based on link I sent before.

Use the image you built with openssl-1.0.2u...

According to that link, you have to remove the expired root certificate (DST Root CA X3) from the trust store. You do that by editing the file at /etc/ssl/certs/ca-certificates.crt.

If you need this baked into the image, it means modifying the file in ca-bundle before you use IB.

To delete the offending certificate from ca-certificates.crt, remove all the following lines (lines 800 to 819 as provided by ca-bundle - 20210119-1):

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

Unfortunately I can't test it now as I no longer have a box with openSSL 1.0.2u. It doesn't work on wolfSSL:

root@Cam1WRT:/etc/ssl/certs# curl -o /tmp/asdf https://shop.bbc.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (77)  CA signer not available for verification

this on

root@Cam1WRT:/tmp# opkg list-installed | grep -E "wget|ssl|cert|bundle"
ca-bundle - 20210119-1
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.7.0.66253b90 - 4.7.0-stable-2
luci-ssl - git-20.244.36115-e10f954
px5g-wolfssl - 3
wpad-basic-wolfssl - 2020-06-08-5a8b3662-35

...BUT it should work on openSSL 1.0.2u according to them.

Do you really need to use openSSL 1.0.2u? What build are you on? 18.06.09? 19.07.7 uses openSSL 1.1.1k and that one should work with no black magic.

1 Like

As stangri already implied, the problem is not so much about curl the standalone tool, but libcurl as linked to by numerous other packages. Easy to sort out on the source level, but not so nice to deal with on a binary level - without exploding package permutations further up the chain.

If the issue is wolfSSL, as openSSL 1.1.1 works... hasn't this been fixed by wolfSSL yet?

21.02.0 uses wolfSSL 4.7.0. Latest is 4.8.1 and is available in snapshot.

It's a bit uphill for me but I can try rebuild the 21.02.0 image I use with the 4.8.1 wolfSSL package, hoping that works. That way there's no need to come up with libcurl variants.

If you are running your own server, changing the cert on the server will work (tested on my hundreds of devices with wolfssl as the curl SSL lib already)

Rebuilt the image I use with the latest wolfSSL library (v4.8.1):

$ ssh root@wrt


BusyBox v1.33.1 (2021-08-31 22:20:08 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 21.02.0, r16279-5cc0535800
 -----------------------------------------------------

root@WRT:/root# ldd /usr/bin/curl
	/lib/ld-musl-mips-sf.so.1 (0x77d7e000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x77d2e000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77d00000)
	libwolfssl.so.4.8.1.39c36f2f => /usr/lib/libwolfssl.so.4.8.1.39c36f2f (0x77b60000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77b3c000)
	libc.so => /lib/ld-musl-mips-sf.so.1 (0x77d7e000)

root@WRT:/root# opkg list-installed | grep wolf
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.8.1.39c36f2f - 4.8.1-stable-1
px5g-wolfssl - 3
wpad-basic-wolfssl - 2020-06-08-5a8b3662-35

It was an interesting exercise... the toolkit needs a package updated (libtool) in order to build the latest wolfSSL.

Regardless, it makes no difference. It doesn't work.

root@WRT:~# cd /tmp
root@WRT:/tmp# curl -o /tmp/asdf https://shop.bbc.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (77)  CA signer not available for verification

@biangbiangmian has the easiest solution... change the cert on the server(s).

If that is not an option, install openSSL (on v19 or v21). Will work for anything connecting to letsencrypt and not using wolfSSL.

No decent workarounds for curl other than manually building it and linking it to anything but libwolfssl.

Wow, lots of posts since the last time I looked. I can definitely try editing the bundle and see what happens.
To edit the cert out, do I have to remove all instances?

~# grep -r "MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMD" /etc/ssl/
/etc/ssl/cert.pem:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
/etc/ssl/certs/2e5ac55d.0:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
/etc/ssl/certs/DST_Root_CA_X3.crt:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
/etc/ssl/certs/ca-certificates.crt:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/

I removed it in /etc/ssl/cert.pem but no change.

%  curl  https://shop.bbc.com/
curl: (77)  CA signer not available for verification

I'll remove the rest and try again.

I removed the instance and the files and no change.

It's not just my own problem, it's others who might be in the same situation. If they don't have a clear way of fixing this, they might stop using the project and move on to something else. The project needs to understand that it's not only those that know openwrt inside out that it is catering to but countless others that usually get by without needing to know the inside out of the code or coding. Those are the ones that quietly leave out of frustration.

In my case editing the /etc/letsencrypt/site.name/fullchain.pem file to remove the last
certificate and restarting nginx did it for me.

I tried that based on some advise I saw in some posts on LE but that didn't work for me. The only way to get things back online was to swap the LE cert to a costly GoDaddy one.

Going to an older version is fine for me. I don't need anything special in my build. I'll give that a try.

Just tried using 19.07.8, same result.

You have both ca-bundle and ca-certificates installed? I don't think you need both. ca-bundle is enough.

Just to make sure... that curl is using OpenSSL right? ldd /usr/bin/curl and ldd /usr/lib/libcurl* must not return references to libwolfssl.so .

I'm rebuilding another test w/curl+openssl just to ensure it works. Should be able to test it before night.

New image confirms that wolfSSL is the issue. Took the image that failed w/wolfSSL, rebuilt just by changing the SSL library for curl from wolf to open.

This is 21.02.0:

$ssh root@wrt


BusyBox v1.33.1 (2021-08-31 22:20:08 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 21.02.0, r16279-5cc0535800
 -----------------------------------------------------

OpenSSL installed:

root@WRT:~# opkg list-installed | grep -E 'ssl|ca-'
ca-bundle - 20210119-1
libopenssl1.1 - 1.1.1l-1
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.8.1.39c36f2f - 4.8.1-stable-1
luci-ssl - git-20.244.36115-e10f954
px5g-wolfssl - 3
wpad-basic-openssl - 2020-06-08-5a8b3662-35

curl & libs linked to OpenSSL:

root@WRT:~# ldd /usr/bin/curl
	/lib/ld-musl-mips-sf.so.1 (0x77de0000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x77d8a000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77d5c000)
	libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0x77cdc000)
	libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x77b02000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77ade000)
	libc.so => /lib/ld-musl-mips-sf.so.1 (0x77de0000)
root@WRT:~# ldd /usr/lib/libcurl*
	ldd (0x77e2e000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77daa000)
	libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0x77d2a000)
	libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x77b50000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77b2c000)
	libc.so => ldd (0x77e2e000)

It works:

root@WRT:~# curl -o /tmp/whatever https://shop.bbc.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  448k    0  448k    0     0   634k      0 --:--:-- --:--:-- --:--:--  767k

1 Like

Agree. That's why I'm having fun with it. Although your problem is pretty specific: you want curl from the router to talk to a letsencrypt SSL secured server. Most people already have a workaround for opkg to work or can use wget instead. Don't underestimate how special you are. :wink:

Power to them. They can go back to stock firmware. I'm sure that allows curl to random locations.

I'm trying my best for you not to be one of them.

1 Like

Believe it, I appreciate it even if text sometimes doesn't sound like it :).
Yes, I'd love to find a solution and more importantly, be able to write SOLVED on the first message to help others that find this.

Hey, no worries.

Well, again, the root cause of your problem is that wolfSSL doesn't work with letsencrypt certificates. An updated libwolfsssl is not going to be available anytime soon as even the latest stable version used in snapshots still won't work with letsencrypt.

For everything but curl you have a workaround, such as installing libustream-openssl to make wget work.

There's really no way out for curl just using the IB. Besides switching server certificates (which is really not a OpenWRT solution) your only option is to rebuild it.

Here are the official instructions to build a single package:

Those instructions need to be followed to set up the build environment.

To configure the build for curl:

  • select Target System and Target Profile to match your device.
  • under Network then File Transfer select curl as a module.
  • under Libraries then libcurl and then options within (so <enter>) you will need to Selected SSL library and ensure OpenSSL is selected and not wolfSSL.

After saving everything you should be able to build following the instructions above for building a single package.

The packages/components that need to be built are:

  • tools
  • toolchain
  • libnghttp2
  • libopenssl
  • libcurl
  • curl

Hope I didn't miss anything.

My luck building one package is always hit and miss. It might be easier to do a full build. Takes longer, but takes less typing and usually always works. Even though I run full builds, I usually just take from it the packages I need which is what I would recommend for this.

IB should let you build an image including libopenssl. I suggest you don't include curl and libcurl in it as there is no point in wasting space on the wolfSSL packages. Take instead the curl and libcurl packages from the bin/ directory, scp them to the /tmp/ directory in the device once it is up and running, and opkg install them.

curl should work fine at that point.

Your other alternative might be to wait for OpenWRT v21.02.1 and hope that one uses OpenSSL instead of wolfSSL but I wouldn't hold my breath for it. You could also wait for an OpenSSL variant of curl but that doesn't seem to be coming online quickly either. If waiting for a while with no clear indiction of how long is not an option, the build above is IMO your only alternative.

1 Like

I just ran into this fun little issue myself trying to get acme.sh working...I think this is probably a more common issue than some of you are making it out to be. Recompiling with support for openssl did the trick. I'm a bit confused as to what the namespace collision issues here are-is it that things might fail to compile against a curl-openssl package? Can that not be bypassed with creative dependency requirements?

1 Like

I'm poking at it the way you would poke a bear with a stick. Never done either before.

The first issue is how much of this is to be owned by OpenWRT. Packages are usually pretty slim... as in get source tarball, patch whatever is required, and build binaries. This seems to need a level of gunk above that would be OpenWRT specific. Would be great if upstream curl had the option of building all flavours but either it doesn't or I haven't seen it yet.

Then there is libcurl. There is no good way to have a "libcurl-wolfssl" and a "libcurl-openssl" flavour. Programs like curl expect it to be "libcurl" and that's it. A separate problem is that libcurl is not built separately from curl.

One easy way is to make curl use OpenSSL by default. That will add a lot of pressure to devices with small flash. Not sure it's a bad solution since curl is not a base package. wolfSSL is certainly smaller and that's to be valued.

That brings wolfSSL up. That's the root cause of the issue in 21.02.0. My concern is that nobody has made it known to them that their library won't connect to a letsencrypt server, as far as I can see from their forums. Not only that means the issue will never be fixed, but also that this is a pretty small and specific use of their library.

So again... maybe curl should be built by default with OpenSSL, and if anyone needs a smaller version, they can custom build with wolfSSL.

Why not?
We have several packages with alternative variants, where each variant provides the identically named library X and all variants declare in Makefile that they PROVIDE library package X.

libcurl should provide identical API upward, so I still do not understand what would be the main difficulty in creating the variants. Only one libcurl variant could be installed at one time, but that is normal.

Some library examples:

But I haven't looked deeper into libcurl details. The install-dev section in Makefile is more complicated than normally, so maybe it really contains something that makes the variant creation poisonous...

1 Like

Learning as I go... sorry for the edits, but as I find out more I realize how off the mark they are.

Challenge I see now is all the libcurl options, and how their availability depends on which TLS library is selected. That is much more complex than the examples you posted.

I don't mind poking at it as I'm learning about the build system as I go, but this is above my skills.