21.02.0 Cannot connect to any letsencrypt site using ssl (Letsencrypt global root cert expiration)

% opkg install libstdcpp
Package libstdcpp6 (8.4.0-3) installed in root is up to date.

It's available in my list -

libstdcpp6
7.5.0-2
385.1 KB
GNU Standard C++ Library v3

(Looks like you edited your post)

Seems to be installed. Time to go talk with someone about working on some code I guess.

% opkg list-installed | grep libstdcpp
libstdcpp6 - 8.4.0-3

Yes, sorry, I edited the post because I forgot the l when I searched :slight_smile:

Well, I think that's the problem.

OpenWRT v18's package is libstdcpp (no 6). And it's "small" (at 250K or so). v19 renamed it to libstdcpp6, and it's "large" (at 350K or so). v21 keeps the libstdcpp6 name, but it's back to "small" size.

My guess is that the v19 variety has more stuff in it... exported some extra symbols that your application is missing.

You could try installing the v19 package to the v21 image, but it's almost a sure ticket to bricking the device because it's a base package. I'd try if I had an easy way to debrick the device (a backup loader or so).

Does the app work on a v19 image?

Most probably you will have to rebuild and link that pingdev app to the latest libstdcpp.

1 Like

Have you tried?

We have variants for quite many different packages regarding their SSL support variants, and I am not aware what would make curl exceptionally difficult. As curl already offers the config option for SSL lib selection, I wonder what is the non-easy part. That one comment from dangowrt did not contain any real explanation...

Having an openssl variant available would ease things a lot for imagebuilder users. (Full toolchain users, like me, can already now configure curl as we like)

I'm not 100% sure what is being asked but here you go. This is after rebuilding to 21.02.0 using curl and what ever ca bundle/certs come with it.
These are sites listed on Lets Encrypt as using their certs. All of them fail but sites not using LE work.

% curl  https://shop.bbc.com/
curl: (77)  CA signer not available for verification
% curl  https://labs.criteo.com/
curl: (77)  CA signer not available for verification
% curl  https://www.petco.com/
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

As shown above;

I've built a 21.02.0 version with the current tools using IB.

% opkg list-installed | grep -E "wget|ssl|cert"
libopenssl - 1.0.2u-1
libustream-openssl - 2018-07-30-23a3f283-2
openvpn-openssl - 2.4.5-4.2
wget - 1.19.5-6

Is this actually a curl problem or the ca-bundle/ca-certificates that need updating?

Again... you need libopenssl 1.1.1.

1.0.2x will not work.

This is the version I see available:

libopenssl1.1 - 1.1.1l-1

Letsencrypt has plenty of options here. It involves removing certificates or stuff. I think it would be easier to just use v1.1.1x.

I thought I was doing that by preventing the wolfssl packages from being installed.
As a reminder, I'm using IB so I guess that's why I cannot get past this.

I agree, it would have been great to have curl variants depending on different SSL libraries, but I haven't tried as both champtar and dangowrt concur it's not easy due to the library namespace collisions.

Since there are still issues specifically with WolfSSL, I would still advocate for switching curl dependency to mbedtls by default if wpad can also be made dependent on mbedtls without any sacrifices.

I don't know why I've taken a liking to your problem. Guess I need a hobby.

I have a potential workaround for you, but requires some work outside IB. Based on link I sent before.

Use the image you built with openssl-1.0.2u...

According to that link, you have to remove the expired root certificate (DST Root CA X3) from the trust store. You do that by editing the file at /etc/ssl/certs/ca-certificates.crt.

If you need this baked into the image, it means modifying the file in ca-bundle before you use IB.

To delete the offending certificate from ca-certificates.crt, remove all the following lines (lines 800 to 819 as provided by ca-bundle - 20210119-1):

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

Unfortunately I can't test it now as I no longer have a box with openSSL 1.0.2u. It doesn't work on wolfSSL:

root@Cam1WRT:/etc/ssl/certs# curl -o /tmp/asdf https://shop.bbc.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (77)  CA signer not available for verification

this on

root@Cam1WRT:/tmp# opkg list-installed | grep -E "wget|ssl|cert|bundle"
ca-bundle - 20210119-1
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.7.0.66253b90 - 4.7.0-stable-2
luci-ssl - git-20.244.36115-e10f954
px5g-wolfssl - 3
wpad-basic-wolfssl - 2020-06-08-5a8b3662-35

...BUT it should work on openSSL 1.0.2u according to them.

Do you really need to use openSSL 1.0.2u? What build are you on? 18.06.09? 19.07.7 uses openSSL 1.1.1k and that one should work with no black magic.

1 Like

As stangri already implied, the problem is not so much about curl the standalone tool, but libcurl as linked to by numerous other packages. Easy to sort out on the source level, but not so nice to deal with on a binary level - without exploding package permutations further up the chain.

If the issue is wolfSSL, as openSSL 1.1.1 works... hasn't this been fixed by wolfSSL yet?

21.02.0 uses wolfSSL 4.7.0. Latest is 4.8.1 and is available in snapshot.

It's a bit uphill for me but I can try rebuild the 21.02.0 image I use with the 4.8.1 wolfSSL package, hoping that works. That way there's no need to come up with libcurl variants.

If you are running your own server, changing the cert on the server will work (tested on my hundreds of devices with wolfssl as the curl SSL lib already)

Rebuilt the image I use with the latest wolfSSL library (v4.8.1):

$ ssh root@wrt


BusyBox v1.33.1 (2021-08-31 22:20:08 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 21.02.0, r16279-5cc0535800
 -----------------------------------------------------

root@WRT:/root# ldd /usr/bin/curl
	/lib/ld-musl-mips-sf.so.1 (0x77d7e000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x77d2e000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77d00000)
	libwolfssl.so.4.8.1.39c36f2f => /usr/lib/libwolfssl.so.4.8.1.39c36f2f (0x77b60000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77b3c000)
	libc.so => /lib/ld-musl-mips-sf.so.1 (0x77d7e000)

root@WRT:/root# opkg list-installed | grep wolf
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.8.1.39c36f2f - 4.8.1-stable-1
px5g-wolfssl - 3
wpad-basic-wolfssl - 2020-06-08-5a8b3662-35

It was an interesting exercise... the toolkit needs a package updated (libtool) in order to build the latest wolfSSL.

Regardless, it makes no difference. It doesn't work.

root@WRT:~# cd /tmp
root@WRT:/tmp# curl -o /tmp/asdf https://shop.bbc.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (77)  CA signer not available for verification

@biangbiangmian has the easiest solution... change the cert on the server(s).

If that is not an option, install openSSL (on v19 or v21). Will work for anything connecting to letsencrypt and not using wolfSSL.

No decent workarounds for curl other than manually building it and linking it to anything but libwolfssl.

Wow, lots of posts since the last time I looked. I can definitely try editing the bundle and see what happens.
To edit the cert out, do I have to remove all instances?

~# grep -r "MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMD" /etc/ssl/
/etc/ssl/cert.pem:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
/etc/ssl/certs/2e5ac55d.0:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
/etc/ssl/certs/DST_Root_CA_X3.crt:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
/etc/ssl/certs/ca-certificates.crt:MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/

I removed it in /etc/ssl/cert.pem but no change.

%  curl  https://shop.bbc.com/
curl: (77)  CA signer not available for verification

I'll remove the rest and try again.

I removed the instance and the files and no change.

It's not just my own problem, it's others who might be in the same situation. If they don't have a clear way of fixing this, they might stop using the project and move on to something else. The project needs to understand that it's not only those that know openwrt inside out that it is catering to but countless others that usually get by without needing to know the inside out of the code or coding. Those are the ones that quietly leave out of frustration.

In my case editing the /etc/letsencrypt/site.name/fullchain.pem file to remove the last
certificate and restarting nginx did it for me.

I tried that based on some advise I saw in some posts on LE but that didn't work for me. The only way to get things back online was to swap the LE cert to a costly GoDaddy one.

Going to an older version is fine for me. I don't need anything special in my build. I'll give that a try.

Just tried using 19.07.8, same result.

You have both ca-bundle and ca-certificates installed? I don't think you need both. ca-bundle is enough.

Just to make sure... that curl is using OpenSSL right? ldd /usr/bin/curl and ldd /usr/lib/libcurl* must not return references to libwolfssl.so .

I'm rebuilding another test w/curl+openssl just to ensure it works. Should be able to test it before night.