Cert verify failed: BADCERT_EXPIRED (Letsencrypt global root cert)

Hi,

At 2021-09-30 14:01:16 UTC, our openwrt devices using curl stopped working with the following error.

% curl --cacert /etc/ssl/certs/ca-certificates.crt https://www.somedomain.com/
curl: (51) Cert verify failed: BADCERT_EXPIRED

We logged into one of the devices and updated the ca-certificates package but no change.
Does anyone know what's going on? We didn't make any changes and certs are valid on the sites we've tested against.

perhaps related ?

2 Likes

Yes, the server is using lets encrypt and that was a good call on your part.
I re-generated new certs for the server but no change. Still getting the same error.
Is it something on the openwrt side that needs to change too?

I am having the same issue and I also have letsencrypt on the server.

Which OpenWrt version?
Up-to-date 19.07 or 21.02?
Which ca-certificates version?
Openssl version? (Or which SSL?)

In my case, it's the following:

libustream-wolfssl20201210 - 2020-12-10-68d09243-2
libwolfssl4.8.1.ba20a816 - 4.8.1-stable-1
openvpn-wolfssl - 2.5.3-3
openwisp-config-wolfssl - 0.6.0a-1
wpad-mesh-wolfssl - 2021-09-27-1518638b-38

No way to know as they are all remote to me. However, some would be running older 18.x versions. I was able to get into one of those and update the ca-certificates and openssl.

These are installed on it.

ca-bundle - 20200601-1
libopenssl (1.0.2u-1)
openssl-util (1.0.2u-1)

Most of them are gl.inet mt300n V2, cheap on Amazon if that helps.

I just tested and download from master works ok for me with openssl-based wget

root@router1:/tmp# opkg list-installed | grep -E "wget|ssl|cert"
ca-certificates - 20210119-1
libopenssl-conf - 1.1.1l-1
libopenssl1.1 - 1.1.1l-1
libustream-openssl20201210 - 2020-12-10-68d09243-2
luci-ssl-openssl - git-17.031.53232-b6341bd
openssl-util - 1.1.1l-1
wget-ssl - 1.21.1-1
wpad-openssl - 2021-05-22-b102f19b-37

Too old openssl...
From https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

if clients of your API are using OpenSSL, they must use version 1.1.0 or later. In OpenSSL 1.0.x, a quirk in certificate verification means that even clients that trust ISRG Root X1 will fail when presented with the Android-compatible certificate chain we are recommending by default.

I have installed:

root@Repeater_WZ:/# opkg list-installed | grep -E "wget|ssl|cert"
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.7.0.66253b90 - 4.7.0-stable-2
luci-ssl - git-20.244.36115-e10f954
px5g-wolfssl - 3
wpad-basic-wolfssl - 2020-06-08-5a8b3662-35
root@Repeater_WZ:/#

It's time to release 21.02.1 ..

Might be more difficult, if the problem is in the external wolfssl library itself.

(Current openssl 1.1.1 seems to work ok. But the letsencrypt info page says that older 1.0.x openssl versions will fail.)

I see that "ISRG Root X1" is present in the certs, I tried adding "ISRG Root X2" but it didn't fix it.

It looks like a problem with wolfssl to me too, because I have other devices running openwrt 19.07 and openssl which are not affected.

On these devices where I am using wolfssl I am space constrained and there's not enough space to switch to openssl, otherwise I could have tried switching to openssl, which could have fixed it and confirmed the problem is on wolfssl.

My image creater options:
make image PROFILE=tplink_archer-c7-v2 PACKAGES="luci luci-proto-relay luci-ssl luci-app-commands kmod-usb-storage kmod-fs-ext4 kmod-usb-hid block-mount iperf e2fsprogs fdisk swap-utils tar perl perl-www perl-xml-parser perlbase-math perlbase-storable perlbase-version perlbase-autoloader perl-device-usb luci-theme-material kmod-fs-cifs kmod-nls-base kmod-ath10k -kmod-ath10k-ct ath10k-firmware-qca988x -ath10k-firmware-qca988x-ct"

So wolfssl is default
Has anybody opened i ticket ?

% opkg list-installed | grep -E "wget|ssl|cert"
libopenssl - 1.0.2u-1
libustream-openssl - 2018-07-30-23a3f283-2
openssl-util - 1.0.2u-1
openvpn-openssl - 2.4.5-4.2
wget - 1.19.5-6

How do I get libopenssl 1.1 since opkg doesn't seem to know about it?

Update to a currently supported OpenWrt version: 19.07 or 21.02 (or the development master). They all have openssl 1.1.1.

Support for 18.06 has ended in late 2020.

Is there some way to fake the X1 on the server side so that I can recover those devices otherwise, they all have to be shipped back to me. As of hours ago, because curl is using https, it won't communicate at all so I cannot even find a way to recover them.

Default image -> same libs

'# opkg list-installed | grep -E "wget|ssl|cert"
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.7.0.66253b90 - 4.7.0-stable-2
luci-ssl - git-20.244.36115-e10f954
px5g-wolfssl - 3
wpad-mesh-wolfssl - 2020-06-08-5a8b3662-35

Looks like the letsencrypt forum is now full of help requests.

1 Like

Not sure how much they can do to fix it. It's not that they are going to publish a new root certificate or "unexpire" the old one.
I have a 19.07.7 router with openSSL 1.1.1k and that one is having no issues.
It's our 21.02.0 routers that don't seem to recognize the ISRG Root X1 certificate. The wolfSSL libraries seem to have the same "quirk" in openSSL 1.0.x.
Don't have space in all routers for openSSL... maybe one or two will take it.

What could work is to add support for X1 onto the server, even temporarily, long enough to send the devices an update. I don't know enough about this to know how but would it involve adding an intermediate file from letsencrypt somewhere in the letsencrypt directory structure?