We logged into one of the devices and updated the ca-certificates package but no change.
Does anyone know what's going on? We didn't make any changes and certs are valid on the sites we've tested against.
Yes, the server is using lets encrypt and that was a good call on your part.
I re-generated new certs for the server but no change. Still getting the same error.
Is it something on the openwrt side that needs to change too?
No way to know as they are all remote to me. However, some would be running older 18.x versions. I was able to get into one of those and update the ca-certificates and openssl.
if clients of your API are using OpenSSL, they must use version 1.1.0 or later. In OpenSSL 1.0.x, a quirk in certificate verification means that even clients that trust ISRG Root X1 will fail when presented with the Android-compatible certificate chain we are recommending by default.
I see that "ISRG Root X1" is present in the certs, I tried adding "ISRG Root X2" but it didn't fix it.
It looks like a problem with wolfssl to me too, because I have other devices running openwrt 19.07 and openssl which are not affected.
On these devices where I am using wolfssl I am space constrained and there's not enough space to switch to openssl, otherwise I could have tried switching to openssl, which could have fixed it and confirmed the problem is on wolfssl.
Is there some way to fake the X1 on the server side so that I can recover those devices otherwise, they all have to be shipped back to me. As of hours ago, because curl is using https, it won't communicate at all so I cannot even find a way to recover them.
Not sure how much they can do to fix it. It's not that they are going to publish a new root certificate or "unexpire" the old one.
I have a 19.07.7 router with openSSL 1.1.1k and that one is having no issues.
It's our 21.02.0 routers that don't seem to recognize the ISRG Root X1 certificate. The wolfSSL libraries seem to have the same "quirk" in openSSL 1.0.x.
Don't have space in all routers for openSSL... maybe one or two will take it.
What could work is to add support for X1 onto the server, even temporarily, long enough to send the devices an update. I don't know enough about this to know how but would it involve adding an intermediate file from letsencrypt somewhere in the letsencrypt directory structure?