21.02.0 Cannot connect to any letsencrypt site using ssl (Letsencrypt global root cert expiration)

New image confirms that wolfSSL is the issue. Took the image that failed w/wolfSSL, rebuilt just by changing the SSL library for curl from wolf to open.

This is 21.02.0:

$ssh root@wrt


BusyBox v1.33.1 (2021-08-31 22:20:08 UTC) built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 21.02.0, r16279-5cc0535800
 -----------------------------------------------------

OpenSSL installed:

root@WRT:~# opkg list-installed | grep -E 'ssl|ca-'
ca-bundle - 20210119-1
libopenssl1.1 - 1.1.1l-1
libustream-wolfssl20201210 - 2020-12-10-68d09243-1
libwolfssl4.8.1.39c36f2f - 4.8.1-stable-1
luci-ssl - git-20.244.36115-e10f954
px5g-wolfssl - 3
wpad-basic-openssl - 2020-06-08-5a8b3662-35

curl & libs linked to OpenSSL:

root@WRT:~# ldd /usr/bin/curl
	/lib/ld-musl-mips-sf.so.1 (0x77de0000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0x77d8a000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77d5c000)
	libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0x77cdc000)
	libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x77b02000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77ade000)
	libc.so => /lib/ld-musl-mips-sf.so.1 (0x77de0000)
root@WRT:~# ldd /usr/lib/libcurl*
	ldd (0x77e2e000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x77daa000)
	libssl.so.1.1 => /usr/lib/libssl.so.1.1 (0x77d2a000)
	libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x77b50000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77b2c000)
	libc.so => ldd (0x77e2e000)

It works:

root@WRT:~# curl -o /tmp/whatever https://shop.bbc.com
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  448k    0  448k    0     0   634k      0 --:--:-- --:--:-- --:--:--  767k

1 Like

Agree. That's why I'm having fun with it. Although your problem is pretty specific: you want curl from the router to talk to a letsencrypt SSL secured server. Most people already have a workaround for opkg to work or can use wget instead. Don't underestimate how special you are. :wink:

Power to them. They can go back to stock firmware. I'm sure that allows curl to random locations.

I'm trying my best for you not to be one of them.

1 Like

Believe it, I appreciate it even if text sometimes doesn't sound like it :).
Yes, I'd love to find a solution and more importantly, be able to write SOLVED on the first message to help others that find this.

Hey, no worries.

Well, again, the root cause of your problem is that wolfSSL doesn't work with letsencrypt certificates. An updated libwolfsssl is not going to be available anytime soon as even the latest stable version used in snapshots still won't work with letsencrypt.

For everything but curl you have a workaround, such as installing libustream-openssl to make wget work.

There's really no way out for curl just using the IB. Besides switching server certificates (which is really not a OpenWRT solution) your only option is to rebuild it.

Here are the official instructions to build a single package:

Those instructions need to be followed to set up the build environment.

To configure the build for curl:

  • select Target System and Target Profile to match your device.
  • under Network then File Transfer select curl as a module.
  • under Libraries then libcurl and then options within (so <enter>) you will need to Selected SSL library and ensure OpenSSL is selected and not wolfSSL.

After saving everything you should be able to build following the instructions above for building a single package.

The packages/components that need to be built are:

  • tools
  • toolchain
  • libnghttp2
  • libopenssl
  • libcurl
  • curl

Hope I didn't miss anything.

My luck building one package is always hit and miss. It might be easier to do a full build. Takes longer, but takes less typing and usually always works. Even though I run full builds, I usually just take from it the packages I need which is what I would recommend for this.

IB should let you build an image including libopenssl. I suggest you don't include curl and libcurl in it as there is no point in wasting space on the wolfSSL packages. Take instead the curl and libcurl packages from the bin/ directory, scp them to the /tmp/ directory in the device once it is up and running, and opkg install them.

curl should work fine at that point.

Your other alternative might be to wait for OpenWRT v21.02.1 and hope that one uses OpenSSL instead of wolfSSL but I wouldn't hold my breath for it. You could also wait for an OpenSSL variant of curl but that doesn't seem to be coming online quickly either. If waiting for a while with no clear indiction of how long is not an option, the build above is IMO your only alternative.

1 Like

I just ran into this fun little issue myself trying to get acme.sh working...I think this is probably a more common issue than some of you are making it out to be. Recompiling with support for openssl did the trick. I'm a bit confused as to what the namespace collision issues here are-is it that things might fail to compile against a curl-openssl package? Can that not be bypassed with creative dependency requirements?

1 Like

I'm poking at it the way you would poke a bear with a stick. Never done either before.

The first issue is how much of this is to be owned by OpenWRT. Packages are usually pretty slim... as in get source tarball, patch whatever is required, and build binaries. This seems to need a level of gunk above that would be OpenWRT specific. Would be great if upstream curl had the option of building all flavours but either it doesn't or I haven't seen it yet.

Then there is libcurl. There is no good way to have a "libcurl-wolfssl" and a "libcurl-openssl" flavour. Programs like curl expect it to be "libcurl" and that's it. A separate problem is that libcurl is not built separately from curl.

One easy way is to make curl use OpenSSL by default. That will add a lot of pressure to devices with small flash. Not sure it's a bad solution since curl is not a base package. wolfSSL is certainly smaller and that's to be valued.

That brings wolfSSL up. That's the root cause of the issue in 21.02.0. My concern is that nobody has made it known to them that their library won't connect to a letsencrypt server, as far as I can see from their forums. Not only that means the issue will never be fixed, but also that this is a pretty small and specific use of their library.

So again... maybe curl should be built by default with OpenSSL, and if anyone needs a smaller version, they can custom build with wolfSSL.

Why not?
We have several packages with alternative variants, where each variant provides the identically named library X and all variants declare in Makefile that they PROVIDE library package X.

libcurl should provide identical API upward, so I still do not understand what would be the main difficulty in creating the variants. Only one libcurl variant could be installed at one time, but that is normal.

Some library examples:

But I haven't looked deeper into libcurl details. The install-dev section in Makefile is more complicated than normally, so maybe it really contains something that makes the variant creation poisonous...

1 Like

Learning as I go... sorry for the edits, but as I find out more I realize how off the mark they are.

Challenge I see now is all the libcurl options, and how their availability depends on which TLS library is selected. That is much more complex than the examples you posted.

I don't mind poking at it as I'm learning about the build system as I go, but this is above my skills.

The funny thing is that it's only curl that I need, nothing else.

I have similar issue.

curl https://sub.cloudflare.bet/
curl: (77)  CA signer not available for verification

The RootCA is not same as Letsencrtypt, but I have simliar error message. I installed libwolfssl4.8.1 but issue persists.

Any suggestions?

This seems different. curl+OpenSSL doesn't like it either. curl on Linux fails too.

curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

After looking at it, this is an issue with how the site is configured. It is presenting just the certificate and not the complete chain. If it's your server, append the intermediate and root certificates to the site certificate. That should fix it.

1 Like

If it's a cert change, surely it's a case of updating the appropriate cert package (presumably ca-certificates), not sure where the need for recompilation is coming from?

This is about your server's cert that has been obtained from Letsencrypt. As their old global root certificate expired, some old server certs can not be validated any more as the path to a trusted root certificate has been broken. And some of their semi-recent certs with their "co-sign with a newer root" transition scheme have backfired with certain client SSL libraries, most prominently with wolfssl (or openssl before 1.1).

Those SSL libraries need to be compiled with support for the more complex signing schemes or replaced by other libraries with the required capabilities, or alternatively the complex server certificates edited to be simpler or replaced with cert from other providers.

Updating ca-certificates will not help. Recent versions of it have contained all needed root certs.

Thanks. It seems the server doesn't send the full chain of certificate. I don't own the server.
At first, wget has simliar error message. I manually imported the intermediate certificate into openwrt, then wget worked.
But curl sitll failed. It seems it's different issue with this thread. I'll open a new one.

Yes, it is a different issue, but root cause is still the same library. It would seem wolfSSL cannot:

  • validate a chain if intermediate certificate is not provided (somehow you made wget do that, maybe on OpenSSL)
  • validate letsencrypt cross signed root certificate

The only short term options I see for your problem is that you build your own curl/libcurl packages with OpenSSL, or modify your code to use wget instead.

1 Like

Okay, I just ask as in a couple of cases (for other software NZB being one), I was able to fix things by pulling down a new cert.pem file from the curl homepage (which is ultimately the one put together by the mozilla project), and looking at the chain for a cert that formerly failed and now works it had an identical set of certs signing it. I guess though that a more recent version of curl was being used.

This bloody curl has got me dead in the water. I need to give my vpn a new public key and the only way their instructions show to do this is with curl which returns the error 77 about the CA signer not available. So I am now paying for a VPN I cannot use. This is bad I am now naked to the internet while trying to remedy this!

FYI this is what mullvad asks me to do after generating the private and public keys:
`curl https://api.mullvad.net/wg/ -d account=YOURMULLVADACCOUNTNUMBER --data-urlencode pubkey=YOURPUBLICKEY

But of course I get error 77 as I said. Is there a alternative that does not involve compiling software? That is out of my league.

Edit: Saved by a downloadable configuration file! yes! Got my vpn back but the curl issue is still there.

I've just solved this literally minutes ago without compiling. The workaround for wolfssl was backported to 21.02.0 and made available as a package ~3 days ago. Here's how you update:

  • Go to System -> Software
  • Hit Update lists, wait for list update to complete
  • Switch to the Updates tab. There should be an update available for libwolfssl4.7.0.66253b90 to version 4.7.0-stable-3
  • Click on Upgrade to install update

I was able to use curl with Lets Encrypt server certificates afterwards.

3 Likes

I installed reinstalled OpenWRT last night before I posted and redid everything from scratch. I updated all packages 12 ago 910-22@5am) and libwolfssl4.7.0.66253b90 was not there but none the less you are correct the updated libwolfxxxxx is there now and updating it fixes curl with respect to mullvad vpn so thank you to you and everyone who fixed this I know it was not just for me none the less credit is due to those who got the ball going and whose who did the work. Thank You. I am not the thread starter but as far as I am concerned this could be marked solved.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.