21.02.0 Cannot connect to any letsencrypt site using ssl (Letsencrypt global root cert expiration)

I'm trying to build an image for a gl.inet mt300n v2 but keep running into problems with ssl.

Using the 21.02.0 and curl toward a letsencrypt site, I get;
curl: (77) CA signer not available for verification

Using 18.06.2 curl gives this error;
curl: (51) Cert verify failed: BADCERT_EXPIRED

I know that letsencrypt changed its root CA today but why is this happening only with openwrt curl? I've tested the same sites from Linux and Windows and they work fine. Only openwrt is not working when using curl.

1 Like

You can try curl -k but I don't think that will work. Let us know how you go

I am letting you know how it's going, I've been at this all day :).
I cannot get a build to work with any letsencrypt site.

% curl https://shop.bbc.com/
curl: (51) Cert verify failed: BADCERT_EXPIRED
% curl https://help.behance.net/
curl: (51) Cert verify failed: BADCERT_EXPIRED
% curl https://www.pinterest.com/
Works fine.

Seems to me curl is the problem. wget works. Ran a quick test for you:

root@wrt:/tmp# wget https://shop.bbc.com
Downloading 'https://shop.bbc.com'
Connecting to 23.227.38.74:443
Writing to 'index.html'

Download completed (458940 bytes)
root@wrt:/tmp# curl https://shop.bbc.com
curl: (77)  CA signer not available for verification

Turns out curl is linked to wolfssl:

root@WRT:/tmp# ldd /usr/bin/curl
	/lib/ld-musl-armhf.so.1 (0xb6e9e000)
	libcurl.so.4 => /usr/lib/libcurl.so.4 (0xb6e49000)
	libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0xb6e1a000)
	libwolfssl.so.4.7.0.66253b90 => /usr/lib/libwolfssl.so.4.7.0.66253b90 (0xb6ca4000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0xb6c88000)
	libc.so => /lib/ld-musl-armhf.so.1 (0xb6e9e000)

Since your build is including curl, it pulls wolfssl in.

Try rebuilding curl to use openssl instead.

Like @apccv says, it is possible to build curl to use openssl (like I do for my builds), but that requires using the full toolchain so that you can configure build options for curl.

Imagebuilder that @LarryPowel uses, will not work, as it has only the default curl with wolfssl.

Ah, now we're getting somewhere. Yes, I (and many others) am using Imagebuilder so what options do we have? Will curl get updated in the packages so that it works for us non source folks?

Well, curl has been for ages that way, SSL configurable via build options, but no formal variants.

It's current maintainer is @stangri , so he might consider creating variants for curl, so that both wolfssl and openssl variants would be built, and could be used by the imagebuilder.

Imagebuilder has been a minor quick and dirty tool for years, but it has lately got more popularity. But it is still, and will be, quite limited regarding package tailoring possibilities. Full toolchain is for that, and is actually quite easy to use ( if you are skillful enough to use imagebuilder).

I don't know of any options short term.

Building from scratch is not that bad. This guide helped me a lot. If you attempt it, I found that building the toolkit first with no changes works better, and tweaking packages afterwards has better success ratio.

I suggest you give it a try. I would keep using imagebuilder for the image, and provided you have space left, install a custom-built curl/libcurl with openSSL on top. You could leave all your devices as is and patch the with the rebuilt curl packages.

I did use source for a while but found that image builder was all I needed since I wasn't messing with the code, just picking which packages I wanted. Trying to keep track of the options and packages using source was frustrating since I would build something, come back six months later and have to re-learn it every time. With IB, I simply wrote a small script that contains my list of packages and other options so was done in seconds.

I hope someone updates curl and the SSL certs for these newer versions otherwise, many that don't want to get into using source will quietly go away in frustration.

Products like this should not be only for those that know how to write it, it should be for those that want to put it to use as well. I hope some of the developers believe this.

I hear you. Code tinkering is not for everyone. I'm sure that for most people just flashing the vanilla image is quite a leap of faith akin to landing a rover in Mars. It was for me, at least, when I started with this.

Let's not forget that OpenWRT is much larger than the code managed by the community. An issue in wolfSSL's library (and the maybe outdated version used by OpenWRT) is the problem here. OpenWRT just takes that library and packages it. I'm sure once the upstream library is updated or made available we will see new wolfSSL libraries for 21.02.x. Until then, I really appreciate having the possibility to fix the problem instead of having to wait for a vendor firmware update.

The decision to use wolfSS was probably as tough as any other. Just a few years ago openSSL was having security issues left, right and centre. I'm sure wolfSSL tests the life out of their code, but it's impossible to catch all issues. Without knowing all the details, the way letsencrypt is signing their certificates looks like quite a creative monkey wrench that requires very recent SSL implementations.

Anyways, just to say it's complicated. IMO what makes OpenWRT so special is that, if you cannot wait for a "vendor" to update the firmware, you always have the option of fixing it yourself.

2 Likes

So in my case for example, I don't care what I use for SSL, so long as it's current and working with all SSL providers. I don't use Luci and the only requirement for SSL is curl using https and sometimes, using an openvpn tunnel.
With that in mind, what is the problem that I'm facing, is it using openssl vs wolfSS or is it the ca-bundle or ca-certificates? Knowing this, I have to assume there is a way for me to move forward using IB.

I've built a 21.02.0 version with the current tools using IB.

% opkg list-installed | grep -E "wget|ssl|cert"
libopenssl - 1.0.2u-1
libustream-openssl - 2018-07-30-23a3f283-2
openvpn-openssl - 2.4.5-4.2
wget - 1.19.5-6

It still won't connect to LE protected sites but does to any other sites.
Is this actually a curl problem or the ca-bundle/ca-certificates that need updating?

Sadly, creating variants of libcurl is not easy. I've raised an issue to make curl dependent on the mbedtls/openssl by default because of previously reported issues like this and that.

If OP wants to create another issue and link https://github.com/openwrt/packages/issues/16674 from it, it may be helpful.

@hnyman, do you know why wpad by default depends on wolfssl and not mbedtls?

If I remember right, wpa3 compatibility was possible only with openssl and wolfssl a while ago. Mbedtls failed in something.

Discussion around here
https://www.mail-archive.com/openwrt-devel@lists.openwrt.org/msg54028.html

http://lists.openwrt.org/pipermail/openwrt-devel/2020-July/030433.html

1 Like

You need openSSL 1.1 or later.

v1.0 is as good as wolfSSL

What I would do is leave your image as is (21.02.0 with libopenssl 1.1+, and wolfSSL a is the default, and without curl). Use the toolchain to compile curl and libcurl, linking those two to libopenssl. Then opkg install curl and libcurl.

One problem after another.
Curl on 21.02 is fine so long as GoDaddy doesn't change it's SSL root.
Now I find a new problem. An old program that someone wrote, worked on up to 19.x but now gives this.
It's used to regularly ping a device. Pretty simple really but...

Error relocating /bin/pingdev: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE10_M_replaceEjjPKcj: symbol not found
Error relocating /bin/pingdev: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_mutateEjjPKcj: symbol not found
Error relocating /bin/pingdev: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev: symbol not found
Error relocating /bin/pingdev: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_assignERKS4_: symbol not found
Error relocating /bin/pingdev: _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE4findEcj: symbol not found
Error relocating /bin/pingdev: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_appendEPKcj: symbol not found
Error relocating /bin/pingdev: _ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE7compareEPKc: symbol not found
Error relocating /bin/pingdev: _ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE9_M_createERjj: symbol not found

Does anyone know what this means, what I should be looking for, maybe a package, something in the new version. I hope for a lead so I can contact someone about fixing it.

Missing library.

/bin/pingdev is the program?

do: ldd /bin/pingdev

Maybe it will show you what it's looking for.

Yeah, but if you are using openssl 1.0 you are as good as using wolfssl. wolfssl will work with GoDaddy's root.

Besides, v1.0.2u has several vulnerabilities. Here. This one looks like fun. (I did a lot of emergency patching when the heartbleed vulnerabilities were found... my love for openSSL goes way back.) Maybe they don't apply to your use case. Regardless... I don't see the point of going through the hassle of building an image with openSSL that would give you what you get out of wolfSSL as is.

I'd rebuild curl and libcurl so that libcurl is bound to openssl and so that curl is properly linked to libcurl. You end up with your existing wolfssl image, but you opkg install 2 packages on top.

That's what I'm doing to the routers I'm having issues with... and then I can stop using those 2 packages and openssl once wolfssl is updated.

% ldd /bin/pingdev
/lib/ld-musl-mipsel-sf.so.1 (0x77dfc000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x77cfd000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77cd9000)
libc.so => /lib/ld-musl-mipsel-sf.so.1 (0x77dfc000)

That's the fanciest thing there. Should be provided by ibstdcpp6 which you probably already have.

Dunno. Probably the code was linked to older versions of libraries. I'd try pilfer an older version of ibstdcpp from 19.07.8 but probably it's not going to work. This one no clue, sorry.

1 Like