# opkg update
Downloading https://downloads.openwrt.org/releases/21.02.0/targets/x86/64/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/targets/x86/64/packages/Packages.gz
Downloading https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/base/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/base/Packages.gz
Downloading https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/luci/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/luci/Packages.gz
Downloading https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/packages/Packages.gz
Downloading https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/routing/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/routing/Packages.gz
Downloading https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/telephony/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/telephony/Packages.gz
Collected errors:
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/targets/x86/64/packages/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/base/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/luci/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/packages/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/routing/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/telephony/Packages.gz, wget returned 5.
If I try to download one of the files myself using wget, I get this error:
It seems to be using uclient-fetch as wget, libustream-wolfssl20201210 (/lib/libustream-ssl.so) is installed, as is ca-bundle (/etc/ssl/certs/ca-certificates.crt).
And it's not that it can't download from any https site: https://google.com and https://microsoft.com work. However, https://kernel.org also says "Invalid SSL certificate". Perhaps it's Let's Encrypt certificates that it's having problems with? https://www.ssllabs.com/ssltest/analyze.html?d=downloads.openwrt.org&s=168.119.138.211 shows two certification paths, one of which has an expired root CA certificate (DST Root CA X3 expired on 2021-09-30, which is today). However, there's another path rooted at ISRG Root X1, which is still valid. And the ca-certificates.crt file seems to have the ISRG Root X1 certificate in it, so I don't know why it's not being used. Maybe I'll try removing expired DST Root CA X3 cert from ca-certificates.crt and see if that fixes things. (opkg was working fine a few weeks ago; it seems very likely that the DST Root expiration is causing this)
That didn't help; exact same error. While I know the basics about SSL certificates, I don't know the details of how certificate chains are validated. https://letsencrypt.org/2020/12/21/extending-android-compatibility.html shows the default chain that Let's Encrypt issues, which is rooted at the now-expired DST certificate. And it looks like that's the chain that downloads.openwrt.org is sending.
The description of WOLFSSL_ALT_CERT_CHAIN at https://www.wolfssl.com/docs/wolfssl-manual/ch2/ says "Default wolfSSL behavior is to require validation of all presented peer certificates." Does that mean that since the server is sending the expired DST cert, wolfSSL is going to want to validate it? Someone at https://www.wolfssl.com/forums/post6232.html#p6232 suggests that the WOLFSSL_ALT_CERT_CHAIN behavior is actually recommended, and should be enabled by default.
Anyways, maybe switching from wolfSSL to OpenSSL would fix this, but that seems tricky to do.
As you have installed also openssl, switching to that is pretty easy:
just remove libustream-wolfssl and install libustream-openssl to get openssl-based SSL for uclient-fetch, LuCI, wget etc.
Edit /etc/opkg/distfeeds.conf changing the URLs from https to http opkg install libopenssl opkg install openssl-util opkg --force-depends remove libustream-wolfssl20201210 opkg install libustream-openssl20201210
Edit /etc/opkg/distfeeds.conf changing the URLs back to https opkg update now works service uhttpd restart
opkg --install libustream-openssl20201210
You mean install not --install ?
didn't work either.
I had to manually download the package and install it.
Now opkg update works indeed.
Oops, sorry, yes, it should be install, not --install. I had typoed the command the first time, and accidentally copied that one instead of the correct one.
Did you change the distfeed.conf URLs to http? The previous opkg installs will work with --no-check-certificate, but once you remove libustream-wolfssl20201210, there won't be any SSL library installed and it won't be able to download from an https site no matter what. Changing the download URL to http will work around that (or downloading the package on another machine and copying it over).
I understood, seems obvious fast typo error
Yes I changed distfeed.conf to http
I manually installed libustream-openssl20201210
changed back distfeed.conf to https
Now it works.
I'm on a x64 router.
I have read about a certificate issue for today, but it seems related to old stuff. It never occured to me that OpenWrt will be impacted
The globally most painful part is probably the openssl 1.0.2 incompatibility, as that concerns lots of semi-new devices and software. openssl 1.1 is rather new (as a widely used version)
That will cause problems for lots of letsencrypt signed sites.
For opkg and downloads.openwrt.org specifically, seems like the server admins could change the certificate chain to the one that doesn't include the expired DST root. From what I read, the reason Let's Encrypt sends the one that includes DST by default is because they worked out a deal with one of the certificate authorities to keep old versions of Android working for a little longer. But I doubt if downloading packages from old versions of Android is a priority over letting recent versions of OpenWRT download packages.
That is exactly what I am trying to do but information for this is rather scarce... there's mumbling about requesting a different preferred chain in recent ACME clients but I couldn't find any concrete example. Specifically I am seeking to implement Workaround #3 as mentioned in https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
For now I ended up simply removing the LE-shipped cross-signed ISRG Root X1 from the ACME client generated chain and that appears to make the server compatible with uclient-fetch in OpenWrt 21.02.0 by forcing it to fall back to the system ISRG Root.
Could you please provide some details on how to do that, or where that should be located? On my box I tried moving /etc/ssl/certs/ISRG_Root_X1.pem to another location and rebooting, but that did not change anything.
This is supported by recent versions of the acme.sh client at least, using this option:
--preferred-chain <chain> If the CA offers multiple certificate chains, prefer the chain with an issuer matching this Subject Common Name.
If no match, the default offered chain will be used. (default: empty)
See: https://github.com/acmesh-official/acme.sh/wiki/Preferred-Chain
Not sure there is an easy option changing an existing config. You can force it by editing the config file, adding a new variable. But the value format is pretty obscure:
The value is "ISRG Root X1"` base64 encoded between __ACME_BASE64__START_ and __ACME_BASE64__END_.
I did this for some of my Let's Encrypt certificates (use quite a few of them both private and for work), but left the majority at the default chain. Haven't really heard of any issues. Most clients seem to ignore the intermediate signed by the expired root just fine. Wolfssl is the exception. But most of us have few such clients, if any.
I tried that, it didn't solve the problem in my particular case - maybe it is a ACME client specific (acme.sh) issue though.
In particular, the acme.sh client produces a full certificate chain PEM file which includes a cross-signed ISRG Root X1 certificate referencing the expired DST Root CA X3 at the end. Removing that one using a text editor form the fullchain.pem file and reloading nginx made things work. The server will not send a cross-signed ISRG Root X1 anymore, forcing TLS clients to verify the intermediate R3 using their non-cross-signed system-installed ISRG Root X1.
Neither --preferred-chain "ISRG Root X1" nor --preferred-chain "DST Root CA X3" will produce a fullchain.pem that does not include a cross-signed ISRG Root X1 at the end.
Will try to spend some more time later to debug acme.sh in order to see if there's a simple way to sneak in a different cert.
Strange. Using --preferred-chain "ISRG Root X1" works for me. It produces a fullchain.pem file with just two certificates: The server certificate and the signing intermediate "C=US, O=Let's Encrypt, CN=R3", signed by "ISRG Root X1". Nothing else. The cross signed "ISRG Root X1" certificate is not included.
Yes, it works for me too, now. My mistake was trying to switch the preferred chain for a certificate update, I had to re-issue the certificate completely instead of updating it (in acme.sh terms) for the setting to become effective.
