# opkg update
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/targets/x86/64/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/base/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/luci/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/packages/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/routing/Packages.gz
*** Failed to download the package list from https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/telephony/Packages.gz
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/targets/x86/64/packages/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/base/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/luci/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/packages/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/routing/Packages.gz, wget returned 5.
* opkg_download: Failed to download https://downloads.openwrt.org/releases/21.02.0/packages/x86_64/telephony/Packages.gz, wget returned 5.
If I try to download one of the files myself using wget, I get this error:
It seems to be using uclient-fetch as wget, libustream-wolfssl20201210 (/lib/libustream-ssl.so) is installed, as is ca-bundle (/etc/ssl/certs/ca-certificates.crt).
And it's not that it can't download from any https site: https://google.com and https://microsoft.com work. However, https://kernel.org also says "Invalid SSL certificate". Perhaps it's Let's Encrypt certificates that it's having problems with? https://www.ssllabs.com/ssltest/analyze.html?d=downloads.openwrt.org&s=126.96.36.199 shows two certification paths, one of which has an expired root CA certificate (DST Root CA X3 expired on 2021-09-30, which is today). However, there's another path rooted at ISRG Root X1, which is still valid. And the ca-certificates.crt file seems to have the ISRG Root X1 certificate in it, so I don't know why it's not being used. Maybe I'll try removing expired DST Root CA X3 cert from ca-certificates.crt and see if that fixes things. (opkg was working fine a few weeks ago; it seems very likely that the DST Root expiration is causing this)
Edit /etc/opkg/distfeeds.conf changing the URLs from https to http opkg install libopenssl opkg install openssl-util opkg --force-depends remove libustream-wolfssl20201210 opkg install libustream-openssl20201210
Edit /etc/opkg/distfeeds.conf changing the URLs back to https opkg update now works service uhttpd restart
Oops, sorry, yes, it should be install, not --install. I had typoed the command the first time, and accidentally copied that one instead of the correct one.
Did you change the distfeed.conf URLs to http? The previous opkg installs will work with --no-check-certificate, but once you remove libustream-wolfssl20201210, there won't be any SSL library installed and it won't be able to download from an https site no matter what. Changing the download URL to http will work around that (or downloading the package on another machine and copying it over).
For opkg and downloads.openwrt.org specifically, seems like the server admins could change the certificate chain to the one that doesn't include the expired DST root. From what I read, the reason Let's Encrypt sends the one that includes DST by default is because they worked out a deal with one of the certificate authorities to keep old versions of Android working for a little longer. But I doubt if downloading packages from old versions of Android is a priority over letting recent versions of OpenWRT download packages.
For now I ended up simply removing the LE-shipped cross-signed ISRG Root X1 from the ACME client generated chain and that appears to make the server compatible with uclient-fetch in OpenWrt 21.02.0 by forcing it to fall back to the system ISRG Root.
Could you please provide some details on how to do that, or where that should be located? On my box I tried moving /etc/ssl/certs/ISRG_Root_X1.pem to another location and rebooting, but that did not change anything.
This is supported by recent versions of the acme.sh client at least, using this option:
--preferred-chain <chain> If the CA offers multiple certificate chains, prefer the chain with an issuer matching this Subject Common Name.
If no match, the default offered chain will be used. (default: empty)
The value is "ISRG Root X1"` base64 encoded between __ACME_BASE64__START_ and __ACME_BASE64__END_.
I did this for some of my Let's Encrypt certificates (use quite a few of them both private and for work), but left the majority at the default chain. Haven't really heard of any issues. Most clients seem to ignore the intermediate signed by the expired root just fine. Wolfssl is the exception. But most of us have few such clients, if any.
I tried that, it didn't solve the problem in my particular case - maybe it is a ACME client specific (acme.sh) issue though.
In particular, the acme.sh client produces a full certificate chain PEM file which includes a cross-signed ISRG Root X1 certificate referencing the expired DST Root CA X3 at the end. Removing that one using a text editor form the fullchain.pem file and reloading nginx made things work. The server will not send a cross-signed ISRG Root X1 anymore, forcing TLS clients to verify the intermediate R3 using their non-cross-signed system-installed ISRG Root X1.
Neither --preferred-chain "ISRG Root X1" nor --preferred-chain "DST Root CA X3" will produce a fullchain.pem that does not include a cross-signed ISRG Root X1 at the end.
Will try to spend some more time later to debug acme.sh in order to see if there's a simple way to sneak in a different cert.
Strange. Using --preferred-chain "ISRG Root X1" works for me. It produces a fullchain.pem file with just two certificates: The server certificate and the signing intermediate "C=US, O=Let's Encrypt, CN=R3", signed by "ISRG Root X1". Nothing else. The cross signed "ISRG Root X1" certificate is not included.
Yes, it works for me too, now. My mistake was trying to switch the preferred chain for a certificate update, I had to re-issue the certificate completely instead of updating it (in acme.sh terms) for the setting to become effective.