Opkg upgrade broke uhttpd SSL

I ran an upgrade of packages on my router @ 17.01.4. It upgraded these packages:

root@LEDE:~# opkg update
Downloading http://downloads.lede-project.org/releases/17.01.4/targets/ipq806x/generic/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_core
Downloading http://downloads.lede-project.org/releases/17.01.4/targets/ipq806x/generic/packages/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/base/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_base
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/base/Packages.sig
oSignature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/luci/Packages.gz
pkg Updated list of available packages in /var/opkg-lists/reboot_luci
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/luci/Packages.sig
list-Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/packages/Packages.gz
upgradable
Updated list of available packages in /var/opkg-lists/reboot_packages
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/packages/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/routing/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_routing
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/routing/Packages.sig
Signature check passed.
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/telephony/Packages.gz
Updated list of available packages in /var/opkg-lists/reboot_telephony
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/telephony/Packages.sig
Signature check passed.
root@LEDE:~# opkg list-upgradable
openssl-util - 1.0.2n-1 - 1.0.2o-1
libmbedtls - 2.7.0-1 - 2.7.2-1
libopenssl - 1.0.2n-1 - 1.0.2o-1
root@LEDE:~# ./do-upgrades.sh 
+ + + xargs opkg upgrade
opkg list-upgradable
awk {print $1;}
Upgrading openssl-util on root from 1.0.2n-1 to 1.0.2o-1...
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/base/openssl-util_1.0.2o-1_arm_cortex-a15_neon-vfpv4.ipk
Upgrading libmbedtls on root from 2.7.0-1 to 2.7.2-1...
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/base/libmbedtls_2.7.2-1_arm_cortex-a15_neon-vfpv4.ipk
Removing obsolete file /usr/lib/libmbedx509.so.2.7.0.
Removing obsolete file /usr/lib/libmbedtls.so.2.7.0.
Removing obsolete file /usr/lib/libmbedcrypto.so.2.7.0.
Removing obsolete file /usr/lib/libmbedcrypto.so.1.
Upgrading libopenssl on root from 1.0.2n-1 to 1.0.2o-1...
Downloading http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/base/libopenssl_1.0.2o-1_arm_cortex-a15_neon-vfpv4.ipk
Configuring libopenssl.
Configuring openssl-util.
Configuring libmbedtls.
root@LEDE:~# exit

After which, restarting uhttpd resulted in a failure to restart. If I disable https in /etc/config/uhttpd, it will work again (but that's not what I want). Any idea what about the change in libmbedtls broke uhttpd?

root@LEDE:/etc/config# opkg list-installed
ath10k-firmware-qca9984 - 2017-01-11-ab432c60-1
avahi-dbus-daemon - 0.6.32-1
avahi-utils - 0.6.32-1
base-files - 173.1-r3560-79f57e422d
block-mount - 2017-06-30-bdcb075f-1
busybox - 1.25.1-4
cfdisk - 2.29.2-1
cgdisk - 1.0.1-1
dbus - 1.10.4-1
dnsmasq - 2.78-6
dosfstools - 4.0-2
dropbear - 2017.75-4
e2fsprogs - 1.43.3-2
fdisk - 2.29.2-1
firewall - 2017-05-27-a4d98aea-1
fstools - 2017-06-30-bdcb075f-1
fwtool - 1
gdisk - 1.0.1-1
hdparm - 9.50-1
hostapd-common - 2016-12-19-ad02e79d-7
ip6tables - 1.4.21-3
iptables - 1.4.21-3
iptables-mod-conntrack-extra - 1.4.21-3
iptables-mod-ipopt - 1.4.21-3
iw - 4.9-1
iwinfo - 2016-09-21-fd9e17be-1
jshn - 2018-01-07-1dafcd78-1
jsonfilter - 2016-07-02-dea067ad-1
kernel - 4.4.92-1-8cece9aa696af7c1e88bf73a61f5f82a
kmod-ata-ahci - 4.4.92-1
kmod-ata-ahci-platform - 4.4.92-1
kmod-ata-core - 4.4.92-1
kmod-ath - 4.4.92+2017-01-31-3
kmod-ath10k - 4.4.92+2017-01-31-3
kmod-cfg80211 - 4.4.92+2017-01-31-3
kmod-crypto-crc32c - 4.4.92-1
kmod-crypto-hash - 4.4.92-1
kmod-fs-ext4 - 4.4.92-1
kmod-fs-hfs - 4.4.92-1
kmod-fs-hfsplus - 4.4.92-1
kmod-fs-vfat - 4.4.92-1
kmod-gpio-button-hotplug - 4.4.92-2
kmod-ifb - 4.4.92-1
kmod-input-core - 4.4.92-1
kmod-ip6tables - 4.4.92-1
kmod-ipt-conntrack - 4.4.92-1
kmod-ipt-conntrack-extra - 4.4.92-1
kmod-ipt-core - 4.4.92-1
kmod-ipt-ipopt - 4.4.92-1
kmod-ipt-nat - 4.4.92-1
kmod-leds-gpio - 4.4.92-1
kmod-lib-crc-ccitt - 4.4.92-1
kmod-lib-crc16 - 4.4.92-1
kmod-mac80211 - 4.4.92+2017-01-31-3
kmod-nf-conntrack - 4.4.92-1
kmod-nf-conntrack6 - 4.4.92-1
kmod-nf-ipt - 4.4.92-1
kmod-nf-ipt6 - 4.4.92-1
kmod-nf-nat - 4.4.92-1
kmod-nls-base - 4.4.92-1
kmod-nls-cp437 - 4.4.92-1
kmod-nls-iso8859-1 - 4.4.92-1
kmod-nls-utf8 - 4.4.92-1
kmod-ppp - 4.4.92-1
kmod-pppoe - 4.4.92-1
kmod-pppox - 4.4.92-1
kmod-sched-cake - 4.4.92+2017-01-28-9789742c-1
kmod-sched-connmark - 4.4.92-1
kmod-sched-core - 4.4.92-1
kmod-scsi-core - 4.4.92-1
kmod-slhc - 4.4.92-1
kmod-sound-core - 4.4.92-1
kmod-usb-audio - 4.4.92-1
kmod-usb-core - 4.4.92-1
kmod-usb-dwc3 - 4.4.92-1
kmod-usb-dwc3-of-simple - 4.4.92-1
kmod-usb-ledtrig-usbport - 4.4.92-1
kmod-usb-ohci - 4.4.92-1
kmod-usb-phy-qcom-dwc3 - 4.4.92-1
kmod-usb-storage - 4.4.92-1
kmod-usb2 - 4.4.92-1
kmod-usb3 - 4.4.92-1
lede-keyring - 2017-01-20-a50b7529-1
libattr - 20160302-1
libavahi-client - 0.6.32-1
libavahi-dbus-support - 0.6.32-1
libblkid - 2.29.2-1
libblobmsg-json - 2018-01-07-1dafcd78-1
libc - 1.1.16-1
libdaemon - 0.14-5
libdb47 - 4.7.25.4.NC-5
libdbus - 1.10.4-1
libexpat - 2.2.0-1
libext2fs - 1.43.3-2
libfdisk - 2.29.2-1
libgcc - 5.4.0-1
libgcrypt - 1.6.6-1
libgdbm - 1.11-1
libgpg-error - 1.12-1
libip4tc - 1.4.21-3
libip6tc - 1.4.21-3
libiwinfo - 2016-09-21-fd9e17be-1
libiwinfo-lua - 2016-09-21-fd9e17be-1
libjson-c - 0.12.1-1
libjson-script - 2018-01-07-1dafcd78-1
liblua - 5.1.5-1
libmbedtls - 2.7.2-1
libmount - 2.29.2-1
libncurses - 6.0-1
libnl-tiny - 0.1-5
libopenssl - 1.0.2o-1
libpcap - 1.8.1-1
libpopt - 1.16-1
libpthread - 1.1.16-1
librpc - 2015-11-04-a921e3de-1
librt - 1.1.16-1
libsmartcols - 2.29.2-1
libstdcpp - 5.4.0-1
libubox - 2018-01-07-1dafcd78-1
libubus - 2017-02-18-34c6e818-1
libubus-lua - 2017-02-18-34c6e818-1
libuci - 2018-01-01-141b64ef-1
libuci-lua - 2018-01-01-141b64ef-1
libuclient - 2017-11-02-4b87d831-1
libusb-1.0 - 1.0.21-1
libustream-mbedtls - 2016-07-02-ec80adaa-3
libuuid - 2.29.2-1
libxml2 - 2.9.4-1
libxtables - 1.4.21-3
logd - 2017-03-10-16f7e161-1
lua - 5.1.5-1
luci - git-18.061.17832-d092772-1
luci-app-firewall - git-18.061.17832-d092772-1
luci-app-sqm - 1.1.3-2
luci-app-uhttpd - 1.0.0-1
luci-base - git-18.061.17832-d092772-1
luci-lib-ip - git-18.061.17832-d092772-1
luci-lib-jsonc - git-18.061.17832-d092772-1
luci-lib-nixio - git-18.061.17832-d092772-1
luci-mod-admin-full - git-18.061.17832-d092772-1
luci-proto-ipv6 - git-18.061.17832-d092772-1
luci-proto-ppp - git-18.061.17832-d092772-1
luci-ssl - git-18.061.17832-d092772-1
luci-theme-bootstrap - git-18.061.17832-d092772-1
mdns-utils - 576.30.4-1
mdnsd - 576.30.4-1
mdnsresponder - 576.30.4-1
mtd - 21
nano - 2.7.5-1
netatalk - 3.1.10-1
netifd - 2017-01-25-650758b1-1
odhcp6c - 2017-01-30-c13b6a05-2
odhcpd - 2018-03-02-2da5850f-3
opkg - 2017-03-23-1d0263bb-1
ppp - 2.4.7-12
ppp-mod-pppoe - 2.4.7-12
procd - 2017-08-08-66be6a23-1
procd-nand - 2017-08-08-66be6a23-1
px5g-mbedtls - 5
rpcd - 2017-12-07-cfe1e75c-1
sfdisk - 2.29.2-1
shadow-common - 4.2.1-5
shadow-groupadd - 4.2.1-5
shadow-groupmod - 4.2.1-5
shadow-useradd - 4.2.1-5
shadow-usermod - 4.2.1-5
sqm-scripts - 1.1.3-2
swconfig - 11
tc - 4.4.0-10
tcpdump - 4.9.2-1
terminfo - 6.0-1
ubi-utils - 1.5.2-1
uboot-envtools - 2015.10-1
ubox - 2017-03-10-16f7e161-1
ubus - 2017-02-18-34c6e818-1
ubusd - 2017-02-18-34c6e818-1
uci - 2018-01-01-141b64ef-1
uclient-fetch - 2017-11-02-4b87d831-1
uhttpd - 2017-11-04-a235636a-1
uhttpd-mod-ubus - 2017-11-04-a235636a-1
usbreset - 4
usbutils - 007-7
usign - 2015-07-04-ef641914-1
wpad-mini - 2016-12-19-ad02e79d-7
zlib - 1.2.11-1
root@LEDE:/etc/config#

OK, I finally found a log file after bringing up SSL-less uhttpd:

Mon Apr  2 20:41:06 2018 daemon.err uhttpd[1720]: Failed to load ustream-ssl library: Error loading shared library libmbedcrypto.so.1: No such file or directory (needed by /lib/libustream-ssl.so)

Looks like the repos need an update for libustream-mbedtls? My system has version 2016-07-02-ec80adaa-3
Or maybe the repo for this architecture has something odd in the package names, opkg thinks it's not new but I see a new one (dated April 2) on this page:
http://downloads.lede-project.org/releases/17.01.4/packages/arm_cortex-a15_neon-vfpv4/base/

Please post all code, config, and log output within code brackets.

I think that's it right there -- something that wasn't upgraded links to a now-obsolete library. opkg isn't apt in getting dependencies worked out just right.

opkg --force-reinstall <a list of the packages that depend on TLS>

might get you running again.

Indeed, I was able to get going again by reinstalling libustream-mbedtls.
What seems to be the root problem here is that a new version of libustream-mbedtls was posted for this architecture, with a different library name inside, but the exact same package name as the previous package, namely:
libustream-mbedtls_2016-07-02-ec80adaa-3_arm_cortex-a15_neon-vfpv4.ipk
Should this have a newer version name, so that opkg knows it's a newer version?

Same problem. But in my case the upgrade of uhttpd broke the access to luci web via http:. I have to execute mv uhttpd-opkg uhttpd.
The libmbedcrypto.so exists but with other names:

$ ls -1 /usr/lib/libmbedcrypto*
/usr/lib/libmbedcrypto.so
/usr/lib/libmbedcrypto.so.2
/usr/lib/libmbedcrypto.so.2.7.2

But /usr/sbin/px5g try to access to libmbedcrypto.so.1.

I've solved it with: opkg install --force-reinstall px5g-mbedtls

curl is also broken:

# curl
Error loading shared library libmbedcrypto.so.1: No such file or directory (needed by /usr/lib/libcurl.so.4)

I wonder if core devs should bump the libustream version after SSL lib upgrades to more clearly indicate that libustream-XXX needs to be also reinstalled so that it links to the correct underlying SSL lib version.

The problem does not materialize if you build a new firmware with all new packages, but it may surface if you use opkg to detect packages for upgrading (and then the need for upgrading ustream-XXX is not noticed as its version has not changed). Onesolution might be to somehow include the reference to the version/commit/date of the underlying SSL lib in the ustream-XXX version.

cc @jow @nbd

Found this:
https://github.com/openwrt/packages/issues/5871

libmbedtls is updated with "opkg update"
So package: curl ustream-ssl openvpn px5g need update too.

https://github.com/openwrt/openwrt/pull/852

Temporary solution: opkg install libmbedtls_[oldversion].ipk --force-downgrade

Creating a symlink as indicated by the github issue seems to fix this.

ln -s /usr/lib/libmbedcrypto.so.2.7.2 /usr/lib/libmbedcrypto.so.1

Not sure if it will cause problems when fix is released.

1 Like

Hi!
The current snapshot (lib) seems to resolve this problem. Thanx!

I stumbled on same problem while trying to work around "PRNG missing" error in curl_7.52 when I was trying to push a file on server using curl_7.52 from LEDE 17.01.4 ImageBuilder.
Trying packages curl_7.54 and curl_7.59 firmware came up with curl missing /usr/lib/libmbedcrypto.so.1.

Making a firmware I have resolved this problem by using libmbedtls_2.8.0-1_mips_24kc.ipk package from current snapshot:

root@HRmeteohub:~# ls -al /usr/lib/libmbedcrypto*
lrwxrwxrwx 1 root root 18 Apr 10 14:14 /usr/lib/libmbedcrypto.so -> libmbedcrypto.so.1
lrwxrwxrwx 1 root root 22 Apr 10 14:14 /usr/lib/libmbedcrypto.so.1 -> libmbedcrypto.so.2.8.0
-rwxr-xr-x 1 root root 218356 Apr 10 01:57 /usr/lib/libmbedcrypto.so.2.8.0

I really can't say what is better, making a symlink and using old library, or simply going for new package.

is, in my opinion, the best. The symlink "assumes" that the new version is backward-compatible with the old, which may not be the case. Better to know another bit of code requires the older version so that code can be identified and updated, at least the way I think.

This is a dependency problem with opkg in that it didn't properly identify that the change to the library should have forced updates of all the packages that used that library as well. Lots of moving parts in the end-to-end system that occasionally get out of sync. Yet another reason to prefer full-image upgrades over "upgrade everything with opkg".

Well, after install libmbedtls_2.7.2, then
“opkg install --force-reinstall curl libcrul” did fix curl missing /usr/lib/libmbedcrypto.so.1
But after I install libmbedtls_2.8.0
Now curl missing /usr/lib/libmbedcrypto.so.2, and “opkg install --force-reinstall curl libcrul” do not work now.
Creating a symlink to fix this.
ln -s /usr/lib/libmbedcrypto.so.2.8.0 /usr/lib/libmbedcrypto.so.2