18.06.01: after opkg upgrade uhttpd crashes on HTTPS connect

Installing and Using OpenWrt
1

This is on a Buffalo WZR-HP-AG300H:

libmbedtls and uhttpd both had upgrades this morning. After applying both and restarting uhttpd, it crashes when I try to connect with HTTPS:

Thu Jan 31 10:08:53 2019 kern.info kernel: [  293.449255] do_page_fault(): sending SIGSEGV to uhttpd for invalid write access to 77eee0f4
Thu Jan 31 10:08:53 2019 kern.info kernel: [  293.457638] epc = 77f58cc0 in libc.so[77f30000+92000]
Thu Jan 31 10:08:53 2019 kern.info kernel: [  293.462754] ra  = 77f58cac in libc.so[77f30000+92000]

Edit: The upgrades this morning were:

luci-lib-ip - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-theme-bootstrap - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
libmbedtls - 2.12.0-1 - 2.14.1-1
luci-app-firewall - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-ssl - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-proto-ppp - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-mod-admin-full - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-base - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-proto-ipv6 - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
uhttpd - 2018-11-28-cdfc902a-1 - 2018-11-28-cdfc902a-2
luci-lib-nixio - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci-lib-jsonc - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
luci - git-19.020.41695-6f6641d-1 - git-19.030.57138-4c45a52-1
2

One should never try to upgrade all packages with current (v18) or prior OpenWrt releases.

opkg isn't apt or the like and is not designed to do an in-place upgrade of the system at this time.

2 Likes
3

This is strange... you're the 3rd person using okpg upgrade in the last few days.

opkg upgrade should not be used under any normal circumstances. This broke your OpenWrt because of package dependencies, kernel versions, and so on.

See this thread and this one, too.

2 Likes
4

Appreciate your detailed report -- I would guess from your list that the change from mbedTLS v2.12 to v2.14 may be the root of the issues.

Resolution remains to backup up your config robustly, reset to the defaults of your ROM (assuming that it is v18.06.01 or later), restore your config, and go from there. You are unlikely to get the combination of package versions "just right" without backing up to the start.

1 Like
5

The router is working without (apparent) issues ATM and I can SSH in, so for immediate recovery I assume a sysupgrade to the just-released 18.06.02 will work.

If opkg list-upgradable etc isn't the right way to get updates, what's best practice for staying up to date with security patches?

For my (and apparently others' too) enlightenment: If opkg isn't how we upgrade, what's the "target"/point of these newly-available binary builds within a release's stream? Trying to build/fix the big picture here...

6

Flash a new ROM, install from the self-consistent set of packages associated with that ROM.

In the event that there is a single package that needs a security update, then I would imagine that clear instructions to update that specific package would come from a recognized developer/maintainer of OpenWrt (in contrast to a random user of the forum, even if their title is "Regular").

In the last couple years, significant security patches have been promptly released in well, a Release.

7

OK, and again honest question to help me understand: I happened to already have luci and luci-ssl, but what if I hadn't and had only decided to install them today - wouldn't have I picked up the same non-consistent builds? In other words, for packages that aren't baked into the release image, what's the consistent source? I had assumed opkg was it.

Note I haven't touched the opkg configuration (/etc/opkg/*), can they somehow have been changed to point to the wrong source over the sysupgrades through the 17.x and 18.x releases? They seem ok to me:

# cat /etc/opkg/distfeeds.conf 
src/gz openwrt_core http://downloads.openwrt.org/releases/18.06.1/targets/ar71xx/generic/packages
src/gz openwrt_base http://downloads.openwrt.org/releases/18.06.1/packages/mips_24kc/base
src/gz openwrt_luci http://downloads.openwrt.org/releases/18.06.1/packages/mips_24kc/luci
src/gz openwrt_packages http://downloads.openwrt.org/releases/18.06.1/packages/mips_24kc/packages
src/gz openwrt_routing http://downloads.openwrt.org/releases/18.06.1/packages/mips_24kc/routing
src/gz openwrt_telephony http://downloads.openwrt.org/releases/18.06.1/packages/mips_24kc/telephony
8

The reason is likely that opkg does not notice some dependent libraries (like the libustream-mbedtls wrapper library used by luci HTTPS) that have not changed by themselves, although the underlying mbedtls has changed version and requires also a matching recompiled version of the respective ustreamssl lib.

Those have been recompiled for the download repo, but as the version looks identical than the earlier compilation, opkg does not include it in the upgradable list. Reinstalling just the libustream-mbedtls package would have likely fixed your problem.

Since a few days the master branch has more strict library version dependency regime, but that is not in 18.06

Ps. Note also that some packages have not yet been recompiled for all targets. Compiling everything for all targets takes 1-2 days. Also the fact that images and some packages are built on "phase1" buildbot, while most normal packages are built with "phase2" packages buildbot with the SDK created in phase1, can create timing difficulties, which can materialise.
E.g. right now 18.06 ar71xx (mips_24kc) has mbedtls 2.14 download available, while ipq806x (arm_cortex-a15_neon-vfpv4) is still missing that, as the build is still ongoing...

1 Like
9

Let's look at "happy path" first. You've got nothing related to TLS installed.

Each package, at this time, has a list of dependencies, by name. So when you ask to install, say, luci-ssl, it, among other things, has a dependency on libustream_mbedtls, which, among other things, has a dependency on libmbedtls. On the happy path, you've got none of these installed, so you get the full set, built at the same point in time.

Most of the time, this works out just fine. However, as you suspect, there are flaws in not knowing which specific versions are compatible and which are not. In the above, if you already an older version of one of the dependencies, it wouldn't have been re-installed, and might have lead to problems. Generally, newer versions are backward-compatible with older versions for a reasonable period of time (a year is not uncommon), so generally this is not a big issue.

For OSes where the package management system is intended for upgrading the OS, there is usually a replacement of all dependent packages when a package is upgraded, whether needed or not, or a way to specify which versions are compatible and filtering that list.

The "sledgehammer" approach isn't a good one for flash-based systems, between limited flash and flash wear. A package in ROM is often around half the size of one on the overlay. Using that approach would "explode" many of those packages to full size if a common dependency were upgraded. On a flash-based file system, every time you write "a file" you burn up one of the cycles you have. While 10,000 may seem like a lot, they can be used up pretty quickly, eventually rendering the flash (and the device) unreliable to useless.

As @hnyman points out, the OpenWrt project is moving toward a structure that can help mitigate these problems by more specific tracking of dependencies. The groundwork is being laid now, so look toward future releases (no, I don't know the schedule, but I wouldn't hold your breath) for these changes to be applied to the hundreds of packages involved, the dependencies worked out, and the changes to flow into tools like opkg.

1 Like
10

Extending this, continuous in-place upgrades require a lot more effort on the development/ packaging side (easily >10 times more to check for ABI incompatibilities (no, upstream isn't always careful enough), do symbol versioning downstream, etc.), which probably can't be covered by the current developer team) - and it also requires sonames for library packages. Beyond the expected side effect of squashfs+overlay whiteouts, this would also require that multiple sonames can be installed in parallel (because package bar still requires the old libfoo1, while the newly installed package baz depends on libfoo2 (and yes, there are even more funny aspects in terms of indirect linking)), this doesn't work for devices with a dozen MB of total storage. The increase in (package) meta data alone (needed for the package manager) would probably kill most devices (both in terms of flash and RAM usage). The situation for the kernel and its module (kmod) packages isn't solvable within this scope anyways, as you usually can't touch vmlinuz (usually residing outside the rootfs, on a dedicated partition - often using a dynamic partition split, meaning vmlinuz mustn't every get larger during (hypothetical) in-place upgrades) on the running system at all.

Yes, doing this 'properly' (like the big distros, e.g. Debian, do) would be nice - but it's simply not an option for most target devices. Both opkg, the optimized-for-size-not-because-it-would-be-good package manager used, nor the system specifications of OpenWrt's target devices can cope with this (you'd need at least ~1 GB for the rootfs and a more reliable failsafe boot option on the hardware side, as well as at least 128 MB RAM). There is a reason why the minimum hardware requirements of general purpose distributions (such as Debian - at least the minimum figures stated there are imho too optimistic) are much more demanding than OpenWrt, keeping this working does take (huge) efforts and flash/ RAM.

2 Likes
11

Thanks for the follow-ups everyone, this helped clear things up a lot. The "new build in repo with same version name due to flash constraints" was the main blind spot for me.

My take-away then is that to add packages to my router and (if feeling lucky) to also pick up updates even in the absence of a new official release, I should be using imagebuilder and sysupgrade. Correct?

12

For me, at least, using the full build environment gives me a self-consistent set of packages that have been "compacted" into the ROM. At least from what I read, the requirements for the full build environment aren't much more than the image builder. Using the build system has the advantage that you can pick a good point in time (for you) and stick with it until you decide there is a better point to move to. Especially on master, there are occasionally hours or days when there is subtle "breakage" that makes it unsuitable for use. (As an example, there was a lot of work going on with wireless early in 2018 that temporarily broke some advanced networking features.)

Either way, there are so many advantages to building your own ROM that yes, this would be the preferred method for between-release upgrades, or even significant additions of packages.

13

Re: master vs. releases in image builder: wouldn't e.g. the image builder for 18.06.02 pick up 18.06.02 builds, vs. master builds? Or are those two overlapping?

14

Master and 18.06 branch are quite separate.

18.06.2 imagebuilder would pick 18.06 packages, quite properly.

15

Is the lack of that point release significant? I.e. with 18.06.2 out yesterday, would it be correct to say that whether using the imagebuilder from 18.06.1 or 18.06.2, today one would get the same image contents?

Trying to understand what corresponds to these new builds that are available via opkg, even if not advisable to just opkg upgrade to them. Do they mean "new builds in the same release, in preparation for some, undetermined when, point release"?

16

If these things are of importance to you (and they are to me), the full build system will give you control over exactly which point in the main archive, be it a release branch, master, or a staging tree is in use, as well as each of the package feeds.

As far as I know, the default behavior of the image builder is to use the oft-updated package repository. A specific repository can be specified as described at https://openwrt.org/docs/guide-user/additional-software/imagebuilder#configure_package_repositories

17

It seems that luci-ssl-openssl and its dependencies are now mandatory.

1.) service uhttpd stop
2.) opkg remove luci-ssl
3.) opkg remove libustream-mbedtls
4.) opkg update
5.) opkg install luci-ssl-openssl
6.) service uhttpd start

or with one-liner:
service uhttpd stop && opkg remove luci-ssl && opkg remove libustream-mbedtls && opkg update && opkg install luci-ssl-openssl && service uhttpd start

It worked for me.

1 Like
18

No, it is not mandatory, you’ve simply found how difficult it is to unwind the errors of bulk upgrading and used a sledgehammer. You removed the mbedTLS dependency and replaced it with an OpenSSL one. By doing so you ignored the compressed versions in ROM (they’re still there, consuming flash), replacing them with uncompressed versions that provide similar functionality in the overlay. I’d guess you lost a MB of storage or so.

The substitution of OpenSSL for mbedTLS as described above may fail for other sets of packages. It will fail if the user is already tight on storage, potentially requiring a re-flash when the device can’t write to the overlay. Adding fuel to the fire is not a recommendable approach.

1 Like
19

Everything you defined is true.
But it is still a viable workaround for some (including me).

Do you know of any way that would ease the instalation process after flashing.
I would take me a day to reconfigure everything from scratch if I re-flash the router...
Config files could be different between new sets of packages included in newer release and I am not willing to risk misconfiguration...

As for opkg upgrading this is the first time for me that I had any kind of problems / errors in any logs / version mismatches. Also never before stumbled upon any kind of explicit warning that upgrading trough opkd is so dangerous.

Also thank you for your accurate response.

20

With any re-flash / upgrade, you want to make sure that the sysupgrade process retains the proper files.

sysupgrade -l

will list the files that it will back up. The list can be modified with entries in /etc/sysupgrade.conf Adding new packages usually adds their config, but you should confirm.

The list of packages can be obtained with opkg list-installed and there are some scripts floating here to try to narrow that down to "user-installed" packages (those not in the ROM). The version of sysupgrade on master looks to have a new feature to capture a list

                # Format: pkg-name<TAB>{rom,overlay,unkown}
                # rom is used for pkgs in /rom, even if updated later
                find /usr/lib/opkg/info -name "*.control" \( \
                        \( -exec test -f /rom/{} \; -exec echo {} rom \; \) -o \
                        \( -exec test -f /overlay/upper/{} \; -exec echo {} overlay \; \) -o \
                        \( -exec echo {} unknown \; \) \
                        \) | sed -e 's,.*/,,;s/\.control /\t/' > ${INSTALLED_PACKAGES}

(I haven't kept track of this as I almost always just build what I need into a ROM)

Creating your own backup of /etc/ can be valuable for comparison, or pulling a single file. Something along the lines of the following, then copy with scp to a safe place

tar cvf /tmp/office.etc.2019-02-01_0650.tar /etc

The truly paranoid might want to backup for offline, reference purposes all of /overlay/ (all files changed relative to the ROM). (Yes, I've done that myself, especially when changing hardware.)

With that in hand, sysupgrade should be smooth, your config preserved, and only a handful of packages to install from that list, if any.

1 Like