Uci_load_validate: not found

Installing and Using OpenWrt
1

As the 18.06.03 release appears to be close (or did it already happen?), I remembered that I haven’t updated the packages running on my Archer C7 v2 in a few weeks. Did the usual opkg update && opkg upgrade $(opkg list-upgradable | awk '{print $1}’) dance but after a while the router dropped the connection and the upgrade may or may not have continued in its tmux session on the router.

As the router did not come back after a while, I decided to reboot but ran into the recently reported libjson issue and thus could not connect via http or ssh any more. Ouch! After regaining access to the router's failsafe mode and executing a soft (and then a hard) reset I attempted the opkg upgrade again - but the same thing happened!

I tried only upgrading some packages and sometimes this worked, but after when I attempted to e.g. only upgrade dropbear and busybox, I ran into:

/etc/rc.common: line 32: uci_load_validate: not found

The same was printed in the system log during boot, and thus dropbear was unable to start:

Fri Jun 21 12:26:29 2019 daemon.notice procd: /etc/rc.d/S19dropbear: /etc/rc.common: eval: line 117: uci_load_validate: not found

I tried to use sysupgrade to install a fresh image to the router, but ran into other issues and sysupgrade wasn’t doing anything. Almost bricked the thing by trying to revert back to the TP-Link firmware and needed to use the TFTP recovery to finally make it back to 18.06.2, r7676-cddd7b4c77 again - right where I started :slight_smile:

With all that happening I wonder what’s going on and why opkg upgrades are currently broken. Is this a known issue and possibly related to the upcoming release? If so, I can wait of course but I’m kinda worried that sysupgrade to 18.06.3 (or whatever comes next) will fail and this all starts over again.

Does anyone have some ideas on what’s going on here?

Thanks,
C

2

Bad plan.

Don’t upgrade packages “ever”. Reflash ROM, if needed with a full, self-consistent set.

2 Likes
3

@ckujau, welcome to the community!

BTW, this shouldn't have locked you out of SSH.

4

@Jeff: why is upgrading a bad idea? I've seen the comment about "not being able to upgrade some packages due to broken dependencies", but generally I want to keep my device as current as all other systems running software. Stable releases are only released ever so often, so I always thought that opkg upgrade would be the natural routine inbetween releases. If opkg upgrade is generally not advised, maybe a more strongly worded comment or even a --no-really-do-an-upgrade switch is in order.

@lleachii - Thanks, but it did: dropbear was unable to start and next to those uci_load_validate: not found messages the system log had:

Error loading shared library libjson-c.so.2: No such file or directory (needed by /bin/ubus)

In the end, dropbear was really not running. I used the GUI to reset the router, and sometimes failsafe trick, but when the GUI was giving me an error 500 I was out of luck and had to use the TFTP trick to get this thing going again.

I'll wait for the next release to arrive now but I don't have a good feeling of not updating my device every now and then, especially from a security updates point of view. Building images myself may be a way out, but I don't plan on setting up a full blown build environment just to have my router not being compromised. Or maybe I'm missing something here. Fun fact: in the course of getting my device back online yesterday night I TFTP'ed the most current version of the stock TP-Link firmware onto the device - this was an image from January 2018, so good to know OpenWRT is still more current than that :slight_smile:

5

This is a known issue occurring at this time, there was an update to that library recently. The issue with it and new versions of OpenWrt are being fixed as of the time of this posting.

This is one good reason and case why it's not wise to upgrade packages in place.

1 Like
6

I'm not sure how familiar you are with how executables and libraries work, so forgive me if I start off with some background. On most non-microcontroller systems, when you build AppA that uses LibraryB and LibraryC, AppA contains a list of things that it needs from LibraryB and LibraryC, rather than replicating the code. At run time, the OS helps out when AppA calls LibraryB's do_something_complicated(a,b,c) and finds LibraryB and looks at its table of things that it offers and translates do_something_complicated(a,b,c) into "load LibraryB's code into memory, put a, b, c onto the stack, then jump to the subroutine at LibraryB's starting address plus 0x23969868".

Now, there is a new version of LibraryB. Two challenges, the first is how does the OS find what file corresponds to LibraryB? The second is what happens if LibraryB updated and before a was an 16-bit integer, now it is a 32-bit integer? Either will break the process and AppA will fail, often very ungracefully. This is what happens quite often with a "simple" package manager like opkg. Breakage is often "invisible" to users, but the "oh no" posts are fast and furious when it is something like a TLS library that breaks LuCI, VPN, ...

When resources are "unlimited" (compared to devices that already can't store more than the most basic functions on flash and have barely enough RAM to run), you can have libB.so.2, libB.so.3, ..., libB.so.17 and not worry about running out of space. Some libraries (TLS, for example) can be over 100 kB or even 500 kB, more space than many users have available. To be efficient, you also need to store all the metadata that says, for example, "AppA uses LibraryB version3", which again, on devices without "big" disks. Looking at one of my servers, that package database is 13,420,544 bytes right now -- more than the entire flash memory of many routers. Then there's the question of what you do with all that data, especially when you don't have enough RAM to process it in one chunk.

Why not just replace every dependency? Old-school package managers did that, and when you consider that when you replace package X, you need to replace everything that depends on it, plus everything that it depends on, then repeat the process for everything you've replaced, then repeat ... you end up replacing just about everything if you touch anything "interesting". One of the reasons OpenWrt fits into restricted flash is that it compresses the ROM, roughly 2:1 over what the package would be if installed later. ROM isn't erased when you replace a package, it just gets hidden. So if you've got 4 MB of compressed packages in your 8 MB ROM that you need to replace, you end up adding around 8 MB to that. All of a sudden, your router crashes, since you've got 4 MB of "active" ROM, 4 MB of "hidden" ROM, and 8 MB of new packages. On a device with 16 MB of flash, which includes a 3 MB kernel and another 1 MB for system data.

Yes, I'd prefer something like quarterly releases of new packages as a good balance. I handle it myself by building my own images from source for OpenWrt, but quarterly for me on my FreeBSD and Linux-based OSes is plenty fast for my needs (critical security patches aside).

2 Likes
7

Additionally this also gets more complex taking the handover between read-only squashfs and overlayfs, with its potentially newer library ABIs into account (the overlay isn't around during the very first steps of the boot process, but without a dedicated initramfs this gets a little more complex - one of the reasons why updating musl, ubus/ procd, busybox, netifd and similar low level packages on a running system likely ends up in disaster). Using the kernel as a very similar stand-in double makes explaining it a bit easier, as vmlinuz is outside the normal filesystem (usually directly on a bare mtd, pretty much immutable without a new sysupgrade), while its corresponding kernel modules reside in the rootfs under /lib/firmware/$(uname -r)/, keeping this in sync over multiple upgrade cycles (think about the overlay only appearing later in the boot process again) without being able to install multiple kernels at the same time, with a higher level bootloader and vmlinuz on a read-write fs.

2 Likes
8

Thank you @jeff and @slh for taking the time to explain all that. While I do now how libraries work, I forgot about the space limitations that OpenWRT devices usually have. And while package management in itself and resolving dependencies in particular is a non-trivial matter, I somehow just assumed that both these topics have been accounted for with respect to the core packages (opkg list-installed counts 105 packages here) that came pre-installed.

I'm using OpenWRT for some time now (since Backfire, IIRC) and never gave much thought about opkg upgrade, because it always worked and it just felt natural to update packages, especially since the package manager provides such an option. But maybe I will look into building my own images now if that's the only {support,recommend}ed way forward.

2 Likes
9

We have already https://openwrt.org/meta/infobox/upgrade_packages_warning, but are more detailed explanation at https://openwrt.org/meta/infobox/upgrade_packages is still missing.

Would be nice if someone knowledgeable could crate this page. This would save you from explaining the same thing over and over again.

10

"Avoid upgrading core packages" -- wow! And I assumed the inverse: core packages are OK to upgrade, no guarantees are given for the rest. That's...really good to know. Even more incredible that I've never run into issues with opkg upgrade before. Thanks for the heads-up!

1 Like
11

Usually it is okay to upgrade core packages (when being aware of the flash size usage implications). This incident here was exceptional since a series of events led to a broken repo, which is not something supposed to happen in the first place.

2 Likes