To upgrade or not to upgrade packages

There are several posts in this forum warning people against upgrading individual packages on a running OpenWrt installation. The reasons given seem quite reasonable, I have been bitten myself when I tried to upgrade busybox or dropbear. So far, so good.

On the other hand, there is a page on 19.07's LuCI dedicated to list the upgradeble packages, where the user can upgrade any of them, just by pressing a button. And this seems contradictory with the "do not upgrade packages" recommendation. Also, the image builder always picks the latest version of each package.

Perhaps the "do not upgrade packages" recommendation should be revised?
Perhaps there should be a warning sign on LuCI's page?

8 Likes

I managed to get hit last weekend when I tried to add a feature to busybox. I knew the risks, but dared to take the chance :wink: and did need to use TFTP recovery :frowning:

The low-level core packages are the real problem. Mass upgrade of critical packages can lead to incoherent set of packages, (at worst in the middle of the opkg upgrade task itself.) And that can lock you out of the system. Busybox is naturally the worst, as even the failsafe mode seems to be affected by it.

Upgrading known non-system-critical vanilla userspace program should be usually ok, but the casual user will not know which packages are safe, so I tend to discourage all too frequently proposed automatic scripts to upgrade everything.

That is no problem, as there is supposed to be a coherent set of binary packages available for imagebuilder. And even there failures are possible if some critical package like libubox, libiwinfo etc. have just been updated and rebuilt, but phase1/phase2 buildbots are not in sync.

So, I am still in favor of discouraging mass upgrades.

4 Likes

It is considered bad parenting to say NO without providing an alternative :wink:

OpenWrt official releases are few and far apart; building from source is not an option for a lot of people; image builder is not much easier to get going with; there is no OpwnWrt branded image generation tool with a web interface (there some unofficial ones, but then there is a trust issue).

What is an easy to use alternative?

P.S. The message to not upgrade packages is contradictory: “do not do it, but here are the tools if you insist”, which adds confusion.

1 Like

Linux expects that the user knows what he is doing. :slight_smile:
(Yes, I ran into rm -rf * myself once...)

Honestly, I wouldn't know myself either.

We have https://openwrt.org/meta/infobox/upgrade_packages_warning, but that's rather generic and generalized. I'm all in for making this warning a bit more usefull for the users, e.g. by listing packages

  • that are known to be problematic and should never ever be upgraded
  • that are known to be unproblematic and can easily be upgraded without worries

Me too. If even experienced users fall into this pit, then it can be no good to let unexperienced users run into it.

Imagine this (IIRC we actually had a similar case in the forum already):
"I have found this one-liner for automatic upgrade of all packages. I don't understand what exactly it does, but it upgrades my packages. Wonderfull! I will put this into crontab and let it check every hour.... no, every 10min, just to be sure!"

2 Likes

Just wanna warn that most devices use overlayed rootfs, so when you trying to upgrade whole system old packages will not be overwritten, but overlayed and all new system should fit r/w part of rootfs (which is very small usually). So, always check free space before upgrade, if your device have 32+mb flash, you can try

1 Like

Only a noob would do that... we professionals do "rm -rf / somefolder/"! :wink:

5 Likes

Another complication is the ability and ease of adding a package.
Was I taking a significant risk in adding WireGuard-related packages? Is it because the packages were corrupted that I can't get it to work? I see a newer version of those packages, but would it be unwise to upgrade them?

Almost certainly, no, you were not taking a risk. It is not to say that every package is perfect and can't cause conflicts, but the main risk when installing a new package (with the standard opkg install or LuCI equivalent) is simply a configuration error. Provided one does not use the package upgrade feature, the available packages to be installed were built alongside the respective OpenWrt system build, so all packages are all self-consistent with the OpenWrt version for which they were built. Regarding corruption -- the packages go through a signature check which should ensure that the file is both intact and has not been altered/tampered, so it seems unlikely that you got a corrupt installation.

Further, with the popular/common packages (such as WireGuard), it is well tested by nature, and you would see lots of threads about system level problems if the package was known to cause them (we typically see issues related to the configuration not working, rarely a system wide problem caused by simply installing WireGuard or similar).

While configuration errors could have significant impacts, it is quite different than the risks posed by the package upgrade process which can result in an unbootable system or major stability issues due to things like major incompatibilities at the ABI/Kernel level, including kernel version mismatches which will easily take down the entire system.

1 Like

Would it be correct to say that the packages that were included components of an OpenWrt release - e.g. 19.07.2 - should not be upgraded unless you enjoy dealing with soft bricks, but installing a new package are generally safe, especially if the package is a popular one?

What you mean here is that it's fine to install and upgrade new packages (especially popular ones), but you should not upgrade those packages that came with the release?

Right now, I have 30 packages listed on my updates tab of LuCI, but I can't tell which ones are for the included packages and which ones are for the packages that I later installed.

It's so counter-intuitive to prohibit myself from upgrading when I'm presented with a list of available upgrades and convenient upgrade buttons. When someone says "some upgrades are fine," it makes me want to know which ones...

I think this is a good general rule. There are certainly plenty of exceptions where included packages upgrade without any issues, but best to avoid any risk, IMO, unless you are prepared for the consequences.

Installing new packages is indeed usually safe, but not all packages are stable/robust, you could have configuration issues, and it is also possible to install things such as different versions of device drivers that could also cause problems. I wouldn't go so far as to generalize based on popularity, though, but maybe more about the functional implications (so installing a VPN protocol or file system tool is likely to be a lot less likely to cause a problem than installing a different wifi driver that isn't fully compatible with your system, even if it is a popular one).

EDIT: I stand corrected about upgrading packages like WG and the associated kmods. Nevertheless, I would recommend excercising caution anytime an upgrade is being performed just in case...

No. I said it is fine to install packages, but I would suggest that upgrading packages, even those that were not included in the image, may still cause problems. For example, let's say you install Wireguard. The package from the repo that gets installed should have been built alongside the base image build that you're using (maybe 19.07.2 which is current as of this moment). Wireguard has a kmod (kernel mod) dependency, so it will install that, and it should be consistent with the Kernel version in the original image. However, if you were to later upgrade Wireguard and it also upgrades the kmod with a new one that is based on a newer Kernel, it is plausible that it could cause problems either for Wireguard functionality, or the entire system.

Yeah -- I understand. There is an active set of discussions about how to manage this situation -- balancing the general desire of many users to stay up-to-date with the fact that sometimes such upgrades can cause massive issues, and it isn't always clear if things will be safe or not.

This is why I personally always discourage the upgrades, unless the user is knowledgable about what the upgrade does and why they need it, as well as informed about the risks involved.

I know that opinion is not popular here but personally I'd say that a blanket statement like that is not correct. Packages in release branches are updated for a reason, with the exception of LuCI [1] every single base package update in the 19.07.2 branch is fixing a bug. Let's bisect the list of upgradable packages on a base x86/64 image (except LuCI packages):

  1. cgi-io - 16 - 19
    Fixes excessive memory usage on file uploads which unbreaks sysupgrades for various ram constrained devices, helper utility used by LuCI, cannot cause a soft-brick when failing

  2. opkg - 2020-01-25-c09fe209-1 - 2020-05-07-f2166a89-1
    Fixes excessive memory usage of opkg on some operations. Cannot cause a soft-brick when failing, only break opkg itself

  3. rpcd - 2019-11-10-77ad0de0-1 - 2019-12-10-aaa08366-2
    Extends acl format and adds a respawn parameter to the init script. Cannot cause a soft-brick when failing, worst case is that LuCI does not load anymore.

  4. dnsmasq - 2.80-15 - 2.80-16.1
    Two shellscript-only changes to the init script. Cannot cause a soft-brick when failing, worst case is lack of DNS or DHCP.

  5. procd - 2020-01-24-31e4b2df-1 - 2020-03-07-09b9bd82-1
    Cosmetic change to a debug printf message, no functional changes. The postinstall or prerm actions do not perform any service stopping or starting, so installing it does not cause interruptions

  6. rpcd-mod-file - 2019-11-10-77ad0de0-1 - 2019-12-10-aaa08366-2
    See 3.

  7. odhcpd-ipv6only - 2019-12-16-e53fec89-3 - 2020-05-03-49e4949c-3
    Bugfix for FS#3056. Cannot cause a soft-brick when failing, worst case is lack of DHCPv6 service in the LAN

  8. uhttpd - 2020-02-12-2ee323c0-1 - 2020-03-13-975dce23-1
    Performance fixes for HTTP and timeout fixes for TLS. Cannot cause a soft-brick when failing, worst case is lack of HTTP connectivity to the router (LuCI).

  9. rpcd-mod-iwinfo - 2019-11-10-77ad0de0-1 - 2019-12-10-aaa08366-2
    See 3.

I did multiple tests upgrading all base packages on 19.07.2 now and failed to produce any errors. Neither upgrading opkg with opkg, nor upgrading procd with opkg caused noticeable issues apart from the increased flash memory usage.

With the exception of procd, none of the above packages nor any of the LuCI package update can possibly break booting or cause a soft brick. The procd update itself could be seen as critical, however it did not cause any issues in my tests.

In general, the following considerations apply:

  • Package upgrades on top of the latest point release until the next point release are considered safe
  • Package upgrades on top of older point releases within the same release branches are usually safe
  • Developers pay attention to not change the ABI of core libraries within release series. If it happens anyway, it is considered a bug and regression
  • Usually only bug-fixes or security fixes are done. There usually are not major base library updates at all
  • For snapshot builds, no guarantees are made, as a rule of thumb, likeliness of breakage correlates with the ago of the installed snapshot

Package upgrades on top of the latest releases that cause soft-bricks or boot issues are bugs that require investigation and fixing. They're not expected.


[1]: The remaining upgradable packages in 19.07 are LuCI updates. LuCI receives regular bugfixes and feature backports. Caveat here is that components are sometimes tightly coupled so all LuCI packages must be upgraded at once to prevent ui corruption in some case. However, even a failure there does not cause soft-bricks, the router still remains reachable via SSH.

5 Likes

Special care is spent in release branches to ensure that exactly this situation is avoided. Userspace packages in release branches must not be updated in a way that renders them incompatible with the released kernel.

1 Like

Packages for kernel modules have a dependency for the exact same kernel and build options. In the situation you described, opkg would refuse to install the updated kmod package.

2 Likes

For some users, a non-working DHCP is pretty close to a soft-bricked device. Yes, it's easy to configure a static IP on a computer and log into the router, but it requires some knowledge and time. I would not consider this as acceptable during an unatended update procedure.

Note that I described the worst case scenario here. In none of my tests I experienced any interruptions in DNS/DHCP service.

1 Like

Candidate for breaking things is unbound 1.9 -> 1.10.....best guess to start with the lib

Well unbound is not a base package. Also that entire DNSoWhatever support proved so volatile over the last months that all bets are off imho. However also a broken unbound service is not a "soft brick".

Can we implement a UX design like this, or is there a better design:

On LuCI's software updates tab, there is a group of radio buttons for the user to select his/her risk preference, and the listed upgradable packages will be filtered according to the preference.

  • Negligible: Everybody will be surprised if upgrading these packages causes a problem, and even if it does, you can undo the changes as easily as upgrading a package using LuCI, and make the problem disappear.
  • Moderate: You might have to go through some technical procedure such as setting a static IP address on your computer and http or ssh to your router to restore a backup, but the skills required are not high, and there is no doubt that peace will be restored (if you made a backup just before upgrading.)
  • Significant: You are an expert of OpenWrt, or you are in a laboratory environment.
1 Like

No we can't as the underlying infrastructure isn't present.

@jow, @eduperez - thanks for the correction. I've updated my post above with a strikethrough so that it doesn't confuse people later.

3 Likes