It's not that simple. There isn't currently a list of packages to be avoided. Please understand that upgrading all packages is not recommended and carries a lot of risk. Like I said in the other thread, you have been warned not to use this method to blindly upgrade all packages.
There are too many software packages updated every day,
if can't upgrade blindly,
I can only wait for the sysupgrade image to update.
That is a good approach. Otherwise only upgrade the packages that are included in the security advisories.
2 Likes
@jow
Would it make sense to mark procd, ubus, libubus, libubox, libblobmsg etc. plus busybox as "hold" with PKG_FLAGS:=hold ???
As far as I understand from opkg_upgrade.c, marking them with "hold" would prevent accidental upgrading of them.
if (old->state_flag & SF_HOLD) {
opkg_msg(NOTICE, "Not upgrading package %s which is marked "
"hold (flags=%#x).\n", old->name, old->state_flag);
return 0;
Like I said earlier, I feel that a major part of the problems come from accidental destruction of the procd/ubus/ubox & messaging functionality. Busybox is a bit separate but quite as essential (as it e.g. provides the "ash" shell) and its failure can cause havoc e.g. by preventing failsafe from working.
Based on my own experience with debugging the procd/ubus/libubox messaging problems at new year, breaking their operations can be quite fatal.
The deepest core utils are so tightly intertwined in their Makefile / cmakelist dependencies & references, that having a mixed combination of them may be fatal. As opkg upgrades packages in a series of operations instead of a single atomic operation, (right???), it is quite possible that at some point there is binary incompatibility between the utils in use at that moment.
The challenge in user upgrading them via opkg is that they form an essential part of the process trigger/launch/messaging mechanisms even during the upgrade operations. Breaking that functionality for a moment can easily kill the opkg ops and effectively brick the system, as an incoherent set of core utils is left. And trying to update themselves in separate actions can naturally cause that breakage.
I am not sure what is the minimum set of procd/ubus/messaging/busybox packages that keeps the low-level process and init handling system working, but marking those packages with "hold" might help preventing a lot of bricking faults. Not sure if my above list of the "deepest core packages" for "hold" is right, but those are the ones that I see as the most prone to cause bricking for the core system.
(Regarding using LuCI for upgrades, not sure if also rpcd and something related should be marked "hold" (or forced to restart after upgrade, if necessary))
Using opkg for mass-upgrades is not recommended, but there are people who are used to that kind of behaviour. Preventing them from bricking easily might be the first step to mitigate the challenge.
5 Likes
That would prevent installing security updates for them.
I know, but I am unsure if casual users are able to opkg upgrade those in any case, without breaking / bricking the system. (For example, is procd in reality upgradeable in a live system if also ubus & co need upgrading at the same time?)
Personally I always build new full versions, so the whole "let's use opkg to upgrade" discussion is a non-issue, but it seems to affect a growing lot of users who are more used to Ubuntu-like "you can freely upgrade all" approach.
So I would like to invent mitigations that would prevent the most "likely bricking" scenarios.
2 Likes
I just did multiple tests with a vanilla 19.07.2 image, both in an x86/64 vm and on an ageing tl-wr1043nd. I tried both upgrading all packages at once (using the opkg list-upgradable | xargs opkg upgrade bruteforce approach) and upgrading one-by one by clicking through LuCI top-down, I didn't encounter any specific problems. Upgrading rpcd via LuCI logged me out of LuCI and upgrading uhttpd via LuCI never finished loading (because the web server was restarted), after hitting f5 things were back to normal.
I noticed the missing LuCI menu that was reported, but that was fixed by a reboot of the device.
So far I am unable to reproduce any of the reported soft-brick issues.
It really appears that there is a reasonable population of users that think that if there is an upgrade, they should apply it. Maybe this is an extension of the Hasty Generalization or other such logical biases (such as the 'if it is more expensive, it must be better').
As @hnyman states, many people are used to this type of update mechanism on the bigger OSs.
The challenge faced in this discussion is how to balance the idea of protecting users from inadvertently damaging their installation vs enabling easy use of the package upgrade system.
Since the general (although not universal) advice, especially for new users, is to avoid upgrading packages and to keep up to date with maintenance releases, would it be too extreme to simply add a boolean property to the /etc/config/system config file that enables/disables the upgrade argument in the package manager? This setting could be exposed in LuCI as a check box with an info (?) field that explains in one or two sentences that 'upgrades must be performed with care; mass upgrading packages is not recommended' or something like that? This could serve as the 'sudo warning' I mentioned earlier -- a technical means to prevent the upgrade use for those who probably don't need it or shouldn't use it, but easy enough to enable such that it isn't a hard block for anyone. This is actually very similar to the 'hide file extensions' feature that Windows had (has?) to prevent users from messing up file associations.
1 Like
Were there some updated packages recently that broke dependencies that had been in place and stable for a relatively long time (i.e. a few weeks or a month or more)?
EDIT (for clarity): I know that there are always development activities happening that have the potential to break certain dependencies with various types of changes, including kernel versions and such. But I'm wondering if there were any recent significant changes to some core packages or other low level dependenceis that carry a more significant failure risk than normal which would have been picked up by the package upgrade mechanism in the last few days or so?
Unless it is just a coincidence, these issues appear to be extremely prevalent right now (unless it is just because it is the active discussion and people are adding their own issues to the mix) -- more than the usual here-and-there type frequency.
I see two populations out there:
- I have upgraded packages via script for 2...5 years now, without having a single problem
- I have upgraded packages for the first time and my device is practically bricked
We need to find the commonalities and differences between both.
1 Like
I used to do it also for a year or so without issues. Then at some point a year ago, a seemingly routine upgrade on luci rendered my device unusable. At that point a lot of users complained for the same thing here.
I used to blindly upgrade all packages for years, and never found any issue. This makes sense, as the image builder always picks the latest version of each package, and it is not expected to create a non-working image.
Then, some day I tried to upgrade a specific package (busybox, dropbear, I can't remember), and the upgrade process failed and soft-bricked the device.
This was not a case of an incompatible package, but an upgrade procedure that failed to upgrade a program that was running and necessary for the upgrade process.
My two cents.
3 Likes
Same here, years ago, upgrading busybox or dropbear resulted in a brick.
I'm not a long time OpenWrt user, but i've upgraded my packages from time to time.
Just until today. Today I was wondering what does Overwrite files from other package(s) checkbox mean. I've googled it, as it isn't obvious what it means, and there is no helpful explanation in the panel, in any way…
Anyway, only then, I've stumbled upon https://openwrt.org/meta/infobox/upgrade_packages_warning and this particular thread.
The conclusion: I won't by upgrading the individual packages anymore.
The question: why is there NO info in the LuCI panel about that?!
Just put a warning, that most of the users, for most of the time, should NOT use this panel. And the link to the wiki for more explanation.
I'm a web developer, I can search for this in GH or whatever. Should I make a PR or is there any reason why it's not already there?
The reason is that I as a main LuCI maintainer disagree with the „never upgrade packages“ sentiment.
1 Like
For some context, I wrote the original proposal over 3.5 years ago now... at the time, we were seeing a lot of soft-bricked routers from people mass-upgrading packages.
I personally still stand by the "don't upgrade packages" sentiment (I'll post our boilerplate warning below for reference), but I respect that @jow and others have different opinions and experiences. (for reference, this has been the topic of respectful debate for a long time... I have no desire to undermine @jow's position, but I do have a different take, as explained below)
Over the past several years, we have gained a bit more insight into the issues that happen here... more often than not, it is related to running out of storage space on the router (common problem for older or lesser equipped all-in-one/embedded devices).
I believe that we have also seen it happen in situations where some dependency change is not handled properly by the opkg system (which doesn't have sophisticated ABI checks the way that the 'big distro' package managers do) [@jow - feel free chime in here if this is incorrect, misinterpreted, or incomplete information]
With that said, users with large amounts of storage and/or more experience have been leveraging package upgrades for years without issue.
The upgrade feature becomes less of risk when become more experienced and/or use hardware with more storage, and/or run only targeted upgrades (for specific patches or new features rather than a mindset of "if there's an upgrade, it must be better... go!").
But because problems arise, especially in the context of a novice running a blind-mass-upgrade operation, it is easier and safer to simply say "don't try this at home, folks."
Below is the promised language we use, especially for those new to OpenWrt who have either asked about the upgrade buttons/scripts, or potentially already run into issues (this may also be further upthread somewhere, but worth reposting for the context):
Upgrading packages (via the CLI opkg upgrade command or the LuCI Upgrade... button) can result in major problems. It is generally highly discouraged, unless you know what you are doing or if there is specific instruction to do so.
Hmm… But are the problems listed above (still) existing?
If they do (like upgrading a single package might break/brick the system, because no real ABI checks are done or sth like that) -> then I would argue that a warning would be a good thing to do.
I mean: I wouldn't be so hard to say that one should never upgrade -> allow it, just inform, do it if you really know what you're doing.
Imagine this:
- I'm a noob(ish) user
- I'm used to run all the available updates, as 21 century keeps telling me that I need them for security/new_features reasons – most of the software nowadays does it by itself in the background (iOS for example)
- I see an update -> I wanna click it
Should I click it? Can I do it with no problems?
If yes, then cool.
If no, then (at least) a warning is a good idea.
Please remember that I'm speaking from a noob POV. I really am in this area. I really am (used to) click updates without long thinking about it (maybe reading what gets updated and why, if the info is easily available) -> with the reasoning that I have to do it anyway to be up do day with security issues, etc.
1 Like