Limitations of package system (opkg)

[brainchild]

@IanC

A flag also saying if they are present in firmware would be useful, although that might be confusing if it didn't account for it being added by the user via Image Builder.

The general idea I was offering was that the build system would generate the image such that included packages are properly marked. Anyone using the build system should be guaranteed that the resulting image would be valid with respect to such a constraint. The files with representing data would be intended not to be modified by the running system, including masking by layered file systems.

A more advanced iteration of such a system might give the image builder further flexibility about how to flag different packages, but such a variety of cases is not necessary to consider carefully in the beginning. The starting point might be simply a flag indicating a package is included in the firmware.

[KAD]

@IanC

I've aimed to read this thread to understand what the real problem is. I understand that opkg could probably be better but am not sure how often that would benefit how many people in their typical use of OpenWrt. For me the typical use cases are [...]

The typical use cases you list are typical because that's how OpenWrt lets you use it conveniently.

Users with other use cases have probably moved elsewhere. Probably to platforms which provide automatic security updates (including kernels) and has removed the need to build your own images, in any form. Yes, that probably mean they have moved to another type of hardware too (including virtualized).

For me, clearly part of a minority, who uses hardware that could handle a full-sized package manager, it would be very nice to have that in OpenWrt.

[brainchild]

@KAD

The typical use cases you list are typical because that's how OpenWrt lets you use it conveniently.

Yes, the discussions have largely emphasized a very constrained set of use cases, with probably the larger variety of use cases excluded, not because they are not helpful to those who might use the system, but because those who might be helped by them have been excluded from practical use of the system.

The emphasis on each device owner building a customized system from a source tree is not practical or necessary.

[jow]

@brainchild

As argued, however, such a constraint is not ultimately a reason against more automated and intelligent operation of the package manager.

That requires work on a package manager with more robust dependency solver routines and better metadata availability. Neither is available to OpenWrt within the constraints imposed by the typically targeted devices.

There is some ongoing effort to port apk to OpenWrt which might or might not address some of your concerns. As it stands, OpenWrt's ancient fork of opkg is as good as it gets for the time being. There won't likely be any substantial improvements to it before OpenWrt switches to another package manager.

[brainchild]

@jow

That requires work on a package manager with more robust dependency solver routines and better metadata availability. Neither is available to OpenWrt within the constraints imposed by the typically targeted devices.

Are you explaining that it is impossible for a package manager in general to support the functionality within the constraints, or simply that no system is currently available, for use by OpenWrt, as an existing external project?

[jow]

The latter.

[brainchild]

What is the history of opkg, with respect to OpenWrt? Is opkg currently maintained? Is documentation available? The comments above indicate that the version in use is a fork. What is the difference? Is the fork maintained against an official trunk?

[jow]

OpenWrt uses an old fork of opkg. Current upstream at Yocto is maintained but switched to libarchive and libsolv which made it infeasible for use in OpenWrt due to the storage size requirements of the dependencies (libarchive alone is several hundred KB).

Its RAM usage is also too high because it needs to build the entire dependency graphs of all repo provided packages in memory which might lead to several dozens of MB in vm rss size.

Not sure about documentation, but there should be sone in the repo.

[IanC]

@brainchild

My experience with the current system leads me to a less optimistic view.

Just briefly perhaps can you recall which packages? It might lead to a realisation that improvments within current capabilites could be made for them.

[brainchild]

My induction into opkg occurred with an innocent and naive attempt to install wpad-mesh-wolfssl, following instructions, but getting blocked, with a cryptic error message, which I later learned was caused by a conflict with the wpad-basic package included in the installation image.

Comments earlier in the discussion suggest that a similar kind of conflict might occur with ip-full and ip-tiny.

[IanC]

@brainchild

Yes, the discussions have largely emphasized a very constrained set of use cases, with probably the larger variety of use cases excluded, not because they are not helpful to those who might use the system, but because those who might be helped by them have been excluded from practical use of the system.

There's a lot of "not"s in there, but I think I follow. I agree that OpenWrt and hence opkg is for a limited set of all cases, but would also say that set is difficult to meet via other approaches.

We could start with the view that a "compact" Linux distribution like Gentoo is a suitable approach for everyone - just scale it to fit your needs. For wireless devices all someone would need to do would be create the toolchain to put it on a range of SoC, and fit the limited resource requirements. And probably a few other things like understand flash layouts and uboot.

Wherever we start there is likely to be a subset of cases that are not currently be addressed, but could feasibly be if someone did the work. It seems to me that is where opkg on OpenWrt is. It has the right features for most users, but not for all.

Everything you say makes theroretical sense and if it was in place it might have helped me a bit in the past, so I have no objection to it in principle. I would still wonder if it was a problem worth solving compared with other challenges such as the end-to-end upgrade process, which are largely package related.

[IanC]

@brainchild

My induction into opkg occurred with an innocent and naive attempt to install wpad-mesh-wolfssl, following instructions, but getting blocked, with a cryptic error message, which I later learned was caused by a conflict with the wpad-basic package included in the installation image.

Were the messages less cryptic once you understood the cause, and would clearer messages be an interim approach to reducing frustration while not risking resource management?

[IanC]

@jow

There is some ongoing effort to port apk to OpenWrt which might or might not address some of your concerns.

Hopefully there is some clarity on why apk might be better and still suitable. For example, more features but an inability to run in roughly the same resource limits as opkg wouldn't be a great win. Is it discussed anywhere?

[brainchild]

@IanC

Were the messages less cryptic once you understood the cause

No. In fact, think the premise of the question is rather circular. A message should explain the cause, not assume it's known.

Broadly, an issue persists that the tools in the project are very hard to use for those unfamiliar with their quirks, and that such familiarity is almost indispensable for success performing even simple tasks. A good error message succinctly expresses the problem, while also giving a clear hint of where to look for more information on the relevant component or subsystem.

@IanC

would clearer messages be an interim approach to reducing frustration

Surely, a large improvement would come from simply revising the message to indicate that the cause of the failed installation was a conflict with existing packages, and giving the name of one such package, if not a complete list of all that are in conflict.

[brainchild]

@IanC

There's a lot of "not"s in there, but I think I follow.

Yes. My language was somewhat circular, but only to emphasize the circularity of the premise. The question generally emerges of whether to emphasize a core set of features and uses, or to attract a broader user base through support for broader cases. The present user base of a particular system, however, rarely provides a complete measure of features that might be used more broadly if they were available.

@IanC

We could start with the view that a "compact" Linux distribution like Gentoo is a suitable approach for everyone... For wireless devices all someone would need to do would be create the toolchain to put it on a range of SoC...

OpenWrt includes specific components for administration, such as Luci and UCI, that make it suitable for provisioning many recurring use cases on single-function embedded targets. Making a generic Linux installation behave as one wishes a consumer wireless router to behave is not an activity everyone wishes to undertake.

One might conceive of porting these components to Gentoo, perhaps with an integrated build system.
Otherwise, they are only available through OpenWrt. along with all its quirks and limitations.

@IanC

It seems to me that is where opkg on OpenWrt is. It has the right features for most users, but not for all.

I understand that at the moment the options for making substantial gains in flexibility and automation are limited. Merging the upstream opkg project has been explained as infeasible, and options for another system are limited, and would require a major overhaul even in the best case.

For reference, we might outline a minimum feature set for a package manager to offer better handling of conflicts. The minimum requirements are that for two packages, A and B, it is possible for B to be specified in the package descriptor as a functional a subset of A, such that if both would be selected for installation, the system would only perform the installation process for A, and simply set a flag on B. If, later, A were removed, the system, detecting the flag on B, would then perform the installation process for B, after completing removal of A, and then assign to B a normal status of being installed. Thus, both packages may be logically selected even while the system resolves that only one is to be effectively installed.

In principle, it seems, such a solution may be added to the existing system without excessive demands on system resources.

[IanC]

@brainchild

Surely, a large improvement would come from simply revising the message to indicate that the cause of the failed installation was a conflict with existing packages, and giving the name of one such package, if not a complete list of all that are in conflict.

That was the sort of thing I meant. When I reflash my routers I get SSH warnings which handily tell me the command ot use to remove the old hash, and I then just retype my original command.

To avoid the risk of bloat to opkg I'd have thought it would be reasonable for it to say somehting like "wpad-xxx conflicts with wpad-yyy. Consider using these commands to force replacement if xxx is a superset of yyy: ..."

You would have to know that xxx was a superset of yyy, but IMHO at the level that OpeWrt operates that is a reasonable responsibility to impose on the user.

[IanC]

@brainchild

The minimum requirements are that for two packages, A and B, it is possible for B to be specified in the package descriptor as a functional a subset of A, such that if both would be selected for installation, the system would only perform the installation process for A, and simply set a flag on B. If, later, A were removed, the system, detecting the flag on B, would then perform the installation process for B, after completing removal of A, and then assign to B a normal status of being installed. Thus, both packages may be logically selected even while the system resolves that only one is to be effectively installed.

To help build out the design and ensure the right approach is taken it would be interesting to see what set of metadata could describe this. A first pass might be to find which packages "clash" on at least one file. Hopefully they would be a good guide the the super / sub sets of functionality.

For actual functionality I suspect it would be asier to just handle the "going from sub to super" scenario, which I think was what you hit. Tagging wpad-mesh-wolfssl as a "provider of" wpad-basic and allowing "atomic" uninstall of the latter while installing the former woiuld be a positive step for me. It wouldn't deal with uninstalling any no-longer required dependencies of wpad-basic or provide the reverse path of super-sub, but the latter case could still take the "message based" approach mentioned above.

[brainchild]

@IanC

It wouldn't deal with uninstalling any no-longer required dependencies of wpad-basic or provide the reverse path of super-sub,

I think it would have bad effects not to handle the reverse process, in many cases making a situation even more confusing for administrators than the current. The assumption behind using package managers is that returning to a previous package selection also returns the installed assets to a previous selection. It would not reflect a good design choice to bake irreversible effects into a system through a choice to install a package.

[brainchild]

@IanC

To avoid the risk of bloat to opkg I'd have thought it would be reasonable for it to say somehting like "wpad-xxx conflicts with wpad-yyy. Consider using these commands to force replacement if xxx is a superset of yyy: ..."

Keep it succinct, if printed to the console, e.g.:

wpad-basic: Package installation failed; conflict with wpad-basic

Or a cleanly-formatted version if possible in the graphical interface:

Installation failed for package wpad-mesh-openssl.

Conflict wtih installed packages:

• wpad-basic

[IanC]

@brainchild

I think it would have bad effects not to handle the reverse process

I admire your ambition. Hopefully you've seen enough positive engagement to be able to nudge that forwards and seek further help when needed.