I'd like to suggest that OpenWrt restores previously installed opkg packages when possible. This should be true for both manual restoration using a backup archive and when flashing a new firmware image while preserving configs.
I know that this has been suggested a few times before and was rejected due to technical limitations. However, in my opinion there are rather easy solutions to circumvent the technical limitations I am aware of, so I'd like to suggest this again.
The backup shouldn't include the opkg packages itself, but a list of installed packages. When restoring a backup, opkg should try to re-install all previously installed packages from the package repository. This avoids incompatibilities between OpenWrt versions.
The backup shouldn't include a full list of opkg packages installed, but a list of manually installed packages. Dependency trees change from time to time, and not every package that was required before might still be required or even exist in a later version of OpenWrt. Thus opkg should start to distinguish between a automatically and manually installed package.
This doesn't have to be bulletproof: opkg install should write the package names it received to a file, and opkg remove should remove matching entries from this file. This should be limited to packages from a package repository, not for packages fetched using a FQDN, because we can't guarantee that they match. I don't think that this limitation is a real problem: Restoring backups was considered a convenience feature from the beginning, possibly requiring manual interference, not as a bulletproof solution to restore exact copies of the saved system. Thus I'd consider this a valid design decision.
If opkg fails to restore previously installed packages (e.g. due to a missing internet connection, because the package doesn't exist anymore, or the user previously installed a opkg package using a FQDN) OpenWrt should print a (dismissable) warning, e.g. via /etc/banner and a notice in LuCI. It's up to the user to resolve these issues.
Restoring opkg packages should be a separate choice for the user. I'd vote for enabling it by default, but the user should be able to disable this option, e.g. if the user already knows that it won't work properly (e.g. due to a missing internet connection).
Why am I suggesting this? Because I manage various OpenWrt devices remotely which I access via OpenVPN. Doing a firmware upgrade via a VPN connection works just fine. However, it's one of the OpenWrt devices that provides the OpenVPN connection - making it very hard for me to do a firmware upgrade of this device. The only thing I do after upgrade is executing opkg update and opkg install openvpn-openssl - that's it. Everything else (esp. the configs) is already included in the backup.
Another issue is me forgetting to install a package after firmware upgrade: Just today I wondered why one of my smart home devices wasn't accessible even though everything looked fine. The issue: I forgot to install the driver of a USB wifi dongle connected to one of my OpenWrt devices. This is, without any question, my mistake, but restoring previously installed opkg packages would solve it nevertheless.
Sure, this is indeed possible (more or less), however, it requires manual interference (or a custom firmware image). I suggest to automatically reinstall previously installed opkg packages without any manual interference.
The main issue is that opkg doesn't distinguish between user installed packages (let's call it "class A"), packages installed just to fulfill dependencies (call it "class B") and pre-installed packages (call it "class C"). opkg list-installed includes all packages, no matter why they have been installed. opkg_list_dest can only distinguish the installation target, i.e. between "class C" and "class A+B" (even though this isn't strictly correct). However, to automate this we solely need "class A" packages. Not doing so will cause issues at some point - which is, following the previous discussions I've read, the reason why this idea was rejected in the past. By keeping track of just user installed packages (i.e. "class A") we can circumvent these issues.
Everything else, i.e. the actual process of installing the packages, isn't really a big deal. It's already there, it's just not being used yet.
part of the issue here are the stacked multiple layers... some more ambiguous than others...
lets say you make it the default...
then someone from a small memory device with extroot-overlay re-installs and BOOM, disk full....
hang-on.... we can differentiate between previous install dest and make sure we only restore to equivalent install dest..., you've just added about 12 layers of complexity... as even the install dest determination needs un-ambuigification...
I fully agree about the medium term need for proper 'user post-install' added packages... and some package name / changes dbs... i.e.
could probably use a category for 'internet-critical' packages... and on that point... i'd propose this step when implemented is a 'pre-sysupgrade' step... a'la 'chef-imagebuilder'...
handles the no-internet component... and a user is warned pre-upgrade if the supplimental package.zip could not be made / packages could not be resolved / were too large in total etc... of course this would exclude many low ram devices... but it's the smartest way to do it... and also offers gui based tick-deselect or even tick-ignore-config per-package...
internet connectivity is fairly predictable, not dependant on said pkg/s ( and said packages dont pose significant risk of breaking connectivity i.e. mwan / pbr etc... )
checks for 'for pkg is-installed?' ( before opkg update so you are not wasting time / repeat installs )
some logger action to keep notifying the user on repeated fails so they can find out say a package name changed / the system is failing again and again...
possible space pre/post checks
but yes, for a few packages on most devices... a pretty simple 'for pkg; if not-installed > install; done' should work pretty well...
What happens if a user chooses to run opkg install with a number of large packages on a small storage device? opkg will bail. Since restoring packages is post sysupgrade it won't damage the device - one just has to perform a factory reset and try again, this time without ticking the "Reinstall opkg packages" checkbox.
The goal is not to make firmware flashing and restoring backups bulletproof, they never were and will likely never be, the goal is best effort and improving the current situation. Just take packages installed using a FQDN (see original post): We can't be sure that the installed package matches the one in the package repository, thus we should notify the user and skip this package. You brought up packages with new names, which is basically a special case of a package not being available anymore (see original post). Let's notify the user, the user will deal with it.
All these issues are edge cases. I assume >90% of restorations will run just fine. What matters is that introducing this process won't ever make the situation worse. An example of making the situation worse would be to brick a device due to installing incompatible packages, or installing unnecessary packages eating up valuable storage. My suggestion targets these major issues by providing a way to ensure compatibility, and ensuring that only packages the user chose are being installed.
I don't think waiting for the "perfect solution" with checking the device beforehand, providing a database of alternative packages, introducing package categories, adding a GUI allowing one to (un)tick single packages, etc. is a good thing. This "perfect solution" will likely never happen from the start. Let's start with a best effort solution in the short term and improve it over time.
just adding a small note on this thread as it's kind of related... ( not packages but streamlining conf backups @ handling of ext4 or pure extroot systems )
sysupgrade needs some tweaks for non rom based installs... at the moment the 'help' is not specific about which features are ROM only features...
i.e.;
[root@dca632 /usbstick 43°]# sysupgrade -c -l
Cannot find '/overlay/upper/etc', required for '-c'
[root@dca632 /usbstick 43°]# sysupgrade -o -l
Cannot find '/overlay/upper/etc', required for '-c'
###### this is the only help line re: rom
# -u skip from backup files that are equal to those in /rom
might create a new thread regarding improvements to config migration etc. as above in the future...( ext4 tweaks ) if anyone else is interested, create a thread and tag me...
Automatic install of manual selection of opkg packages after sysupgrade
/etc/config/package
config package
option name 'wireguard-tools'
config package
option name 'luci-app-wireguard'
/etc/rc.local
# wait for ntp to get current date and time
sleep 120 && \
package-automatic-installation >/dev/null 2>&1 && \
package-reboot-after-sysupgrade &
exit 0
#! /bin/sh
result=0
i=0
package="$(uci -q get package.@package[$i].name)"
while [ -n "$package" ]; do
if [ ! -e /usr/lib/opkg/info/"$package".control ]; then
if [ ! -e /tmp/opkg-lists ]; then
opkg update || exit 1
fi
package_remove="$(uci -q get package.@package[$i].package_remove)"
if [ -n "$package_remove" ]; then
if [ -e /usr/lib/opkg/info/"$package_remove".control ]; then
if ! opkg remove "$package_remove"; then
result=1
fi
fi
fi
if ! opkg install "$package"; then
result=1
fi
fi
i="$((i+1))"
package="$(uci -q get package.@package[$i].name)"
done
exit "$result"
/usr/bin/package-reboot-after-sysupgrade
#! /bin/sh
if [ ! -f /etc/os-release ]; then
echo "$0: /etc/os-release: File not found" >&2
exit 1
else
VERSION="$(sed -n '/^VERSION=/{ s/^VERSION="//; s/"$//; p; }' < /etc/os-release)"
if [ -z "$VERSION" ]; then
echo "$0: /etc/os-release: No version found" >&2
exit 1
else
VERSION_OLD="$(uci -q get package.@rebooted_after_sysupgrade[-1].os_release_version)"
if [ "$VERSION_OLD" != "$VERSION" ]; then
if [ ! -e /etc/config/package ]; then
touch /etc/config/package
fi
uci -q delete package.@rebooted_after_sysupgrade[-1]
uci add package rebooted_after_sysupgrade >/dev/null
uci set package.@rebooted_after_sysupgrade[-1].os_release_version="$VERSION"
uci commit package
sync
reboot
else
exit 0
fi
fi
fi