When using OpenWRT on a generic x86 platform with an expanded filesystem and customization, what is the best way to perform upgrades without losing system configuration?
Specifically, I have:
ext4 x86-64 on a 120GB mSATA with 1G expanded root partition
Non-UCI configs for Stubby and a few others in /etc/*
QEMU vm disk images in /vdisks and startup scripts in /etc/init.d
I've been doing it manually for a long time now which involves:
Getting a list of current installed packages using
echo $(opkg list_installed | awk '{ print $1 }')
Manually backing up /etc and /vdisks using SCP
Using the image builder for the new version to build with all packages from step 1 with 1GB rootfs
Booting into a Linux Live disk and dd-ing the new image to the drive
Manually restoring the appropriate config files from step 2 over SCP
As this is a fairly manual process and takes out my network while it's happening because I have several VLANs configured so I also have to drag a laptop up to the attic (where the router is) to get on a local connection to restore the configs over SCP. As such, I'm frequently putting it off a PITA task so I'm always several versions and vulnerability patches out of date.
Is there an easier way to do this?
A couple of things I've been pondering...
Putting the /vdisks directory for my VMs onto a seperate partition higher up the disk. Will this partition be 'safe' if I then use the sysupgrade process? If I add /etc/fstab as part of sysupgrade will it automagically be mounted exactly where I left it?
Working out what exactly needs to be added to the sysupgrade 'includes' list. But will sysupgrade support my 1GB rootfs?
sysupgrade -l will list what would be backed up. It can be at least appended to by editing /etc/sysupgrade.conf (I have not personally tested removing files.)
Which version are you upgrading from?
sysupgrade has some significant enhancements on master and probably on v19
Usage: /sbin/sysupgrade [<upgrade-option>...] <image file or URL>
/sbin/sysupgrade [-q] [-i] [-c] [-u] [-o] [-k] <backup-command> <file>
upgrade-option:
-f <config> restore configuration from .tar.gz (file or url)
-i interactive mode
-c attempt to preserve all changed files in /etc/
-o attempt to preserve all changed files in /, except those
from packages but including changed confs.
-u skip from backup files that are equal to those in /rom
-n do not save configuration over reflash
-p do not attempt to restore the partition table after flash.
-k include in backup a list of current installed packages at
/etc/backup/installed_packages.txt
-T | --test
Verify image and config .tar.gz but do not actually flash.
-F | --force
Flash image even if image checks fail, this is dangerous!
-q less verbose
-v more verbose
-h | --help display this help
backup-command:
-b | --create-backup <file>
create .tar.gz of files specified in sysupgrade.conf
then exit. Does not flash an image. If file is '-',
i.e. stdout, verbosity is set to 0 (i.e. quiet).
-r | --restore-backup <file>
restore a .tar.gz created with sysupgrade -b
then exit. Does not flash an image. If file is '-',
the archive is read from stdin.
-l | --list-backup
list the files that would be backed up when calling
sysupgrade -b. Does not create a backup file.
Fast rollback? Well, ZFS would be my preferred answer, but not easy to do even under Debian.
I think sysupgrade is the right answer I just need to get comfortable with it. Does it store the uploaded upgrade file in RAM before writing to disk? So the size of the upgrade img file is limited by the RAM available?
I haven't looked into the upgrade process for ext4-based devices. The flash-based ones are pretty careful to preserve RAM along the way. It's a balance, as the device needs to be running off a RAM-backed root file system while writing to the flash (or old file system, most likely as well). You can see some of the workings of the upgrade process, once switched to the new root (happens in "stage2") in, for example, target/linux/x86/base-files/lib/upgrade/platform.sh.
The "saved settings" look to be handled with
platform_copy_config() {
local partdev
if export_partdevice partdev 1; then
mount -t ext4 -o rw,noatime "/dev/$partdev" /mnt
cp -af "$UPGRADE_BACKUP" "/mnt/$BACKUP_FILE"
umount /mnt
fi
}
got this running yesterday dual-firmware ext4 ( non-x86 ) ... not super difficult... albeit a little time consuming... the only caveat is the initial disk setup ( one or two boot partitions and two rootfs partitions and related grub entries ), you only have to do it once...
do it properly you'd wan't a buildroot and there would be some hacking of the disk generation logic.
with x86 you might want/need to sed the default in grub... nothing stopping you from dumping the config > altpart/sysupgrade.tgz prior to rom flash/upload... if it's huge...
most ext4 images will sysupgrade in a typical manner... basic configs within versions can be kept... and additional packages are re-installed after upgrade.
the OP had a specific case / query, re: huge config.tgz and enlarged partitions... which do not apply for typical ext4 usage.
Well, I just tried it, and it definitely did not go well
It's not a big deal as I am still playing around with my APU4, but that an minor sysupgrade (19.07.2 to 19.07.3) breaks even a basic setup on x86 ext4 images is really not what I would have expected.
It did seem to have preserved my SSH settings, so I can still access the router, but most(all?) opkg packages I had installed are gone and the Luci UI (I had that configured to go through nginx) is also broken now.
keep settings keeps most things. packages that are non-default and store files outside of /etc/config probably need their additional data directories added to /etc/sysupgrade.conf
this is true for ANY IMAGE custom/ext4 or not
to save previous packages ( list )... you need to make use of the 'sysupgrade -k -b' or export a list prior to installing new image. then manually restore from this later.
this is true for ANY IMAGE custom/ext4 or not
the only thing imagebuilder adds to the process is;
( with hacks ) alter the default partition sizes etc.
ability to 'pre-roll-in' extra packages / config / data
You would be best to create a new thread about how sysupgrade or the imagebuilder works ( or even better, search and read up on the topic first )... as your questions are not related to what the OP is asking or anything specific to ext4.
Thanks, that should help. I did search for stuff exactly like that in the forums and also the OpenWRT documentation... seems all very badly documented to be honest (only stating the fact, not complaining. I know documentation is an annoying topic).
I disagree that this isn't on topic. My setup is like the OP's one and I am wondering about the exact same thing. A good way to upgrade a x86 ext4 installation without losing all the configuration and installed packages + data etc. It might also apply to other non ext4 setups, but that is rather theoretical as only with a x86 ext4 setup one is going to have larger partitions with lots of packages and data installed.
The first part of the upgrade process is to prepare for the upgrade. This includes documenting programs and settings that will need to be re-installed or restored after the upgrade, locating and downloading the correct OpenWrt upgrade image for your hardware. [...]
After the OS upgrade, there are usually some additional configuration steps required to re-install additional packages not part of the base OpenWrt install [...]
After a successful upgrade, you will need to reinstall all previously installed packages. You made a list of these above. Package configuration files should have been preserved due to steps above, but not the actual packages themselves.
I also have an x86_64 with a large partition and lots of packages and upgrade very frequently as I run off master. The way I've chosen works well for me, but is not scalable to multiple devices as it is a manual upgrade process.
I build from source and so my build always compiles the same packages. I have a script that mounts my router with sshfs and copies all the configuration files and any custom files to the files/ hierarchy in my build tree and they then get baked directly into the image.
I have one boot partition and two root partitions on my router. The grub.cfg has two entries, each one pointing to a different kernel and one of the two root partitions.
When upgrading, I simply copy the new kernel to the boot partition, mount the other root partition, erase it, untar the root file system archive and reboot. The upgraded version comes online with exactly the same configuration and packages.
Total time to upgrade is a few minutes at most and total downtime is the time taken to reboot.
(Which is also not mentioned in the documentation it seems)
Also: If I change it to something bigger than the available RAM (in my case 4GB RAM, but the SSD is 16GB), will that break it with the next update as the old image will not fit into the RAM?
Yup, that's pretty much how it's done. Nothing really relates to ram ( except config keep data size during upgrade or perhaps total flashable .bin that can be physically uploaded which don't necessarily equal the values used in IB )... you should just use sysupgrade -p x.bin if you want to sysupgrade with the 'new' partition sizes.
Is there any particular reason for running master on ur router?
Instead of having original files on ur router, I'd suggest setting up a Subversion repo and keep a working copy from where u rsync files to the build environment and to the router. It's safer and u keep track of all changes and easily see what's pending commit to do it or revert.
Since u're running master on the router, then there's no bigger issue on making configs tests on it either. U change a config, restart the service, test, and if it's not good just revert. If it's good, u sync back to working copy and commit.