Upgrade Process for x86-64 ext4 builds [large-disks and data]

Hi folks,

When using OpenWRT on a generic x86 platform with an expanded filesystem and customization, what is the best way to perform upgrades without losing system configuration?

Specifically, I have:

  • ext4 x86-64 on a 120GB mSATA with 1G expanded root partition
  • Non-UCI configs for Stubby and a few others in /etc/*
  • QEMU vm disk images in /vdisks and startup scripts in /etc/init.d

I've been doing it manually for a long time now which involves:

  1. Getting a list of current installed packages using
echo $(opkg list_installed | awk '{ print $1 }')
  1. Manually backing up /etc and /vdisks using SCP
  2. Using the image builder for the new version to build with all packages from step 1 with 1GB rootfs
  3. Booting into a Linux Live disk and dd-ing the new image to the drive
  4. Manually restoring the appropriate config files from step 2 over SCP

As this is a fairly manual process and takes out my network while it's happening because I have several VLANs configured so I also have to drag a laptop up to the attic (where the router is) to get on a local connection to restore the configs over SCP. As such, I'm frequently putting it off a PITA task so I'm always several versions and vulnerability patches out of date.

Is there an easier way to do this?

A couple of things I've been pondering...

  • Putting the /vdisks directory for my VMs onto a seperate partition higher up the disk. Will this partition be 'safe' if I then use the sysupgrade process? If I add /etc/fstab as part of sysupgrade will it automagically be mounted exactly where I left it?
  • Working out what exactly needs to be added to the sysupgrade 'includes' list. But will sysupgrade support my 1GB rootfs?

THANKS!

3 Likes

I forgot to add - bonus points for a method that also allows a quick roll-back if something goes wrong.

sysupgrade -l will list what would be backed up. It can be at least appended to by editing /etc/sysupgrade.conf (I have not personally tested removing files.)

Which version are you upgrading from?

sysupgrade has some significant enhancements on master and probably on v19

Usage: /sbin/sysupgrade [<upgrade-option>...] <image file or URL>
       /sbin/sysupgrade [-q] [-i] [-c] [-u] [-o] [-k] <backup-command> <file>

upgrade-option:
	-f <config>  restore configuration from .tar.gz (file or url)
	-i           interactive mode
	-c           attempt to preserve all changed files in /etc/
	-o           attempt to preserve all changed files in /, except those
	             from packages but including changed confs.
	-u           skip from backup files that are equal to those in /rom
	-n           do not save configuration over reflash
	-p           do not attempt to restore the partition table after flash.
	-k           include in backup a list of current installed packages at
	             /etc/backup/installed_packages.txt
	-T | --test
	             Verify image and config .tar.gz but do not actually flash.
	-F | --force
	             Flash image even if image checks fail, this is dangerous!
	-q           less verbose
	-v           more verbose
	-h | --help  display this help

backup-command:
	-b | --create-backup <file>
	             create .tar.gz of files specified in sysupgrade.conf
	             then exit. Does not flash an image. If file is '-',
	             i.e. stdout, verbosity is set to 0 (i.e. quiet).
	-r | --restore-backup <file>
	             restore a .tar.gz created with sysupgrade -b
	             then exit. Does not flash an image. If file is '-',
	             the archive is read from stdin.
	-l | --list-backup
	             list the files that would be backed up when calling
	             sysupgrade -b. Does not create a backup file.

Fast rollback? Well, ZFS would be my preferred answer, but not easy to do even under Debian.

I'd suggest tar while running under failsafe mode

1 Like

Thanks Jeff.

From version 18.06.2.

I think sysupgrade is the right answer I just need to get comfortable with it. Does it store the uploaded upgrade file in RAM before writing to disk? So the size of the upgrade img file is limited by the RAM available?

I haven't looked into the upgrade process for ext4-based devices. The flash-based ones are pretty careful to preserve RAM along the way. It's a balance, as the device needs to be running off a RAM-backed root file system while writing to the flash (or old file system, most likely as well). You can see some of the workings of the upgrade process, once switched to the new root (happens in "stage2") in, for example, target/linux/x86/base-files/lib/upgrade/platform.sh.

The "saved settings" look to be handled with

platform_copy_config() {
        local partdev

        if export_partdevice partdev 1; then
                mount -t ext4 -o rw,noatime "/dev/$partdev" /mnt
                cp -af "$UPGRADE_BACKUP" "/mnt/$BACKUP_FILE"
                umount /mnt
        fi
}
1 Like

got this running yesterday dual-firmware ext4 ( non-x86 ) ... not super difficult... albeit a little time consuming... the only caveat is the initial disk setup ( one or two boot partitions and two rootfs partitions and related grub entries ), you only have to do it once...

do it properly you'd wan't a buildroot and there would be some hacking of the disk generation logic.

with x86 you might want/need to sed the default in grub... nothing stopping you from dumping the config > altpart/sysupgrade.tgz prior to rom flash/upload... if it's huge...

2 Likes

I am also wondering about a good upgrade procedure for the ext4 x86 images.

Do I really have to manually reinstall all the packages I installed after a sysupgrade?

most ext4 images will sysupgrade in a typical manner... basic configs within versions can be kept... and additional packages are re-installed after upgrade.

the OP had a specific case / query, re: huge config.tgz and enlarged partitions... which do not apply for typical ext4 usage.

Well, I just tried it, and it definitely did not go well :frowning:

It's not a big deal as I am still playing around with my APU4, but that an minor sysupgrade (19.07.2 to 19.07.3) breaks even a basic setup on x86 ext4 images is really not what I would have expected.

It did seem to have preserved my SSH settings, so I can still access the router, but most(all?) opkg packages I had installed are gone and the Luci UI (I had that configured to go through nginx) is also broken now.

1 Like

Ok, so my current thinking is:

If I use the imagebuilder ( https://openwrt.org/docs/guide-user/additional-software/imagebuilder ) to make my own image to install via sysupgrade then the configuration should be preserved and I can include the packages I want into the image.

Anything I might be missing?

1 Like

a basic grasp of how things work.

keep settings keeps most things. packages that are non-default and store files outside of /etc/config probably need their additional data directories added to /etc/sysupgrade.conf

this is true for ANY IMAGE custom/ext4 or not

to save previous packages ( list )... you need to make use of the 'sysupgrade -k -b' or export a list prior to installing new image. then manually restore from this later.

this is true for ANY IMAGE custom/ext4 or not

the only thing imagebuilder adds to the process is;

  • ( with hacks ) alter the default partition sizes etc.
  • ability to 'pre-roll-in' extra packages / config / data

You would be best to create a new thread about how sysupgrade or the imagebuilder works ( or even better, search and read up on the topic first )... as your questions are not related to what the OP is asking or anything specific to ext4.

1 Like

This the normal behavior for upgrading OpenWrt.
opkgscript takes a snapshot of the installed packages and you can reinstall them afterwards.

1 Like

Thanks, that should help. I did search for stuff exactly like that in the forums and also the OpenWRT documentation... seems all very badly documented to be honest (only stating the fact, not complaining. I know documentation is an annoying topic).

I disagree that this isn't on topic. My setup is like the OP's one and I am wondering about the exact same thing. A good way to upgrade a x86 ext4 installation without losing all the configuration and installed packages + data etc. It might also apply to other non ext4 setups, but that is rather theoretical as only with a x86 ext4 setup one is going to have larger partitions with lots of packages and data installed.

If you are refering to the need for re-installation of user-installed packages, then I don't agree.

From https://openwrt.org/docs/guide-user/installation/generic.sysupgrade :

The first part of the upgrade process is to prepare for the upgrade. This includes documenting programs and settings that will need to be re-installed or restored after the upgrade, locating and downloading the correct OpenWrt upgrade image for your hardware. [...]

After the OS upgrade, there are usually some additional configuration steps required to re-install additional packages not part of the base OpenWrt install [...]

Further down you find https://openwrt.org/docs/guide-user/installation/generic.sysupgrade#reinstall_user-installed_packages

After a successful upgrade, you will need to reinstall all previously installed packages. You made a list of these above. Package configuration files should have been preserved due to steps above, but not the actual packages themselves.

Regarding getting a list of installed packages:

https://openwrt.org/docs/guide-user/installation/generic.sysupgrade#identifying_customizations describes multiple ways to get a list of installed packages. Admittedly, the comfortable opkgscript is not mentioned anywhere in the wiki. But since it is a wiki, everybody can edit it * hint * :wink:

2 Likes

I also have an x86_64 with a large partition and lots of packages and upgrade very frequently as I run off master. The way I've chosen works well for me, but is not scalable to multiple devices as it is a manual upgrade process.

I build from source and so my build always compiles the same packages. I have a script that mounts my router with sshfs and copies all the configuration files and any custom files to the files/ hierarchy in my build tree and they then get baked directly into the image.

I have one boot partition and two root partitions on my router. The grub.cfg has two entries, each one pointing to a different kernel and one of the two root partitions.

When upgrading, I simply copy the new kernel to the boot partition, mount the other root partition, erase it, untar the root file system archive and reboot. The upgraded version comes online with exactly the same configuration and packages.

Total time to upgrade is a few minutes at most and total downtime is the time taken to reboot.

5 Likes

Fixed!! :wink:

1 Like

Does it still work like mentioned here?

(Which is also not mentioned in the documentation it seems)

Also: If I change it to something bigger than the available RAM (in my case 4GB RAM, but the SSD is 16GB), will that break it with the next update as the old image will not fit into the RAM?

2 Likes

Yup, that's pretty much how it's done. Nothing really relates to ram ( except config keep data size during upgrade or perhaps total flashable .bin that can be physically uploaded which don't necessarily equal the values used in IB )... you should just use sysupgrade -p x.bin if you want to sysupgrade with the 'new' partition sizes.

2 Likes

Maybe I'm missing something, but is there any reason why this wouldn't work:

  • Install the new version on a USB drive
  • rsync the files in a set of directories over to the system drive, say /bin /sbin /boot /user /lib /lib64 /root and /www

And then of course do the package upgrades.

Or is there something that just won't work about this? Would this cause issues for opkg for instance?

I had made some improvements on the https://openwrt.org/docs/guide-user/installation/openwrt_x86 page, hope it helps to clarify how upgrading may be done.

Is there any particular reason for running master on ur router?

Instead of having original files on ur router, I'd suggest setting up a Subversion repo and keep a working copy from where u rsync files to the build environment and to the router. It's safer and u keep track of all changes and easily see what's pending commit to do it or revert.

Since u're running master on the router, then there's no bigger issue on making configs tests on it either. U change a config, restart the service, test, and if it's not good just revert. If it's good, u sync back to working copy and commit.