ImageBuilder: How to preserve a config file when settings are not saved

For Developers
1

I have built some images with ImageBuilder that deploy a working configuration on a device that has gone through the DSA switch changes.

Deploying various files and using the uci-defaults script allows for a simple way to deploy customisations common to multiple devices that preserve remote access, ssh keys, etc.

This is important as the the DSA change requires the forced upgrade, where no configuration is preserved, and the process works great.

Now, I want to modify the image to additionally preserve the existing WiFi configuration. Specifically, all I want to do is preserve the /etc/config/wireless file that exists on the device already, during the sysupgrade process. I am aware that I can deploy a custom configuration in the imagebuilder /files/etc/config/wireless . This is not what I am trying to do. I want to build an image that would work on two different devices, with to different wireless configurations, and in each case preserve the /etc/config/wireless on the device.

My first thought is that perhaps the sysupgrade process would obey a modified /etc/sysupgrade.conf in the imagebuilder /files/etc/sysupgrade.conf and preserve any files listed within, however that does not seem to be the case when I added '/etc/config/wireless'

I am assuming there is a simple way to do this, but my Google foo and poking around the imagebuilder scripts has not enlightened me so far.

Any pointers appreciated

2

I know you said you don't want to include it in the uci-defaults scripts, but sounds like it may be easier to check for the device ID in the uci-defaults script and configure wifi accordingly.

1 Like
3

My writings below are are based on this statement: DSA change advising to not keep settings, but no need for factory image (like with kernel size changes).

No, the sysupgrade process is based on files of the router, not in the image contents.

Regarding stangri's proposal:
I have done that kind of device id sniffing and config scripting some 11 years ago for WiFi config, base on MAC addresses read from flash art partition (Macs, WiFi calib data etc.. The parition name varies)
Example in Best way to automate OpenWrt configuration - #14 by hnyman

But other angle for you:

DSA change usually only invalidates /etc/config/network and /etc/config/system (LED definitions).

At least with mvebu wrt3200acm I was able just go back and forth over the DSA change barrier by deleting the network and system config files, replacing them with the DSA/non-DSA corrected version, and then sysupgrading with setting kept. (If the system and network files are missing at first boot, the system creates the with defaults.)

From that perspective, ideas for you.

  • Make script that removes or modifies the network and system config files for DSA, run that before sysupgrade and then sysupgrade with settings.
  • If you have a uci-defaults script that modifies all else, make a script that modifies the "kept files" list in the router so that everything else is removed for the list but only wireless is left to be kept. Then after running that before sysupgrade, you could sysupgrade "with settings" (=only wireless) and everything else would be created by your already existing config script. (But note that the keep list is not only /etc/sysupgrade.conf as there are other sources for info).

The possible scenarios depend a bit on whois doing the sysupgrade, and they follow additional instructions.

EDIT:
In case you go to the "keep wireless, drop other config" strategy, here are some hints for you regarding the rules which files are kept. the /etc/sysupgrade.conf is meant as additional user stuff, while the main system ruels are in /lib/upgrade/keep.d/. Already removing the /etc/config related lines from /lib/upgrade/keep.d/base-files would help you a lot:

root@router1:/etc# ls /lib/upgrade/keep.d/
base-files            collectd              luci-base             ppp
base-files-essential  dnsmasq               netifd                uboot-envtools
block-mount           firewall4             odhcp6c               uhttpd
busybox               libopenssl-conf       opkg                  unetd

root@router1:/etc# cat /lib/upgrade/keep.d/*
/etc/config/
/etc/config/network
/etc/config/system
/etc/dropbear/
/etc/profile.d
# Essential files that will be always kept
/etc/hosts
/etc/inittab
/etc/group
/etc/passwd
/etc/profile
/etc/shadow
/etc/shells
/etc/shinit
/etc/sysctl.conf
/etc/rc.local
/etc/config/fstab
/etc/crontabs/
/etc/collectd.conf
/etc/dnsmasq.d/
/etc/nftables.d/
/etc/ssl/engines.cnf.d/devcrypto.cnf
/etc/ssl/engines.cnf.d/padlock.cnf
/etc/luci-uploads
/etc/udhcpc.user.d/
/etc/odhcp6c.user.d/
/etc/opkg/keys/
/etc/ppp/ip-down
/etc/ppp/ip-up
/etc/ppp/ipv6-down
/etc/ppp/ipv6-up
/etc/config/ubootenv
/etc/fw_env.config
/etc/fw_sys.config
/etc/uhttpd.crt
/etc/uhttpd.key
/etc/unetd

root@router1:/etc# cat /lib/upgrade/keep.d/base-files
/etc/config/
/etc/config/network
/etc/config/system
/etc/dropbear/
/etc/profile.d
3 Likes
4

Thanks, that was very informative and helpful. The use case I am solving is one where the wireless settings would not be known (i.e. may have a customized password by somebody else), and I do not want to create an image per device.

The device id sniffing is interesting, but wouldn't work for what I am trying to accomplish - I am sure pulling them from the firmware to set some defaults that match a sticker is useful for a staging, rather than upgrading script.

Using the "keep wireless, drop other config" strategy, I think you are suggesting a modification to how the "preserve settings" works in an attended upgrade, where I would choose to force install the DSA image, and select save settings, knowing that it will only save wireless, due to the proposed ideas for modification. Looks straightforwads, thanks.

I had been looking closely at some uci-scripts that call functions like config_load xxx as well, to do some migrations and preservation of things, and I think at some point I might be able to something like below, maybe a 10_keepwireless

I am not sure about the sequencing, and when the old configuration is available to be loaded, and stored in the new configuration. With your your MAC reading script the firmware values are always there, if the device partition is accessible. With sysupgrade, I am wondering if the load_config functions always load the old configuration during a sysupgrade? How do you differentiate between uci-defaults scripts operating on the old router configuration or the new image configuration. Is it sequence 1_ vs 99_ or is there a folder location for this stuff. There are a lot of places for uci-defaults browsing the imagebuilder folder structure. Maybe /target/root/etc/uci-defaults

It seems my example could be loading old_key and saving new_key as I intend, but maybe it would just load new, modify it, and save new, like a script that is just updating the syntax of the default settings to be platform specific.

config_load wireless 
config_get current_password wifi-iface.@wifi-iface[0].key 
uci set wireless.@wifi-iface[0].key="$current_password" 
uci commit wireless

credit seems due for the silent typo avoiding partner I had draft up that example unmodified>: ChatGPT [write a script to be used in an openwrt firmware image's uci-defaults folder that uses "config_load wireless" to lookup the existing wifi-iface password during a sysupgrade and save it using "uci commit wireless"]

So I can't help myself; As I write that, >:ChatGPT [what is the correct folder location to place the previous script withing the openwrt imagebuilder folder struction in order for it to load the value from the device that is being upgraded prior to replacing the firmware and configuration]

The correct location to place the script within the OpenWrt ImageBuilder folder structure would be in the "package/base-files/files/etc/uci-defaults" folder. This folder is where the UCI default configuration files are stored, which are executed during the system startup and during a firmware upgrade. By placing the script in this folder, it will be executed during the firmware upgrade process and the wireless password will be saved before the new firmware and configuration are applied.

Off to test some stuff out

5

You can have one single image that reads a file in a predefined location for additional info. Something like uci set wireless.@wifi-iface[0].key=$(cat /etc/user/wifi-password)". You will need to check if file exists, though.

6

Sysupgrade is pretty stupid, simple and straightforward.

(Talking about a "normal router", not an SD card based thing here)

Think about it as process of separate stages:

  1. In old currently running firmware: If "keep settings" is checked, then copy all settings to a backup archive. In practice, just copy the files specified in /lib/upgrade/keep.d/* and /etc/sysupgrade.conf to a .tar.gz archive.
  2. Flash the new firmware image: overwrite kernel and the whole rootfs. The new rootfs in the image contains the default config. If settings are kept, then flash also the backup archive to the end of the rootfs.
  3. Reboot, launch the new firmware. Preinit stage of the boot. Initialize the rootfs overlay. If a backup file is found, expand its contents into the rootfs overlay. This will place the copied config files etc. into the system, replacing the possible defaults.
  4. Early boot stage, run the uci-defaults scripts to possibly modify/migrate config files.
  5. Later boot, launch the services etc.

So, in the "keep wireless, drop other config" strategy, you

  • remove all other config files from the copy list, and only save/flash the wireless config.
  • All other config files are in the firmware image, but network and system files will be created on the fly based on the device recipe if not found in the image (usually not) or the backup). Similarly the default wireless config will be created if not found in the image or backup.
  • Your uci-defaults scripts should find the other OpenWrt defaults in place plus your copied wireless config file.

It should be pretty straightforward.

That might be viable if you compile a MAC-keyed database of all the devices of their varying wireless settings and include that database along the script to the image, so that the expand/apply script could use it. (I guess that the device-specific variation in the settings is rather limited, about the channel, SSID, password etc., so it might not be that huge. But sure, if you have multiple SSIDs per device, it can get complex.)

Without knowing if you are talking about a few, a dozen or hundreds of devices, it is pretty hard to assess the feasibility :wink:

7

Okay, I think I get it. If it isn't preserved it the tarball created by keep.d then it will not be present for use in scripts later in the process.

This explains my results when I tried a /files/etc/uci-defaults/2_wireless similar to above and end up removing all the values I touch from the deployed configuration. My script is running, but loads null/empty values due to the old configuration not being available, and the uci commit of the empty values results in their removal from the configuration.

So I went ahead and purged keep.d except for a single entry in base-files pointing to /etc/config/wireless as explained by @hnyman as an option earlier.

It seem that ImageBuilder downloads the base-files package and rewrites the files mentioned above. And so changes must also be placed in FILES.

Otherwise I still end up with a non-DSA network configuration.

Armed with my new one line base-files in files/lib/upgrade/keep.d/ I end up with a device that has no network configuration file.

I am confused as to why this is happening.

/etc/config/network is not being created now that I have removed it from base-files. I need it to be generated in order to have the correct macaddr on the interfaces.

Soo close, and now just need to figure out how to force the generation of the platform specific network configuration file when configuration is being preserved.

8

It should be automatic.

Are you modifying the network config file in your scripts? Changing MAC address?

The relevant boot logic goes like this:
the "boot" init script https://github.com/openwrt/openwrt/blob/master/package/base-files/files/etc/init.d/boot

Not quite sure what you are doing... The new image contains new logic to create the new (DSA based) defaults is no config file is present in the backup (or in the image itself).

You have just removed it from the backup files list, right?
In the device itself, right?
(The image contents have no role here.)

The defaults generating config_generate script should still be quite normally in image.

Note that you can also check the backup contents in the source device with the sysupgrade list command, which just lists the files to be backuped. You can test yyour method. (Similarly, the actual backup file is just .tar.gz, so you can easily check its contents.)

 OpenWrt SNAPSHOT, r21957-a336b6c7cf
 -----------------------------------------------------
root@router1:~# sysupgrade -l
/etc/backup.stats/.placeholder
/etc/backup.stats/stats-20230129T0200.tar.gz
/etc/backup.stats/stats-20230129T2038.tar.gz
/etc/backup.stats/stats.tar.gz
/etc/collectd.conf
/etc/config/adblock
/etc/config/attendedsysupgrade
/etc/config/bcp38
/etc/config/collectd
/etc/config/ddns
/etc/config/dhcp
/etc/config/dhcp-opkg
/etc/config/dropbear
/etc/config/etherwake
/etc/config/firewall
/etc/config/fstab
...
9

That sounds actually wrong, as that files/ implies the image. You need to modify the keep list on the source device (which runs the sysupgrade process), not in the image contents.

Like I said earlier:

10

I tested in my router, and removing the keep.d contents removed the network config from the backup list.

There are still files from other packages, which have opkg conffiles defines, but the base-files stuff like /etc/config/network and system are removed.

root@router1:/lib/upgrade# ls
asrock.sh              do_stage2              linksys.sh             platform.sh
buffalo.sh             fwtool.sh              luci-add-conffiles.sh  stage2
common.sh              keep.d                 nand.sh                zyxel.sh
root@router1:/lib/upgrade# mkdir koekoe
root@router1:/lib/upgrade# mv keep.d/* koekoe/
root@router1:/lib/upgrade# cd keep.d/
root@router1:/lib/upgrade/keep.d# nano base-files
root@router1:/lib/upgrade/keep.d# cat base-files
/etc/config/wireless

root@router1:/lib/upgrade/keep.d# sysupgrade -l
/etc/backup.stats/.placeholder
/etc/backup.stats/stats-20230129T0200.tar.gz
/etc/backup.stats/stats-20230129T2038.tar.gz
/etc/backup.stats/stats.tar.gz
/etc/config/adblock
/etc/config/bcp38
/etc/config/collectd
/etc/config/ddns
/etc/config/dhcp
/etc/config/firewall
/etc/config/irqbalance
/etc/config/luci
/etc/config/luci_statistics
/etc/config/qosify
/etc/config/sqm
/etc/config/ucitrack
/etc/config/uhttpd
/etc/config/wireless
/etc/dropbear/dropbear_ecdsa_host_key
/etc/dropbear/dropbear_ed25519_host_key
/etc/dropbear/dropbear_rsa_host_key
/etc/group
/etc/passwd
/etc/rc.local
/etc/shadow
/etc/sysctl.conf
/etc/sysupgrade.conf
/etc/uhttpd.key
/etc/uhttpd.crt
11

Appreciate your input on this. Some details on my last comments.

When I originally tried to just remove everything from ./build_dir/target/root.orig-platform/lib/upgrade/keep.d/ the imagebuilder downloaded base-files and replaced all the files I deleted, resulting in a non-DSA network configuration being preserved on my target device. On this target device, I was still able to connect by wifi and pull the backup .tar.gz for inspection after it was upgraded and figured out this was happening.

In my imagebuilder folder I have removed /etc/config/network from files/lib/upgrade/keep.d/base-files along with all other lines and added /etc/config/wireless.

I am not doing anything on the target device itself prior to the attended upgrade, where I force upgrade and save setttings using the sysupgrade image I have created with the imagebuilder.

openwrt-imagebuilder-22.03.3$ cat files/lib/upgrade/keep.d/base-files 
/etc/config/wireless

I expect this as well. That the network configuration will be generated if it not present. However that is not the case with my one-line keep.d/base-files configured in the image. After I upgrade, I can again still access my target device via wifi and pull down a configuration backup, which to my surprise is missing the /etc/config/network file completely. How the default wifi bridge works without out it, I do not know, but happy it does.

I did not quite understand this, but when you say source device, I assume you mean my target device with a non-DSA configuration. I have an imagebuilder and my new DSA image for that target device.

I am able to grab the .tar.gz in the non-working scenarios from my target device using the wifi which is why I am able to determine there is no network configuration file present.

I am now concerned the process may require me to modify keep.d on the target device before the upgrade process rather than orchestrating that from the sysupgrade image being deployed.

12

Okay, I think I am starting to understand this more clearly. My modification to my new sysupgrade image will only impact my target device on the next upgrade.

So I have to deploy two images, if I want to to this only using images. One on the current 21.x non-DSA that deploys a keep.d profile that tells the next upgrade to skip the network configuration when settings are preserved.

The 2nd image, with a 22.x DSA configuration, would then be deployed and a network configuration would be regenerated.

I am looking at my most recent failed scenario for the 2nd time, and realized that the backup .tar.gz is missing the network configuration file, because it is following the keep.d configuration I deployed. A quick SSH verified I just have a preserved non-DSA configuration present.

So I feel it is a safe assumption that having a DSA ready image is not going to work without doing something to the target first or having my uci-defaults script completely rebuild the network configuration, similar to the original platform recipe.

1 Like
13

Sure it does.

14

Yeah, sounds right if you really want just images.

You could avoid the first flash by running a script from SSH console on each device, just to clean the keep list before the sysupgrade

15

Yes; ssh wrapper was always an option.

Thanks, I feel I understand the sysupgrade process more clearly now.