Automatic firmware upgrade on multiple devices

In case if we have multiple APs (like 50, 60) running OpenWRT and if we would like to do firmware upgrade automatically in all those APs and this is triggered by some other management controller, which way is available for this ?

Welcome! There are several recent threads on why one should only re-flash a ROM rather than upgrading packages.

Assuming that you're going to re-flash a ROM, sysupgrade can be executed over ssh, using command-bound keys, for example. This would be what I would consider as it can be secure, as well as reasonably robust.

Since you're managing multiple devices, I would recommend building the ROM image(s) you need with the packages you need, rather than trying to install them on each device post facto. Configuration is typically preserved through sysupgrade.

root@OpenWrt:/# sysupgrade --help
Usage: /sbin/sysupgrade [<upgrade-option>...] <image file or URL>
       /sbin/sysupgrade [-q] [-i] [-c] [-u] [-o] [-k] <backup-command> <file>

upgrade-option:
        -f <config>  restore configuration from .tar.gz (file or url)
        -i           interactive mode
        -c           attempt to preserve all changed files in /etc/
        -o           attempt to preserve all changed files in /, except those
                     from packages but including changed confs.
        -u           skip from backup files that are equal to those in /rom
        -n           do not save configuration over reflash
        -p           do not attempt to restore the partition table after flash.
        -k           include in backup a list of current installed packages at
                     /etc/backup/installed_packages.txt
        -T | --test
                     Verify image and config .tar.gz but do not actually flash.
        -F | --force
                     Flash image even if image checks fail, this is dangerous!
        -q           less verbose
        -v           more verbose
        -h | --help  display this help

backup-command:
        -b | --create-backup <file>
                     create .tar.gz of files specified in sysupgrade.conf
                     then exit. Does not flash an image. If file is '-',
                     i.e. stdout, verbosity is set to 0 (i.e. quiet).
        -r | --restore-backup <file>
                     restore a .tar.gz created with sysupgrade -b
                     then exit. Does not flash an image. If file is '-',
                     the archive is read from stdin.
        -l | --list-backup
                     list the files that would be backed up when calling
                     sysupgrade -b. Does not create a backup file.
4 Likes

Thank you... looks feasible...

https://openwrt.org/docs/techref/sysupgrade

3 Likes

This might help get you started....

#!/bin/bash

USER="root"
ROUTERIP="10.2.3.10"
BACKUPNAME="${ROUTERIP}_$(date +%Y%m%d-%H%M)"

mkdir -p backups-$ROUTERIP

cat << 'END_OF_FILE' > /tmp/commands.sh
RBACKDIR="/backup"
	[ -d /overlay ] && {
RBACKDIR="/overlay/backup"
}

LANMAC=$(cat /sys/class/net/br-lan/address | cut -d : -f 1- | sed -e 's/://g')
echo $LANMAC
BACKUPNAME="${HOSTNAME}_$(date +%Y%m%d-%H%M)"


mkdir -p /etc/custom/
mkdir -p /etc/custom/packagelists

	[ ! -f /etc/custom/packagelists/orig.txt ] && {
		echo "Original package list not present -> creating...."
		sleep 1
		opkg list-installed | cut -f 1 -d ' ' > /etc/custom/packagelists/orig.txt
	}

	[ ! -f /etc/custom/packagelists/available.txt ] && {
		opkg list | cut -f 1 -d ' ' > /etc/custom/packagelists/available.txt
		echo "cat /etc/custom/packagelists/available.txt | grep \$1" > /bin/opkglf
		chmod +x /bin/opkglf
	}

echo ">>>>>>>>>>>>> Commencing backup: $BACKUPNAME"

	[ ! -f /lib/upgrade/keep.d/custom ] && {
	echo "keep.d custom exists"
	}

echo "Creating custom backup list /lib/upgrade/keep.d/custom"
echo "/etc/custom/" > /lib/upgrade/keep.d/custom
echo "/root/" >> /lib/upgrade/keep.d/custom
echo "/tmp/rrd/" >> /lib/upgrade/keep.d/custom
echo "/tmp/run/" >> /lib/upgrade/keep.d/custom
echo "/tmp/opkg-lists" >> /lib/upgrade/keep.d/custom
chmod 755 /lib/upgrade/keep.d/custom

echo ">>>>>>>>>>>>> Creating Directories"
mkdir -p ${RBACKDIR}
mkdir -p /backup/
echo ">>>>>>>>>>>>> Updating package metadata"

opkg list-installed | cut -f 1 -d ' ' > /etc/custom/packagelists/$BACKUPNAME.txt

echo ">>>>>>>>>>>>> Packages since last backup"
opkg list-installed | cut -f 1 -d ' ' > /etc/custom/packagelists/this.txt
grep -Fvxf /etc/custom/packagelists/last.txt /etc/custom/packagelists/this.txt;

opkg list-installed | cut -f 1 -d ' ' > /etc/custom/packagelists/last.txt
echo ">>>>>>>>>>>>> Packages since new"
grep -Fvxf /etc/custom/packagelists/orig.txt /etc/custom/packagelists/last.txt;
#cmp

uci export > /etc/custom/${BACKUPNAME}-router.conf

echo ">>>>>>>>>>>>> Running sysupgrade -d ( backup files )"
sysupgrade -b ${RBACKDIR}/${BACKUPNAME}.tar.gz
echo "Finished backup ${RBACKDIR}/${BACKUPNAME}.tar.gz"
sleep 1
END_OF_FILE

cat << 'END_OF_FILE' > /tmp/postcommands.sh
#ls -lah /tmp/*.tar.gz
#rm -f /tmp/*.tar.gz
#ls /tmp/
#du -chs ${RBACKDIR}/${BACKUPNAME}.tar.gz
#rm -f ${RBACKDIR}/${BACKUPNAME}.tar.gz
END_OF_FILE

ssh $USER@$ROUTERIP  "tee -a /etc/dropbear/authorized_keys" < ~/.ssh/id_rsa.pub
ssh $USER@$ROUTERIP '/bin/bash -s' < /tmp/commands.sh
sleep 2
scp $USER@$ROUTERIP:${RBACKDIR}/*.tar.gz ./backups-$ROUTERIP/
ssh $USER@$ROUTERIP '/bin/bash -s' < /tmp/postcommands.sh
ssh $USER@$ROUTERIP "uci export" > ./backups-$ROUTERIP/${BACKUPNAME}-uciconfall.conf
2 Likes

@anon18658076, welcome to the comunity!

This needs to be at least 61 seconds, or it could cause a boot loop (resulting in a soft brick of the device). The suggested stanza used with a reboot command is sleep 70.

1 Like

Yes, if you're triggering on a time, not only that, but you should touch some file after waiting so that when it comes back up the time is at least 1 minute past the trigger (time will be set to the latest time of any file in /etc or some such similar thing.

sleep 70 && touch /etc/timekeeper && reboot

3 Likes

Why not to modify "reboot" to do the touch ? Best, of course, to set the time immediately after net up. And before other services started.

Welcome!

I have fixed the example call

#remote
OPENWRT_HOSTS="host1 host2";
USER_PACKAGES="luci-app-upnp luci-app-mwan3 tcpdump snmpd";
EXTRA_COMMAND="sleep 70 && reboot";
curl -s https://raw.githubusercontent.com/easyinternetat/openwrt-autoupdate/master/bin/remote-auto-update.sh | sh -s "$OPENWRT_HOSTS" "$USER_PACKAGES" "$EXTRA_COMMAND"

For those who are looking for a centralized web based solution, the OpenWISP project is working on a firmware upgrade solution for OpenWRT which is able to perform mass upgrade operations, the code is on github: https://github.com/openwisp/openwisp-firmware-upgrader.

3 Likes

@anon18658076 Where I have to update this command?
If I done this it'll automatically get snapshot sysupgrade?