Unable to unmount rootfs partition

Hi all,

I'm trying to figure out how to get sysupgrade working on one of my old OpenWRT routers. I've noticed that during sysupgrade OpenWRT terminates all the processes then pivots to a ramfs. Presumably so that the rootfs can be unmounted and consequently erased and overwritten.

However, I'm not sure conceptually how this would work as the init executable (procd in this case) would still be referenced by the init process. So how can you possibly unmount rootfs while the init process is running?

I confirmed this by running lsof after sysupgrade had terminated the processes and pivoted to the ramfs. And lsof listed the following as running /mnt/sbin/procd, /mnt/bin/busybox and /mnt/usr/sbin.dropbear for init, console and ssh respectively.

So my question is, how theoretically and practically is it possible to pivot fully to ramfs and unmount the rootfs partition?

Thanks in advance

Take a look at openwrt/package/base-files/files/sbin/sysupgrade (it's a shell script) which references /lib/upgrade/stage2 that does the heavy lifting.

switch_to_ramfs() {
        for binary in \
                /bin/busybox /bin/ash /bin/sh /bin/mount /bin/umount    \
                pivot_root mount_root reboot sync kill sleep            \
                md5sum hexdump cat zcat bzcat dd tar                    \
                ls basename find cp mv rm mkdir rmdir mknod touch chmod \
                '[' printf wc grep awk sed cut                          \
                mtd partx losetup mkfs.ext4                             \
                ubiupdatevol ubiattach ubiblock ubiformat               \
                ubidetach ubirsvol ubirmvol ubimkvol                    \
                snapshot snapshot_tool                                  \
                $RAMFS_COPY_BIN
        do
                local file="$(which "$binary" 2>/dev/null)"
                [ -n "$file" ] && install_bin "$file"
        done
        install_file /etc/resolv.conf /lib/*.sh /lib/functions/*.sh /lib/upgrade/*.sh $RAMFS_COPY_DATA

        [ -L "/lib64" ] && ln -s /lib $RAM_ROOT/lib64

        supivot $RAM_ROOT /mnt || {
                echo "Failed to switch over to ramfs. Please reboot."
                exit 1
        }

        /bin/mount -o remount,ro /mnt
        /bin/umount -l /mnt

        grep /overlay /proc/mounts > /dev/null && {
                /bin/mount -o noatime,remount,ro /overlay
                /bin/umount -l /overlay
        }
}

(Which is one of the reasons you can't use sysupgrade with a "broken" busybox)

Thank-you for your assistance. I've already looked in sysupgrade and /lib/upgrade/common.sh and seen those scripts.

Yet, I still don't understand how it's possible to stop using rootfs. Because, the kernel must make sure the init processes text section is always available while it is running. How does the code above get around that problem?

Finally, you mention a broken busybox. Could you expand on that please? Specifically, what is a broken busybox in this context?

Thank-you

The amazing Jow helped me in the chat room. I'll paste his response below:

"Usually by replacing init. the idea is to instruct pid 1 to exec() into another process not running off the rootfs
on openwrt this is the ubus call system sysupgrade '{ "command": "..." }' which does exactly that
on traditional sysvinit a similar thing can be achieved by modifying /etc/inittab and sending a specific signal."

Hi @UncleDonny, I am facing the same problem when I try to keep setting while upgrade firmware using sysupgrade, but I don't understand how do you solve it.

@AP-0088:(unreachable)/root# ubiattach -p /dev/mtd11
[14181.981324] UBI: attaching mtd11 to ubi0
[14182.366031] UBI: scanning is finished
[14182.384984] gluebi (pid 18312): gluebi_resized: got update notification for unknown UBI device 0 volume 2
[14182.393591] UBI: volume 2 ("rootfs_data") re-sized from 1 to 258 LEBs
[14182.400834] UBI: attached mtd11 (name "rootfs", size 64 MiB) to ubi0
[14182.406365] UBI: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[14182.413108] UBI: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[14182.419789] UBI: VID header offset: 2048 (aligned 2048), data offset: 4096
[14182.426668] UBI: good PEBs: 512, bad PEBs: 0, corrupted PEBs: 0
[14182.432572] UBI: user volume: 3, internal volumes: 1, max. volumes count: 128
[14182.439661] UBI: max/mean erase counter: 42/26, WL threshold: 4096, image sequence number: 348931895
[14182.448787] UBI: available PEBs: 0, total reserved PEBs: 512, PEBs reserved for bad PEB handling: 40
[14182.457929] UBI: background thread "ubi_bgt0d" started, PID 18313
[14182.458829] gluebi (pid 18312): gluebi_create: gluebi MTD device 24 form UBI device 0 volume 1 already exists
UBI device number 0, total 512 LEBs (65011712 bytes, 62.0 MiB), available 0 LEBs (0 bytes), LEB size 126976 bytes (124.0 KiB)

@AP-0088:(unreachable)/root# mount -t ubifs ubi0:rootfs_data /tmp/overlay
mount: mounting ubi0:rootfs_data on /tmp/overlay failed: Device or resource busy

No problem when attach /dev/mtd11 as /dev/ubi0, but fail when mount 'rootfs_data' volume from /dev/ubi0 to /tmp/overlay.

I am using IPQ806x board.

Any comments or suggestions would be greatly appreciated, thank you!

Did you solve your mount problem? I am running into the same mount issue on ipq platform during sysupgrade

You have to pivot off the current root. This is handled by the exec call to stage2 in the sysupgrade process.

I am running into the same problem Karhoey has mentioned, sysupgrade fails to restore the backed up config on overlay partition.

1 Like

Do you have serial logs? sysupgrade is generally pretty good about saving /sysupgrade.tar

As Karhoey pointed out, the issue seems like corner case doesn't happen 95% of time. In failure cases the mounting of overlay fails and the backed up sysupgrade.tgz gets never copied over.

@zer0_0ne I'm still puzzled by the log/description presented above as

  • There's no interactive shell involved with sysupgrade (root# ubiattach -p /dev/mtd11)
  • The overlay is typically on ubiN_1, not ubiN

As there has been a recent flood of questions about pivoting root (whether their OP knows it or not), I had though yours was inquiry was around that, as the linked post certainly seems not to be from the sysupgrade process. Forgive me if I mistakenly associated your query with those other threads' content.

NP Jeff. I agree Karhoey, should have created a new thread instead of posting in this thread. sysupgrade code is straighforward, for some reason mount fails on UBI partition leading to problems.

Which device? If dual-firmware, is it only when flashing over OEM firmware, or over OpenWrt as well?

IPQ platform. It is flashing OpenWrt CC with 4.4 kernel which gets shipped as a part of QSDK

Not much we can help with here if running a QSDK image. While loosely based on an old version of OpenWrt, Qualcomm seems to have changed so many things that support, or even diagnosis, of their proprietary OS is not possible here.

Fair point.. The following issue is eerily similar, I am suspecting the issue is with 4.4 kernel and CC than with Qcom changes

There's some "funkiness" when writing to "raw" NAND, rather than using the UBI utilities. I don't know how QSDK handles upgrades, especially on NAND, but that would be something I would look at.

As one example of how current OpenWrt handles it, once the new root (tmpfs) has been prepared and procd has been signaled to enter sysupgrade mode, effectively doing a pivot_root, the exec-ed process for the Linksys EA8300 (IPQ4019) that I am familiar with is target/linux/ipq40xx/base-files/lib/upgrade/linksys.sh. nand_do_upgrade() can be found in package/base-files/files/lib/upgrade/nand.sh. mtd is a utility that is, apparently NAND-aware.

Notice that the issue you linked solved itself when I reinstalled OpenWrt again; I would blame a fault during the first installation, not a specific kernel version.

The problem is in

platform_copy_config() {
	local nand_part="$(find_mtd_part "ubi_rootfs")"
	local emmcblock="$(find_mmc_part "rootfs_data")"

	if [ -e "$nand_part" ]; then
		local mtdname=rootfs
		local mtdpart

		[ -f /proc/boot_info/$mtdname/upgradepartition ] && {
			mtdname=$(cat /proc/boot_info/$mtdname/upgradepartition)
		}

		mtdpart=$(grep "\"${mtdname}\"" /proc/mtd | awk -F: '{print $1}')
		ubiattach -p /dev/${mtdpart}
		mount -t ubifs ubi0:rootfs_data /tmp/overlay <=== mount fails
		cp /tmp/sysupgrade.tgz /tmp/overlay/
		sync
		umount /tmp/overlay
	elif [ -e "$emmcblock" ]; then
		mount -t ext4 "$emmcblock" /tmp/overlay
		cp /tmp/sysupgrade.tgz /tmp/overlay/
		sync
		umount /tmp/overlay
	fi
}

https://source.codeaurora.org/quic/qsdk/oss/system/openwrt/tree/target/linux/ipq/base-files/lib/upgrade/platform.sh?h=NHSS.QSDK.6.1.0.r6

I’d put ubinfo -a in just before the mount there (check the spelling). My guess is that either another UBI is present, the volume doesn’t exist, or it is zero size, or the like.

One "i" in that command