Power outage during Firmware upgrade causing configuration lost

Installing and Using OpenWrt
1

Hi All,

I am facing issue with User configuration (rootfs_data) lost because of power failure during firmware upgrade.

This is not happening every time but during QA this issue is propping up many times.

As per my earlier experience with HW, there is always be two copies of rootfs_data maintained, if there is a power loss in the middle writing onto one copy, there is always a backup is available.

As I understand this issue is very common across many HWs in OpenWRT. There should be a solution available.

Can someone help me with right pointer to the solution?

regard
Bhagavan

2

When the power fails during the sysupgrade process, as far as OpenWrt is concerned, all bets are off.

If you're lucky, the device has a mature push-button tftp recovery mechanism (e.g. Netgear r7800). Here the focus is on getting the hardware to boot again, the configuration is not important (all defaults).

If you're even more lucky[0], your device has dual-firmware with sensible failover mechanisms (e.g. Linksys WRT3200acm comes to mind, after 3 interrupted power-on cycles, it reverts to 'the other' firmware partition, including its (hopefully working; but you can reset) configuration).

If you aren't lucky, you either need UART level access or have obtained a fancy new doorstop.

What all of these methods have in common, they (mostly) depend on what the OEM bootloader can offer, that's the predominant factor here (and in some (rare, but very lucky-) cases, e.g. lantiq/ mvebu, the SOC itself might provide bootloader-independent -more complicated- recovery mechanisms beyond that).

The integrity of the user configuration is not a factor in any of the considerations, recovering the hardware from hard-bricking is - if you can, factory reset and reconfigure.

--
[0] and vendors mess up, regularly, e.g. the nbg6817 has a dual-firmware setup, but no (known) method to failover into 'the other' firmware partition (contrary to Linksys' solution), but at least it has a very mature push-button tftp recovery, so you can unbrick safely.

1 Like
3

Sounds wrong, at least to some extent.
Usually there is just one rootfs_data.
I have never heard about OpenWrt duplicating/mirroring the rootfs_data in real-time.

Dual boot devices are a rare minority.

And dual-partition decides do not have a backup /duplicate of the current rootfs_data, but instead they have the previous firmware image and the rootfs_data related to that. So the possible fallback is to switch using the previous firmware.

When you start sysupgrade with a typical dual-boor device, it writes the new image and rootfs to the alternative partition, and toggles boot to happen from that. The old firmware and rootfs is left as the fallback. If flashing fails, and the new firmware is unbootable, the boot may be switched back to the old firmware automatically. But if the firmware itself is ok, and just the rootfs settings write phase failed, you might start without settings.

What is your device? Is it a dual-boot device in OpenWrt?

1 Like
4

Sorry to interfere, but is there a list of the dual-boot devices, as you call them ? Or how to find out in advance, before ordering the device itself.

5

Typically dual-boot is mentioned in device wiki in OpenWrt.
Mainly device like Linksys mvebu routers.

6

This isn’t really a OpenWrt unique issue. This disaster can happen with anything you are upgrading nowadays.

My best recommendation is to have a small UPS at least while upgrading.

I did upgrade some other home electronics a couple of weeks ago and the instruction actually said they recommend a UPS during upgrade.

2 Likes
7

Thank you every one for quick replies, very active community
A minor correction in my statement

  • My previous experience is in Non-OpenWRT domian, where backup copy of user configuration is maintained on the HW.

  • Now I am using MTK HW with customized OpenWRT image

8

Thank you very much hnyman :pray: for detailed and quick response.

Fallback to previous image is happening but I am loosing the user-configuration.
This is where I am looking for solution of retaining the 'user-configuration'

I agree with others, this is not very common issue, but is it not a important use-case (power glitch in sysupgrade) to cover openWRT domain?

Note: I am referring the document listed here

I am looking for answers for below questions

  1. Why sysupgrade is touching the rootfs_data (user configuration) partition?
  2. Any link to documentation to implement the second copy of user-config?

Thank you in-advance

9

There are no fixed partitions on MTD based flash. You have to revamp your thinking.

The "partition table" in defined in the DTS included in each firmware image, and the size of the partitions varies according to the size of the kernel and rootfs (impacted e.g. by the number of extra packages included in the image).

You usually have a defined area for the rootfs, and in the flashing process the r/o rootfs is written to flash and the preserved settings archive is temporarily written after it, so that it survives a reboot.
At the first boot after the flashing, the system notices that there is no normal r/w rootfs_data overlay and copies the settings archive into RAM, creates a new rootfs_data overlay starting from the next flash block after the r/o rootfs end. The settings are then written into the new rootfs_data.

So, the rootfs_data is always re-created.
If you change the number of packages included in the image, or packages' size grows/decreases otherwise, the r/o rootfs will have a different size, and r/w rootfs_data will start from a different place.

1 Like