this is a development specific question about a question/thread which started here:
I installed OpenWrt 23.05-rc2 on a WD Mybook Live NAS harddisk and created two additional partitions on it as described in the above link. In my understanding, a sysupgrade should wipe the complete disk including partition tables, but this - thankfully - did not happen in my case. My additional (data) partitions were kept, as was (most of) the configuration. This was what I wanted!
My question is, is this intentional behaviour for the MBL (this would make sense) and can I rely on this happening for the future, i.e. can I safely apply sysupgrades when the NAS device is used in production?
Or, alternatively, how do I find out? I read the sysupgrade script but the real magic seems to happen inside the /bin/ubus binary. Where (in which script, location, binary, ...) does the sysupgrade.bin actually get written to disk?
sysupgrade will never wipe the complete disk. It will replace the partition table if the first two partitions differ, which is logical because it cannot very well work with different partitions than the upgrade expects. It will then continue to write the upgrade into the respective partitions. If the first two partitions' layout is identical, it has no reason to touch the partition table, as you experienced.
Even if the partition table is overwritten, you can recreate the additional partitions with their cornerstone data, and everything will still be there -- provided that the new system's first two partitions are not larger and overwriting the beginning of the additional partitions. That's why I recommend leaving a bit of buffer between the default partitions and additional ones.
(P.S.: I seem to remember rumours that sysupgrade might become even more tolerant of differing partition table layouts, but I don't believe I have seen it manifest yet.)
ok, I owe you something now. I read this page a dozen times and actually downloaded the first firmware from there - but I simply did not scroll far enough down to find this chapter. A lot is more clear now. Sorry about that!
I do have two follow-up question though.
Can I ensure that the partition table is kept unchanged by ensuring that CONFIG_TARGET_ROOTFS_PARTSIZE and CONFIG_TARGET_BOOTFS_PARTSIZE are kept the same across upgrades when building an image?
On the above Wiki page you write:
The reason to do this is that the partition table will be reset to the default MBR partition table if ... you upgrade on a disk with GPT partition table (sysupgrade is not aware of GPT partitions and will indiscriminately overwrite the partition table) ...
This seems incorrect, at least when I upgraded from 23.05-rc2 to 23.05-rc2 (same version but sysupgrade image) the GPT partition table was unchanged and the GPT table was kept intact. Does OpenWrt save a sysupgrade.log somewhere?
root@OpenWrt:/mnt/sda4# gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.9
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 15628053168 sectors, 7.3 TiB
Model: TOSHIBA HDWG480
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): C1FB2C00-...
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 15628053134
Partitions will be aligned on 2048-sector boundaries
Total free space is 16350 sectors (8.0 MiB)
Number Start (sector) End (sector) Size Code Name
1 8192 24575 8.0 MiB 8300 Linux filesystem
2 32768 2060287 990.0 MiB 8300 Linux filesystem
3 2060288 3108863 512.0 MiB 8200 Swap
4 3108864 15628053134 7.3 TiB 8300 Data
root@OpenWrt:/mnt/sda4# fdisk -l /dev/sda
Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: TOSHIBA HDWG480
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: C1FB2C00-275D-49F0-957D-E5C88FF38B87
Device Start End Sectors Size Type
/dev/sda1 8192 24575 16384 8M Linux filesystem
/dev/sda2 32768 2060287 2027520 990M Linux filesystem
/dev/sda3 2060288 3108863 1048576 512M Linux swap
/dev/sda4 3108864 15628053134 15624944271 7.3T Linux filesystem
Is the information on the Wiki site outdated, or was this just coincidence because I upgraded to the exact same version, or has sysupgrade actually become more clever?
I believe so, yes, but I have never tried it myself so I can't corroborate it.
IIRC I last significantly revisited/edited that wiki section in 2021, sysupgrade may have become GPT-aware in the meantime. It would be interesting if and when that happened, maybe we can find out without a number of tests.
Important to note is that, when upgrading to a new version, the capabilities of the new version's sysupgrade are not relevant -- the old version's sysupgrade capabilities are. Even if the new version's sysupgrade is GPT-aware, the sysupgrade you use to upgrade to that version may very well not be.
I'm curious. Can you point me to a location in the source (Git) where the image writing magic happens so I can try to understand how this is implemented? Maybe there is something in the git logs that point to sysupgrade becoming GPT aware.
All I can find on the system itself is a call to ubus binary as a last step inside the /sbin/sysupgrade shell script, which is opaque to me, and the Git source of ubusd doesn't really help either.
Good luck, let us know if you can find something. (A related key word to look for would be "EFI", btw, lots of GPT-related improvements are tied to EFI issues.)
"ping": Upgrading to -rc3 on my device went absolutely fine using my custom built sysupgrade image with squashfs and custom root partition size.
I haven't quite figured out the source yet but this looks stable enough to me. Congratulations to the OpenWrt team, you've done a terrific job at keeping old and (otherwise) unsupported hardware alive!
as the section states if you use a different partition schema to the one defined in the new image, sysupgrade will blindly apply it. will not re-partition or try to guess what was your boot, root, custom partitions;
as the section states, it already happened, that rootfs size was changed in a new release. this answers your question: no, you cannot assume partition sizes will remain in between releases. when this happened in the past it was quite pain full exercise. so it is better imho to keep a blank space between last owrt and first custom partition, or always use a custom image with your preferred boot/rootfs size.
Yes, I read this page (especially now that I have already been caught not fully reading it and asking questions too early ... ).
This is why I am building my own image, and my rootfs is set to 1GB and then I still have 512MB of swap behind that, so I think I have some room for future changes. My 8TB disk doesn't really care about one or two GB missing.
It's better to be safe than sorry and make a backup before you rely on an undocumented feature.
Since its support is not officially declared, it can as well silently disappear upon some code change.
I reworded and updated it slightly, but I would appreciate if a native speaker would look it over. Trying to be as accurate as possible, I have a tendency to overcomplicate my writing.
I'd add one more thing: backup your custom data too, not just partition table! it is always a best practice to assume the worst, so if you data is important for you, backup before upgrade. upgrade usually goes ok, but when sh*t hits the fan you may lose your valuable data not just losing time to reinstall from scratch.
Isn't that generally good advice, upgrade or no? How goes the saying, "data that exists in only one place eventually exists nowhere", or something to that extent?
yes, regular backup is a good practice always. just i was thinking that if we highlight to take partition schema backup, a half sentence to backup data as well does not hurt.
(but there are so many best practices (and documents ) people tend to not follow/read anyway ...)
so up to you thanks for your effort either way
In /rom/lib/upgrade on my MBL, there's a file called wdbook.sh:
# ...
diff="$(grep -F -x -v -f /tmp/partmap.bootdisk /tmp/partmap.image)"
rm -f /tmp/image.bs /tmp/partmap.bootdisk /tmp/partmap.image
if [ -n "$diff" ]; then
echo "Partition layout has changed. Full image will be written."
ask_bool 0 "Abort" && exit 1
return 0
fi
# ...
# iterate over each partition from the image and write it to the boot disk
while read part start size; do
if export_partdevice partdev $part; then
echo "Writing image to /dev/$partdev..."
get_image "$@" | dd of="/dev/$partdev" ibs="512" obs=1M skip="$start" count="$size" conv=fsync
else
echo "Unable to find partition $part device, skipped."
fi
done < /tmp/partmap.image
So it seems that at least there is a definite intention to just replace partition contents, if the partitioning scheme of the original disk has not changed.
I could not find this file on Git, but it is there in my firmware, maybe it is being created during the platform specific build process.
I read this page a dozen times and actually downloaded the first firmware from there - but I simply did not scroll far enough down to find this chapter. A lot is more clear now. Sorry about that!
).