You cannot sysupgrade an extroot partition. When you run sysupgrade, it will flash to the device's internal memory -- it cannot flash to the extroot partition directly, and the image (plus the extroot related packages) must fit within the footprint of the system's native flash memory.
My process for upgrading with an extroot config is as follows:
Make a backup of my config
Remove the extroot device (USB drive, SD card, etc)
Run sysupgrade [EDIT: without keeping settings -- I uncheck the keep settings box]
Install extroot related packages
Reinsert the extroot device
Erase the contents of the extroot device
Pivot to extroot and reboot
Reinstall user-installed packages
Restore the backup and reboot
It is possible to combine 3 and 4 together by simply creating a custom image with the extroot packages pre-installed (this can be done using the firmware-selector.
Then, I have a simple script I upload to the device that automates and combines steps 6-7. Similarly, I keep a list of the packages that I install and this becomes a 1-liner for step 8:
Just another way of saying "run the extroot process"
Correct. You can make a custom image with whatever packages will fit, but the rest need to be installed after you've done extroot. Therefore, since you're already going to be installing at least some packages after running the extroot process, you might as well do all of them after (at least, IMO). So a custom image with the extroot specific packages (and maybe others that are absolutely critical for getting the device running) is the most practical thing to flash, then the rest after extroot.
That makes sense, thanks! I admit, I wasn't understanding extroot correctly. It seems that with or without extroot, the "base" sysupgrade always goes to router embedded flash (right?), so the size limitation. Then packages installed on top go to extroot. That makes sense - again, just me not realizing that. And of course, config files stay in flash, not extroot.
Wondering if there is an automated way to reinstall packages after upgrade. Do you know of any? If not, I may try to generate a script to do this.
Also BTW, the alias for sysupgrade - has this been fixed now, or is it still needed? Just not finding any updates, so thinking this is still needed.
But to your notes above, it seems I don't have to do anything with extroot during the upgrade, just leave it there ... no? I admit, still a bit confused which of the steps on the OpenWrt docs page I need to do (the "extroot process").
Config files added/modified prior to running the extroot process[1] will reside in the unit's built-in flash memory.
Config files added/modified after extroot is running will only exist in the extroot filesystem[2].
To clarify about extroot... once you've set that up, all changes (package installs, config files, etc.) are saved to the extroot media. If that media is removed and then the device is booted without it, the device will still operate 'normally' wherein that normal behavior is the configuration that existed prior to the extroot process. Any changes made with the media removed will be saved in the internal flash memory.
So... provided that you do all of your personal customization with extroot completed and the media inserted, you could basically treat your extroot as your personalized overlay. If you remove the extroot media, the device would look like a nearly-default state of OpenWrt.
I am not aware of any truly automated methods, but it is possible to script this. As I described in my personal procedure, I script the extroot process and then run a 1-liner to do the package installs. You could make a more sophistocated script that could do everything, and that could be copied to the device after flashing, or you could even build it into the base image. It will be a bespoke script, though.
Which alias are you referring to?
Personally, I remove the extroot media before running the upgrade, but it's not really necessary to do so.
How so? It seems you have completed this process in the past. What are you finding confusing? You would just repeat the process again after completing the flash upgrade.
or if the extroot storage media is not available/installed, even if the extroot process has been completed. ↩︎
assuming the extroot storage media is installed/available; otherwise refer to the earlier point ↩︎
That makes sense, and thanks for clarifying. You are right, and I got a bit sloppy with my explanation . Appreciate it!
Two questions then I think,
To your point - when extroot is mounted, packages, config files, etc. are all stored there. Then boot up, and the config files are pulled from router flash. How to sync the config files?
The steps noted here, all clear, but it seems like 1-6 (or 5, really) are all a one time thing, right? So what is needed to be "redone" after upgrade. I admit, that'
s the part that confuses me.
This issue here, and the alias noted. Do you not see this issue now?
This whole process needs to be redone after a flash. You can think of this as a one-time thing per-flash/version. If you upgrade your OpenWrt version (major such as 22.03 > 23.05, or minor such as 23.05.0 > 23.05.2), you need to repeat the process.
This is because basically all of the files from the internal flash memory are copied to the extroot. In order for everything to be consistent after an upgrade, it is necessary to run the extroot process again so that it copies the latest files (i.e. from the newly flashed firmware version).
I think the easiest way to conceptualize this is as follows:
When you run extroot, you effectively mirror the contents of internal flash memory to your extroot media.
The boot process begins using the internal flash memory and proceeds normally until the point at which it would mount the overlay partition (that is where all your configs and user-installed packages reside).
At that point, the system uses the extroot partition to read configs, including all boot processes that occur after the overaly partition is mounted (normally, the overlay would be in internal flash memory, but now we have it on the extroot media).
Regarding syncing of configs... when you run the process and mirror the files, that is when the config files are in sync. However, beyond that point, unless you make specific efforts to keep them synchronized (and there is usually no reason to do that), the config files may diverge based on whatever changes you make. When the extroot media is inserted and used during boot, all configs will be saved on the extroot media. If it is absent, all changes will be saved in internal memory.
For all practical use cases, once you've got extroot running, all of your configs (and additional packages) will be installed there, and there is no reason to worry about the config that is sitting in the internal flash memory as it is not unsed unless the extroot media is removed (or fails).
Ah... I think I see what is happening here. In this situation, the user has kept the settings across the upgrade. If that is done, the fstab entry that is responsible for mounting extroot will still exist. Once mounted, the boot process may encounter issues because of the extroot media being inconsistent (per my comments earlier).
The reason I've never even thought or worried about this is because in my process, I do not keep settings... in fact, the internal flash storage is basically completely default except for the extroot packages and maybe a few other odds and ends that might be important in certain circumstances. I just always start from default and I have a script to do everything in one shot, so I re-create the fstab entry and never worry about it.
The other approach, if you keep the settings across the upgrade, is to remove the extroot media after the flash upgrade, then plug it back in post-boot. Mount the drive, erase the contents and run just the tar part of the process to mirror the files, then reboot.
EDIT: I have edited my steps above to clarify that in step 3, I do not keep settings.
Thanks so much for the detailed reply - very much appreciated! I was slow to get back to you, as I wanted to dig into this, understand it a bit more. It's making more sense ... gradually, LOL!
BTW, the reason I care about the config sync, and savings settings - I have no wired connection where the device is at, so the WAN connection is WiFi => need it come up after sysupgrade, or I lose connectivity to the device.
Part of my fight right now is that I seem to keep "losing" my USB thumbdrive. Not sure why, but like below (example) ... then all bets are off
[ 79.604404] usb 1-1: USB disconnect, device number 2
[ 79.629797] blk_update_request: I/O error, dev sda, sector 78624 op 0x1:(WRITE) flags 0x4800 phys_seg 240 prio class 0
[ 79.640825] Buffer I/O error on dev sda1, logical block 76576, lost async page write
[ 79.648843] Buffer I/O error on dev sda1, logical block 76577, lost async page write
[ 79.656749] Buffer I/O error on dev sda1, logical block 76578, lost async page write
[ 79.664702] Buffer I/O error on dev sda1, logical block 76579, lost async page write
[ 79.672646] Buffer I/O error on dev sda1, logical block 76580, lost async page write
[ 79.680610] Buffer I/O error on dev sda1, logical block 76581, lost async page write
[ 79.688570] Buffer I/O error on dev sda1, logical block 76582, lost async page write
[ 79.696787] Buffer I/O error on dev sda1, logical block 76583, lost async page write
[ 79.704830] Buffer I/O error on dev sda1, logical block 76584, lost async page write
[ 79.712796] Buffer I/O error on dev sda1, logical block 76585, lost async page write
[ 80.039754] blk_update_request: I/O error, dev sda, sector 78872 op 0x1:(WRITE) flags 0x4800 phys_seg 240 prio class 0
[ 80.052150] blk_update_request: I/O error, dev sda, sector 78864 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 80.064991] blk_update_request: I/O error, dev sda, sector 79112 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 80.079814] blk_update_request: I/O error, dev sda, sector 79120 op 0x1:(WRITE) flags 0x4800 phys_seg 240 prio class 0
[ 80.095384] blk_update_request: I/O error, dev sda, sector 79360 op 0x1:(WRITE) flags 0x4800 phys_seg 240 prio class 0
[ 80.111041] blk_update_request: I/O error, dev sda, sector 79608 op 0x1:(WRITE) flags 0x4800 phys_seg 240 prio class 0
[ 80.124237] blk_update_request: I/O error, dev sda, sector 79600 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
[ 80.139958] blk_update_request: I/O error, dev sda, sector 79856 op 0x1:(WRITE) flags 0x800 phys_seg 232 prio class 0
[ 80.152265] blk_update_request: I/O error, dev sda, sector 79848 op 0x1:(WRITE) flags 0x800 phys_seg 8 prio class 0
mkfs.ext4: I/O error while writing out and closing file system
Need to figure this out, or I can't get too far - unfortunately. BTW, I also found that the device is recognized as usb-storage, so I had to add kmod-usb-storage. It's not showing up as a block device (like the general instructions say), rather it needs to be seen as usb-storage. Perhaps that's a clue.
You note about "when you run extroot" - but I never really run it, agreed? You are right, it seems that configs ONLY come from the overlay, whatever that device is. I was thinking overlay would be a combination of internal and extroot, but it seems to only be extroot (if mounted). Agreed?
And to your comment,
I do agree! It seems like during boot ... if extroot is there, all configs come from only there, right?
In that case, if you remove the extroot media first, you should be fine to keep settings across (most) upgrades. Re-insert the media once your system is finished with the upgrade and booted... then erase the media and redo the tar part of the extroot process.
These look like one or both of the following:
the usb device's memory is worn out and failing.
the usb stick is requiring more power than is available and is browning out.
I suspect the former.
No, I don't agree... at least based on your descriptoin. It seems that you are running extroot. For more precision, we could say "you're running an extroot configuration" (this is the running config with the external media in use for your extroot), and we can also say that "you ran the extroot process to setup the extroot media and to edit fstab to mount as overlay."
So when you say you "never really run it" -- what do you mean? Are we possibly talking about different things?
That makes sense. Or, just rely on / keep the settings on extroot?
I was thinking the same, but interestingly, it seems (from dmesg) that firing up the wireless driver triggers this? I also noticed that restarting network ... USB reconnected, then disconnected. Tried another USB stick, exactly the same thing.
[ 75.391703] b43-phy0: Loading firmware version 666.2 (2011-02-23 01:15:07)
[ 75.399836] usb 1-1: USB disconnect, device number 2
I was misreading what you meant by running . I thought you meant "run a command", vs. "you're running an extroot configuration". My bad - and agree with your logic.
Thanks! Now, to figure out the phy vs. USB mess. Without that, no USB drive connected.
[ 410.400231] phy0-sta0: deauthenticating from 1c:61:b4:d0:72:e4 by local choice (Reason: 3=DEAUTH_LEAVING)
[ 411.055322] usb 1-1: new high-speed USB device number 3 using ehci-platform
[ 411.387900] usb-storage 1-1:1.0: USB Mass Storage device detected
As soon as phy0 is dropped, the USB drive comes back. Some sort of conflict?
No, because the rest of the files (especially those linked against a specific kernel version) will be out of sync... it probably won't boot properly (that was the alias thing you were referencing)... so the extroot media must be re-extrooted (for lack of a better term).
no worries... precision in language is hard but important. Mostly my fault for using the term "running" so loosely.
I can see that for the packages, but also config files? Not a big deal, I can copy them to /rwm (whatever that stands for ... LOL), then upgrade.
On the USB, yep - somehow wireless related. I whacked (deleted) the wireless config, rebooted, USB all good. Set up extroot, rebooted, still all good. Then went to wireless (GUI), did a scan, and immediately,
[ 154.387464] b43-phy0: Loading firmware version 666.2 (2011-02-23 01:15:07)
[ 154.395436] usb 1-1: USB disconnect, device number 2
Somehow that firmware load triggers the USB disconnect . Still digging on that one.
Dang! Seems to be quite consistent . I have tried different driver versions (forward to experimental, back to old-stable), no difference. USB is solid and stable without Wi-Fi enabled, but as soon as I do ... boom. USB disconnects. It doesn't trash the USB drive, as it's disconnected LOL. But the system is then non-functional. Have searched, but no luck so far finding how to fix this
Do you have a powered USB hub? If so, insert that between the USB port on the router and the usb drive.... based on the new experiments, it is quite possible that it is a power issue where the router is unable to provide enough power to keep the usb drive running properly when wifi is enabled. An external powered USB hub will power the usb stick directly, and the router will only need to make a data connection.
Alternatively, you can try a power supply that has greater current capabilities (same voltage, higher current rating) -- this will help if your current adapter is underpowered. It will not help if the internal power supply circuitry is the bottleneck.
Good idea! But ... . Inserted a powered hub, booted up, started Wi-Fi ...
[ 96.086633] usb 1-1: USB disconnect, device number 2
[ 96.092061] usb 1-1.3: USB disconnect, device number 3
Lost the powered hub, and usb-storage . Checked,
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-platform/2p, 480M
|__ Port 1: Dev 2, If 0, Class=, Driver=hub/4p, 480M
|__ Port 3: Dev 3, If 0, Class=, Driver=usb-storage, 480M
Yep, that's what went down (or really, disconnected) ... hub and thumb drive. What the heck?!?! So odd.
BTW, also noted,
I then have to reboot twice to get the USB overlay back. 1 reboot, and the overlay is from flash. Huh?
Why use UUID to identify the USB drive, vs. the device (i.e. /dev/sda1)? UUID changes after an update, device does not.
rwm ... just curious, but do you know what this acronym means.
But the core issue seems to still persists => fire up Wi-Fi (b43), and the USB disconnects. And seems to not be power related?
It is possible it is an internal power supply issue -- maybe the wifi is browning out the USB circuitry (even though we have now reduced the load on the USB side).
Do you have a larger power brick for your router?
I think it is to guarantee that only the desired device is mounted. If you have multiple USB sticks connected, the one that mounts as /dev/sda1 vs /dev/sda2 cannot be guaranteed. UUID ensures that there is no ambiguitiy.
Depends where it is written, but my guess is "read/write memory"
I don't . I checked, it's 12V, 2.5A ... and to get another, the dang plug size is always such a mess.
Agreed, that makes sense.
Could be . It's from the extroot instructions.
OK, so I had another idea ... perhaps try adding a USB Wi-Fi adapter to the powered USB hub? Use that instead of the onboard one, see if that makes a difference? Dumb idea?
I plugged one it, and of course ... snatched defeat from the jaws of victory . Tried adding the driver to the build ... too large then (of course, LOL). So tried to opkg install it, but got,
# opkg install kmod-rtl8812au-ct
Unknown package 'kmod-rtl8812au-ct'.
Collected errors:
* pkg_hash_check_unresolved: cannot find dependency kernel (= 5.15.137-1-c106017892b8556b8c982e72f2b3b7c9) for kmod-rtl8812au-ct
* pkg_hash_fetch_best_installation_candidate: Packages for kmod-rtl8812au-ct found, but incompatible with the architectures configured
* opkg_install_cmd: Cannot install package kmod-rtl8812au-ct.
Huh? Arrgh. Will keep digging. But thinking this may be a way to see if the onboard power is an issue?
OK, was able to copy over the ipkg file, get the driver installed ... rebooted, and a new wireless interface comes up! Whohoo . But - that radio is not working quite yet, scan for example fails.
Still digging, may need to install other drivers to get it to work?
. Inserted a powered hub, booted up, started Wi-Fi ...