Extroot not working

[atmz]

Hi all, tried everything on extroot configuration but no luck. Appreciate any help!

This is a device with 2 USBs:

• 1x 2.0, plugged to a thumbdrive (wannabe extroot)
• 1x 3.0, plugged to a portable hdd (backup)

The thumbdrive was formatted as per extroot and can be mounted. However, upon router reboot it gets mounted to /rom/overlay but space doesn´t get added to /.

I can see there's things referring to /overlay and /rom/overlay, but don´t understand enough of the openwrt overlay process to try more risky things than already tried; this is the main house router and a brick would be a "crisis"

Output of df -h:

Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.0M      4.0M         0 100% /rom
tmpfs                   233.1M      9.8M    223.3M   4% /tmp
/dev/ubi0_1               7.2G     20.3M      6.8G   0% /rom/overlay
overlayfs:/overlay       24.1M      9.4M     13.4M  41% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sdb1                 7.2G     20.3M      6.8G   0% /rom/overlay

Output of mount:

/dev/root on /rom type squashfs (ro,relatime,errors=continue)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/ubi0_1 on /rom/overlay type ubifs (rw,noatime,assert=read-only,ubi=0,vol=1)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,noexec,noatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,noatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
bpffs on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
/dev/sdb1 on /rom/overlay type ext4 (rw,relatime)

Output of ubus call system board:

{
	"kernel": "5.15.137",
	"hostname": "router",
	"system": "ARMv7 Processor rev 0 (v7l)",
	"model": "Linksys EA8500 WiFi Router",
	"board_name": "linksys,ea8500",
	"rootfs_type": "squashfs",
	"release": {
		"distribution": "OpenWrt",
		"version": "23.05.2",
		"revision": "r23630-842932a63d",
		"target": "ipq806x/generic",
		"description": "OpenWrt 23.05.2 r23630-842932a63d"
	}
}

And output of block info; uci show fstab; logread | sed -n -e "/- preinit -/,/- init -/p":

/dev/ubiblock0_0: UUID="2d679df8-7a00ab1e-515e0e8c-dd964625" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/ubi0_1: UUID="f9833ee1-6c13-4243-997a-5c111e946d69" VERSION="w5r0" MOUNT="/rom/overlay" TYPE="ubifs"
/dev/sda1: UUID="556180e0-4dcf-4389-91a3-f8ec02eefc3d" LABEL="LINEXT4_4T" VERSION="1.0" TYPE="ext4"
/dev/sda2: UUID="715C902705595EDB" LABEL="NTFS_4T" TYPE="ntfs"
/dev/sdb1: UUID="d60ef562-121d-41eb-8436-cc3a0b11f12f" LABEL="extroot" VERSION="1.0" MOUNT="/rom/overlay" TYPE="ext4"
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='1'
fstab.@global[0].delay_root='15'
fstab.@global[0].check_fs='0'
fstab.extroot=mount
fstab.extroot.uuid='d60ef562-121d-41eb-8436-cc3a0b11f12f'
fstab.extroot.target='/rom/overlay'
fstab.rwm=mount
fstab.rwm.device='/dev/ubi0_1'
fstab.rwm.target='/rwm'
Wed Mar 20 17:57:48 2024 user.info kernel: [    5.377445] init: - preinit -
Wed Mar 20 17:57:48 2024 kern.info kernel: [    5.603069] usb 1-1: new high-speed USB device number 2 using xhci-hcd
Wed Mar 20 17:57:48 2024 kern.warn kernel: [    5.766202] mtdblock: MTD device 'devinfo' is NAND, please consider using UBI block devices instead.
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.454019] usb 4-1: new SuperSpeed USB device number 2 using xhci-hcd
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.853902] ipq806x-gmac-dwmac 37400000.ethernet eth1: Register MEM_TYPE_PAGE_POOL RxQ-0
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.854679] dwmac1000: Master AXI performs any burst length
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.861064] ipq806x-gmac-dwmac 37400000.ethernet eth1: No Safety Features support found
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.876439] ipq806x-gmac-dwmac 37400000.ethernet eth1: IEEE 1588-2008 Advanced Timestamp supported
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.876569] ipq806x-gmac-dwmac 37400000.ethernet eth1: registered PTP clock
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.884873] ipq806x-gmac-dwmac 37400000.ethernet eth1: configuring for fixed/sgmii link mode
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.891768] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Up - 1Gbps/Full - flow control off
Wed Mar 20 17:57:48 2024 kern.info kernel: [    6.900149] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.010869] UBIFS (ubi0:1): Mounting in unauthenticated mode
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.011016] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 204
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.066624] UBIFS (ubi0:1): recovery needed
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.195137] UBIFS (ubi0:1): recovery completed
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.195223] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.198468] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.206429] UBIFS (ubi0:1): FS size: 28061696 bytes (26 MiB, 221 LEBs), max 231 LEBs, journal size 1396736 bytes (1 MiB, 11 LEBs)
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.216324] UBIFS (ubi0:1): reserved for root: 1325422 bytes (1294 KiB)
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.227945] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID F9833EE1-6C13-4243-997A-5C111E946D69, small LPT model
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.243998] mount_root: loading kmods from internal overlay
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.400326] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Wed Mar 20 17:57:48 2024 kern.info kernel: [    9.535324] usb-storage 1-1:1.0: USB Mass Storage device detected
Wed Mar 20 17:57:48 2024 kern.info kernel: [    9.536057] scsi host1: usb-storage 1-1:1.0
Wed Mar 20 17:57:48 2024 kern.info kernel: [    9.540659] usbcore: registered new interface driver usb-storage
Wed Mar 20 17:57:48 2024 kern.info kernel: [    9.550320] scsi host2: uas
Wed Mar 20 17:57:48 2024 kern.info kernel: [    9.550886] usbcore: registered new interface driver uas
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.553458] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.553797] scsi 2:0:0:0: Direct-Access     Seagate  Expansion        9300 PQ: 0 ANSI: 6
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.571216] sd 2:0:0:0: [sda] 7814037167 512-byte logical blocks: (4.00 TB/3.64 TiB)
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.575626] sd 2:0:0:0: [sda] 4096-byte physical blocks
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.583525] sd 2:0:0:0: [sda] Write Protect is off
Wed Mar 20 17:57:48 2024 kern.debug kernel: [    9.588230] sd 2:0:0:0: [sda] Mode Sense: 4f 00 00 00
Wed Mar 20 17:57:48 2024 kern.notice kernel: [    9.588484] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Wed Mar 20 17:57:48 2024 kern.warn kernel: [    9.593461] sd 2:0:0:0: [sda] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.623263] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.627221] block: extroot: not configured
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.631137] mount_root: switching to ubifs overlay
Wed Mar 20 17:57:48 2024 user.warn kernel: [    9.644783] urandom-seed: Seeding with /etc/urandom.seed
Wed Mar 20 17:57:48 2024 kern.info kernel: [    9.720924] ipq806x-gmac-dwmac 37400000.ethernet eth1: Link is Down
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.780431] procd: - early -
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.780536] procd: - watchdog -
Wed Mar 20 17:57:48 2024 user.info kernel: [    9.782453] procd: Watchdog has previously reset the system
Wed Mar 20 17:57:48 2024 user.info kernel: [   10.363045] procd: - watchdog -
Wed Mar 20 17:57:48 2024 user.info kernel: [   10.363126] procd: Watchdog has previously reset the system
Wed Mar 20 17:57:48 2024 user.info kernel: [   10.368926] procd: - ubus -
Wed Mar 20 17:57:48 2024 user.info kernel: [   10.436124] procd: - init -

Again, thanks lots for any tips!

[psherman]

I only see /dev/sdb showing up...

Have you already completed the extroot process, or did you get stuck in the process?

For now, let's get that working. Don't plug in the portable HDD... only the thumbdrive (many reasons, we'll get to those later).

If you've completed the extroot process successfully, ideally the thumbdrive should mount as your overlay partition. If not, we'll start to troubleshoot.

Please help us understand where things stand currently.

[atmz]

Portable HDD unplugged, rebooted, no change.

Extroot process completed as per instructions, numbering of items is the same as that page:

1. Preparation done and all packages installed:

root@router:~# opkg list-installed | egrep -i "block-mount|kmod-fs-ext4|e2fsprogs|parted|kmod-usb-storage"
block-mount - 2023-02-28-bfe882d5-1
e2fsprogs - 1.47.0-2
kmod-fs-ext4 - 5.15.137-1
kmod-usb-storage - 5.15.137-1
kmod-usb-storage-uas - 5.15.137-1
libparted - 3.6-1
parted - 3.6-1

2. Partitioning and formatting done:

root@router:~# parted -l
Model: SanDisk Cruzer Facet (scsi)
Disk /dev/sda: 8004MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  8003MB  8002MB  ext4         extroot

root@router:~# blkid
/dev/ubi0_1: UUID="f9833ee1-6c13-4243-997a-5c111e946d69" TYPE="ubifs"
/dev/ubi0_0: BLOCK_SIZE="262144" TYPE="squashfs"
/dev/ubiblock0_0: BLOCK_SIZE="262144" TYPE="squashfs"
/dev/mtdblock16: UUID="1216310962" TYPE="ubi"
/dev/mtdblock14: UUID="1662173734" TYPE="ubi"
/dev/sda1: LABEL="extroot" UUID="d60ef562-121d-41eb-8436-cc3a0b11f12f" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="extroot" PARTUUID="bc1091aa-c35e-4b70-9cca-87a00fcdcf3d"
/dev/mtdblock17: UUID="2002990426" TYPE="ubi"

3. Extroot configured (see above uci show fstab output of fstab.extroot section)
4. Data is transferred to USB device and it's mounted in /rom/overlay as it should:

root@router:/rom/overlay# mount | grep sda
/dev/sda1 on /rom/overlay type ext4 (rw,relatime)
root@router:/rom/overlay# ls -al /rom/overlay
drwxr-xr-x    5 root     root          4096 Mar 20 16:01 .
drwxr-xr-x   16 root     root           223 Nov 15 00:38 ..
lrwxrwxrwx    1 root     root             1 Mar 20 16:01 .fs_state -> 2
drwx------    2 root     root         16384 Mar 20 16:00 lost+found
drwxr-xr-x   11 root     root          4096 Mar 20 23:07 upper
drwxr-xr-x    3 root     root          4096 Mar 20 16:01 work

5. rootfs_data/ ubifs configured (see above uci show fstab output of fstab.rwm section)

Other things I tried:

• delay_root="15" and up to 60
• there's no .extroot-uuid within the files
• adding export PREINIT=1 and mount_root to /etc/rc.local

The only thing I think of is the note about "OpenWrt is known to ignore the fstab configuration on devices without overlay partition in /proc/mtd". Looking at the issue on github looks like it may be already fixed in the latest snapshot

Can someone confirm if this looks like the issue I'm seeing? If it does I might as well try the snapshot to check for fixes

[psherman]

@atmz

/dev/sda1: LABEL="extroot" UUID="d60ef562-121d-41eb-8436-cc3a0b11f12f" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="extroot" PARTUUID="bc1091aa-c35e-4b70-9cca-87a00fcdcf3d"

At least /dev/sda1 is showing up now. It wasn't previously.

@atmz

The only thing I think of is the note about "OpenWrt is known to ignore the fstab configuration on devices without overlay partition in /proc/mtd". Looking at the issue on github looks like it may be already fixed in the latest snapshot

I have never experienced this issue personally, but I do recall helping in one or two other threads where the OP had this issue. Let me see if I can find the thread(s) to see if it is this issue (there was a workaround, I think... but don't remember exactly)

[psherman]

I found this one:

Failing typo in Extroot configuration

Hi all, I don't have write access to the wiki, but I'd like to get a modification done that is failing the procedure. change eval $(block info ${DEVICE} | grep -o -e 'UUID="\S*"') by eval $(block info ${USBDEVICE} | grep -o -e 'UUID="\S*"') change mount ${DEVICE} /mnt tar -C ${MOUNT} -cvf - . | tar -C /mnt -xf - reboot by mount ${USBDEVICE} /mnt tar -C ${MOUNT} -cvf - . | tar -C /mnt -xf - reboot

A small modification to the process fixed it for the OP, and the documentation was updated. Does this help?

[atmz]

I'm pretty confused... I've used the new documentation, so it should had worked if this was the issue I presume?

Double-checking with the old page revision before vgaetera's changes, the results would be the same either using the current version (which I did) or the old version with the proposed changes.

[psherman]

It might be worth erasing the extroot media and trying again... and you mentioned that the issue is probably fixed in snapshot, so you could always try that.

[atmz]

Maybe I'll try the snapshot, but have to find a time window where the router isn´t being used... hard.

Meanwhile, I'm trying to understand what should had happened.

What's the meaning/purpose of "rwm" in the below?

root@router:~# cat /etc/config/fstab 

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '15'
	option check_fs '0'

config mount 'extroot'
	option uuid 'd60ef562-121d-41eb-8436-cc3a0b11f12f'
	option target '/rom/overlay'

config mount 'rwm'
	option device '/dev/ubi0_1'
	option target '/rwm'

also, '/rom/overlay' behaviour is weird: the device assigned is always ubi0_1, doesn´t change even if I manually mount the usb stick (8gb) to it:

root@router:~# df /rom/overlay/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/ubi0_1            7592100     74108   7110876   1% /rom/overlay
root@router:~# umount /rom/overlay
root@router:~# df /rom/overlay/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/ubi0_1              24656      9744     13616  42% /rom/overlay
root@router:~# mount /dev/sda1 /rom/overlay
root@router:~# df /rom/overlay/
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/ubi0_1            7592100     74108   7110876   1% /rom/overlay

is that expected?

[psherman]

@atmz

also, '/rom/overlay' behaviour is weird:

that strikes me as odd, too. This is what I have for one of my devices (albeit not a ubi device):

root@OpenWrt:~# cat /etc/config/fstab
config 'global'
	option	anon_swap	'0'
	option	anon_mount	'0'
	option	auto_swap	'1'
	option	auto_mount	'1'
	option	delay_root	'5'
	option	check_fs	'0'

config 'mount'
	option	target	'/overlay'
	option	uuid	'e06a57b2-77b4-46c6-8b47-6ab8b3803731'
	option	enabled	'1'

[atmz]

Something must had been wrong with my config, maybe legacy values somewhere.

Using a factory image of 23.05.03 as well as 23.05.02 worked - whereas the same .02 version didn´t work before.

So I either messed up on creating the extroot (low chance as the commands are the same and I did multiple times throughout reboots and re-flashes), or there was something from olden days in my config somewhere.

Anyway, solved by re-flashing and reconfiguring from scratch, no backup restores etc.

[psherman]

Grest!

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.
Thanks!