Setting up /overlay on usb drive

rzighel · August 9, 2019, 4:22pm

Hello; I am trying to set OpenWRT on an external usb drive. I have been mostly successful except for one thing:

root@Confusion:/home# df
Filesystem 1K-blocks Used Available Use% Mounted on

- /dev/root xx. xx. xx. xx /rom/rom
- tmpfs xx xx xx xx /tmp
- /dev/sda1 xx xx xx xx /rom/overlay
- overlayfs:/overlay xx xx xx xx /rom
- ubi1:syscfg xx xx xx xx /tmp/syscfg
- tmpfs xx xx xx xx /dev
- /dev/sda1 xx xx xx xx /overlay
- overlayfs:/overlay xx xx xx xx /
- /dev/sda4 xx xx xx xx /home
  root@Confusion:/home#

I am pretty sure that lines 1, 3, and 4, should not be there. How do I correct that?

Thank you.

vgaetera · August 9, 2019, 7:58pm

https://openwrt.org/docs/guide-user/additional-software/extroot_configuration#troubleshooting
Analyze the boot log.

ahmar16 · August 9, 2019, 4:43pm

First of all you are putting coding info in basic text which makes it harder to read. You should use the </> icon to put shell output.

Moving on to your problem, I dont understand why /dev/root is mounted to /rom/rom it should be /rom. I think you may have mixed up the copying directories. I suggest you to reset the device to OpenWrt defaults and then use the above link as suggested by @vgaetera to properly configure ExtRoot on your device. Btw which device are you using?

rzighel · August 9, 2019, 4:51pm

The link mentioned is what I used to setup the extroot. I have done it twice, and I suspect that I skipped a step both times.

I can't figure out what step I may have skipped.

(Sorry for the hard to read text. I will use </> from now on).

root@Confusion:/rom/rom/rom# df
1. Filesystem           1K-blocks      Used Available Use% Mounted on
2. /dev/root                 2304      2304         0 100% /rom/rom
3. tmpfs                   255972       140    255832   0% /tmp
4. /dev/sda1             10255636     96656   9618308   1% /rom/overlay
5. overlayfs:/overlay    10255636     96656   9618308   1% /rom
6. ubi1:syscfg              30276       284     28412   1% /tmp/syscfg
7. tmpfs                      512         0       512   0% /dev
8. /dev/sda1             10255636     96656   9618308   1% /overlay
9. overlayfs:/overlay    10255636     96656   9618308   1% /
10. /dev/sda4            1806205244 143176820 1571208388   8% /home

ahmar16 · August 9, 2019, 5:07pm

Well I use my router with ExtRoot so I can just give you an overview on how to do it. I am considering that you have a default config for your router.

First you should identify if your USB is plugged in properly and for this try ls /dev/ and you should see all the partitions. Also make sure you have the correct filesystem installed for example Ext3. Then mount your usb partition with mount /dev/sda1 /mnt. sda1 is the first partition of USB. Edit the /etc/config/fstab to make sure everything runs on boot.
After that you need to copy your current /overlay to usb partition and for this use this command. tar -C /overlay -cvf - . | tar -C /mnt -xf - and once you are done then just umount /mnt and reboot to see the changes.

Edit: Please post the output of cat /etc/config/fstab here

rzighel · August 9, 2019, 5:32pm

cat /etc/config/fstab when running on original drive

root@OpenWrt:/etc/config# cat fstab 

config global
	option anon_swap '0'
	option anon_mount '0'
	option auto_swap '1'
	option auto_mount '1'
	option delay_root '5'
	option check_fs '0'

config mount
	option target '/overlay'
	option uuid '62b8bc5a-ab5b-4ecc-95f0-247758046964'
	option enabled '1'

config mount
	option target '/mnt/sda2'
	option uuid '671198b5-6458-4af0-8133-e4667e06a897'
	option enabled '0'

config swap
	option uuid 'a58d61c8-dc9b-483f-927e-a9cc7bb84ddc'
	option enabled '0'

config mount
	option target '/mnt/sda4'
	option uuid '02501e2d-5f01-4b9d-8f60-f4d414d5bacd'
	option enabled '1'

config mount 'rwm'
	option device '/dev/ubi0_1'
	option target '/rwm'

#config mount 'overlay'
#	option target '/overlay'

cat /etc/config/fstab when running on USB drive

root@OpenWrt:/mnt/upper/etc/config# cat fstab
config 'global'
	option	anon_swap	'0'
	option	anon_mount	'0'
	option	auto_swap	'1'
	option	auto_mount	'1'
	option	delay_root	'5'
	option	check_fs	'0'

config 'mount'
	option	target	'/overlay'
	option	uuid	'62b8bc5a-ab5b-4ecc-95f0-247758046964'
	option	enabled	'0'

config 'mount'
	option	target	'/mnt/sda2'
	option	uuid	'671198b5-6458-4af0-8133-e4667e06a897'
	option	enabled	'0'

config 'swap'
	option	uuid	'a58d61c8-dc9b-483f-927e-a9cc7bb84ddc'
	option	enabled	'0'

config 'mount'
	option	target	'/mnt/sda4'
	option	uuid	'02501e2d-5f01-4b9d-8f60-f4d414d5bacd'
	option	enabled	'1'

df on original:

root@OpenWrt:/etc/config# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 2304      2304         0 100% /rom
tmpfs                   255972        96    255876   0% /tmp
/dev/ubi0_1              24540      1836     21416   8% /overlay
overlayfs:/overlay       24540      1836     21416   8% /
ubi1:syscfg              30276       284     28412   1% /tmp/syscfg
tmpfs                      512         0       512   0% /dev
df: /mnt/sda4: No such file or directory
/dev/sda1             10255636     96660   9618304   1% /mnt
root@OpenWrt:/etc/config#

Also,I just noticed:
root@OpenWrt:/etc/config# ls /overlay/work/work/

Is it supposed to be work AND work?

Thank you.

ahmar16 · August 9, 2019, 6:00pm

It's only supposed to be /overlay/work. Somehow somewhere you are duplicating the direcory structure on your USB device. Have you cleaned up the USB partition? Make sure there's nothing in sda1 when you run the copy command. So you should first mount sda1 at /mnt and then issue rm /mnt/* to remove everything from the usb partition.

vgaetera · August 9, 2019, 6:16pm

logread | sed -n -e "/ preinit /,/ init /p"
block info

rzighel · August 10, 2019, 10:53am

On USB system:

root@Confusion:~# logread | sed -n -e "/ preinit /,/ init /p"
root@Confusion:~# 


root@Confusion:~# block info

/dev/mtdblock5: UUID="9464850" VERSION="1" TYPE="ubi"

/dev/mtdblock7: UUID="1604771158" VERSION="1" TYPE="ubi"

/dev/mtdblock8: UUID="1615526205" VERSION="1" TYPE="ubi"

/dev/ubiblock0_0: UUID="552f44be-88d432a7-08fdfb29-6b14a924" VERSION="4.0" MOUNT="/rom/rom" TYPE="squashfs"

/dev/ubi0_1: UUID="1a6a3db0-4714-4298-87ad-0c0d5882a86f" VERSION="w4r0" TYPE="ubifs"

/dev/ubi1_0: UUID="e1196807-b9c7-4d2e-aeb4-97f23ba0af38" VERSION="w4r0" TYPE="ubifs"

/dev/sda1: UUID="62b8bc5a-ab5b-4ecc-95f0-247758046964" VERSION="1.0" MOUNT="/rom/overlay" TYPE="ext4"

/dev/sda2: UUID="671198b5-6458-4af0-8133-e4667e06a897" VERSION="1.0" TYPE="ext4"

/dev/sda3: UUID="a58d61c8-dc9b-483f-927e-a9cc7bb84ddc" VERSION="1" TYPE="swap"

/dev/sda4: UUID="02501e2d-5f01-4b9d-8f60-f4d414d5bacd" VERSION="1.0" MOUNT="/rom/mnt/sda4" TYPE="ext4"

rzighel · August 10, 2019, 10:53am

root@Confusion:~# dmesg | sed -n -e "/ preinit /,/ init /p"
[    3.054878] init: - preinit -
[    3.111456] scsi 2:0:0:0: Direct-Access     Seagate  Ultra Slim MT    0304 PQ: 0 ANSI: 6
[    3.119793] random: procd: uninitialized urandom read (4 bytes read)
[    3.120883] sd 2:0:0:0: [sda] Spinning up disk...
[    3.227021] mvneta f1070000.ethernet tmp0: renamed from eth0
[    3.272527] mvneta f1034000.ethernet eth0: renamed from eth1
[    3.312513] mvneta f1070000.ethernet eth1: renamed from tmp0
[    3.607112] random: jshn: uninitialized urandom read (4 bytes read)
[    3.624152] random: jshn: uninitialized urandom read (4 bytes read)
[    3.800736] mvneta f1034000.ethernet eth0: configuring for fixed/sgmii link mode
[    3.808575] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    3.814632] mvneta f1034000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[    3.822572] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    4.151145] .
[    5.191144] .
[    6.231144] .
[    6.903954] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 821
[    6.961544] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
[    6.969410] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[    6.979378] UBIFS (ubi0:1): FS size: 27934720 bytes (26 MiB, 220 LEBs), journal size 1396736 bytes (1 MiB, 11 LEBs)
[    6.989861] UBIFS (ubi0:1): reserved for root: 1319425 bytes (1288 KiB)
[    6.996508] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 1A6A3DB0-4714-4298-87AD-0C0D5882A86F, small LPT model
[    7.011733] mount_root: loading kmods from internal overlay
[    7.030102] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    7.055057] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    7.061524] ohci-platform: OHCI generic platform driver
[    7.068956] uhci_hcd: USB Universal Host Controller Interface driver
[    7.075662] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
[    7.146636] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
[    7.155224] block: extroot: device not present, retrying in 15 seconds
[    7.271143] .
[    7.271238] ready
[    7.274874] sd 2:0:0:0: [sda] 3907029167 512-byte logical blocks: (2.00 TB/1.82 TiB)
[    7.314330] sd 2:0:0:0: [sda] Write Protect is off
[    7.319142] sd 2:0:0:0: [sda] Mode Sense: 4f 00 00 00
[    7.319335] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    7.354563]  sda: sda1 sda2 sda3 sda4
[    7.359310] sd 2:0:0:0: [sda] Attached SCSI disk
[   22.866051] EXT4-fs (sda1): 1 orphan inode deleted
[   22.870863] EXT4-fs (sda1): recovery complete
[   22.967167] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
[   22.987074] random: crng init done
root@Confusion:~#

vgaetera · August 9, 2019, 7:28pm

Does this happen upon every boot?

mk24 · August 9, 2019, 7:42pm

The typical extroot is to copy the entire rootfs and overlay files to one filesystem on the external drive, then mount it at /, completely abandoning the internal flash after initial bootup. Since this entire new root partition is equally writeable, there is no concept of an overlay any more.

And yes the new filesystem is improperly prepared with an extra directory level, it would be best to erase it and make a new copy.

rzighel · August 9, 2019, 7:42pm

I rebooted right after your question:

[    7.452300] sd 2:0:0:0: [sda] Attached SCSI disk
[   22.387301] EXT4-fs (sda1): 1 orphan inode deleted
[   22.392114] EXT4-fs (sda1): recovery complete
[   22.488416] EXT4-fs (sda1): mounted filesystem with or

So, it happened in previous reboot and in this one.

rzighel · August 9, 2019, 8:04pm

root@OpenWrt:~# mount /dev/sda1 /mnt
root@OpenWrt:~# cd /mnt
root@OpenWrt:/mnt# ls
etc    upper  work
root@OpenWrt:/mnt# pwd
/mnt

I do not follow. I followed the instruction on the site (twice) with the same result. The site says:

mount /dev/sda1 /mnt
cp -a -f /overlay/. /mnt
umount /mnt

and this is my new 'root' drive:

root@OpenWrt:~# mount /dev/sda1 /mnt
root@OpenWrt:~# cd /mnt
root@OpenWrt:/mnt# ls
etc    upper  work
root@OpenWrt:/mnt# pwd
/mnt

I did copy the whole thing into the roof of the new partition.

rzighel · August 9, 2019, 11:08pm

I am back.

Reformated the /dev/sda1 drive, follow the instructions on the link (https://openwrt.org/docs/guide-user/additional-software/extroot_configuration).

I am back at square 1; although I tried this:

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 2304      2304         0 100% /rom/rom
tmpfs                   255972        96    255876   0% /tmp
/dev/sda1             10255636     39540   9675424   0% /rom/overlay
overlayfs:/overlay    10255636     39540   9675424   0% /rom
ubi1:syscfg              30276       284     28412   1% /tmp/syscfg
tmpfs                      512         0       512   0% /dev
/dev/ubi0_1              24540      1440     21812   6% /rom/rwm
/dev/sda1             10255636     39540   9675424   0% /overlay
overlayfs:/overlay    10255636     39540   9675424   0% /

root@OpenWrt:~# umount /rom/rom
root@OpenWrt:~# umount /rom/overlay
root@OpenWrt:~# umount /rom/rwm

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                   255972        96    255876   0% /tmp
overlayfs:/overlay    10255636     39540   9675424   0% /rom
ubi1:syscfg              30276       284     28412   1% /tmp/syscfg
tmpfs                      512         0       512   0% /dev
/dev/sda1             10255636     39540   9675424   0% /overlay
overlayfs:/overlay    10255636     39540   9675424   0% /
root@OpenWrt:~#

Every filesystem mentioned in /etc/config/fstab gets mounted over /rom.

vgaetera · August 9, 2019, 11:22pm

It seems your issue is hardware-specific or firmware-version-specific.
I use the same method for OpenWrt 18.06.4 on D-Link DIR-620 D1 and it works just fine.

rzighel · August 9, 2019, 11:23pm

I noticed something in the process...

grep -e rootfs_data /proc/mtd. !returns nothing on original system.

/proc/mtab shows:

grep -e rootfs_data /proc/mtd
root@OpenWrt:/rwm/upper# cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00200000 00020000 "u-boot"
mtd1: 00040000 00020000 "u_env"
mtd2: 00040000 00020000 "s_env"
mtd3: 00100000 00020000 "devinfo"
mtd4: 02800000 00020000 "kernel1"
mtd5: 02200000 00020000 "ubi"
mtd6: 02800000 00020000 "kernel2"
mtd7: 02200000 00020000 "rootfs2".   <<<<---- This is where root was located.
mtd8: 02600000 00020000 "syscfg"
mtd9: 00680000 00020000 "unused_area"

vgaetera · August 9, 2019, 11:33pm

Perhaps the built-in extroot function on the preinit stage doesn't know how to properly handle this flash layout.

rzighel · August 9, 2019, 11:36pm

And I found the problem

In order to get new partition to work, I had to:

root@OpenWrt:~# cat /etc/rc.local 
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.

#export PREINIT=1
#mount_root

exit 0
root@OpenWrt:~#

Somehow (don't ask me) rc.local was copied from original partition to usb partition.

system · August 19, 2019, 11:34pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.