Extroot configuration with nvme

cimbaklm · June 4, 2023, 6:22pm

Hello,

I used the note [https://openwrt.org/docs/guide-user/additional-software/extroot_configuration](extroot configuration) to extend the /overlay of my fresh installed BPI R3.

But as the configuration where everything goes fine, I rebooted by BPI and the mount point /overlay stay the same as befre, I mean a partition of the NAND of the BPI.
I installed kmod-nvme, configured a new partition on it and used the provided note to configure the extroot.

Here are some info from my conf

root@OpenWrt:~# lsblk
NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
mtdblock0        31:0    0 122.5M  0 disk
mtdblock1        31:1    0     2M  1 disk
mtdblock2        31:2    0     3M  0 disk
mtdblock3        31:3    0   512K  1 disk
mmcblk0         179:0    0   7.3G  0 disk
├─mmcblk0p1     179:1    0   512K  0 part
├─mmcblk0p2     179:2    0     2M  0 part
├─mmcblk0p3     179:3    0     4M  0 part
├─mmcblk0p4     179:4    0    32M  0 part
├─mmcblk0p5     179:5    0   104M  0 part
├─mmcblk0p65    259:0    0   4.9M  1 part /rom
├─mmcblk0p66    259:1    0  93.7M  0 part /overlay
└─mmcblk0p128   259:2    0     4M  0 part
mmcblk0boot0    179:8    0     4M  1 disk
mmcblk0boot1    179:16   0     4M  1 disk
ubiblock0_2     254:0    0  10.3M  0 disk
└─ubiblock0_2p1 259:3    0   4.9M  1 part
nvme0n1         259:4    0 119.2G  0 disk
└─nvme0n1p1     259:5    0 119.2G  0 part

block info

/dev/ubiblock0_2p1: UUID="e1467c8b-1ef6e438-87d691cb-75dbc264" VERSION="4.0" TYPE="squashfs"
/dev/ubi0_5: UUID="f19df63b-9847-465b-8831-480be62ce636" VERSION="w5r0" TYPE="ubifs"
/dev/mmcblk0p65: UUID="e1467c8b-1ef6e438-87d691cb-75dbc264" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mmcblk0p66: UUID="15781a90-e050-4d8b-b813-54b3e8e85b76" LABEL="rootfs_data" VERSION="1.16" MOUNT="/overlay" TYPE="f2fs"
/dev/nvme0n1p1: UUID="a4e7c094-9004-4d56-b066-1e2dd24ea59a" LABEL="extroot" VERSION="1.0" TYPE="ext4"

df

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.0M      5.0M         0 100% /rom
tmpfs                   996.4M     84.0K    996.3M   0% /tmp
/dev/mmcblk0p66          91.7M     70.9M     20.9M  77% /overlay
overlayfs:/overlay       91.7M     70.9M     20.9M  77% /
tmpfs                   512.0K         0    512.0K   0% /dev

Fstab config

root@OpenWrt:~# uci show fstab
fstab.@global[0]=global
fstab.@global[0].anon_swap='0'
fstab.@global[0].anon_mount='0'
fstab.@global[0].auto_swap='1'
fstab.@global[0].auto_mount='1'
fstab.@global[0].delay_root='5'
fstab.@global[0].check_fs='0'
fstab.@mount[0]=mount
fstab.@mount[0].target='/rom'
fstab.@mount[0].uuid='e1467c8b-1ef6e438-87d691cb-75dbc264'
fstab.@mount[0].enabled='0'
fstab.@mount[1]=mount
fstab.@mount[1].target='/overlay'
fstab.@mount[1].uuid='a4e7c094-9004-4d56-b066-1e2dd24ea59a'
fstab.@mount[1].enabled='0'
fstab.@mount[2]=mount
fstab.@mount[2].target='/rwm'
fstab.@mount[2].uuid='15781a90-e050-4d8b-b813-54b3e8e85b76'
fstab.@mount[2].enabled='0'
fstab.rwm=mount
fstab.rwm.device='/dev/mmcblk0p66'
fstab.rwm.target='/rwm'
fstab.overlay=mount
fstab.overlay.uuid='a4e7c094-9004-4d56-b066-1e2dd24ea59a'
fstab.overlay.target='/overlay'

Logs

Sun Jun  4 18:47:06 2023 user.info kernel: [    6.719238] init: - preinit -
Sun Jun  4 18:47:06 2023 kern.info kernel: [    6.863085] mtk_soc_eth 15100000.ethernet eth0: configuring for fixed/2500base-x link mode
Sun Jun  4 18:47:06 2023 kern.info kernel: [    6.871538] mtk_soc_eth 15100000.ethernet eth0: Link is Up - 2.5Gbps/Full - flow control rx/tx
Sun Jun  4 18:47:06 2023 kern.info kernel: [    6.879932] mt7530-mdio mdio-bus:1f lan1: configuring for phy/gmii link mode
Sun Jun  4 18:47:06 2023 kern.notice kernel: [    6.892810] random: procd: uninitialized urandom read (4 bytes read)
Sun Jun  4 18:47:06 2023 kern.info kernel: [    7.150316] usb 1-1: new high-speed USB device number 2 using xhci-mtk
Sun Jun  4 18:47:06 2023 kern.info kernel: [    7.330960] hub 1-1:1.0: USB hub found
Sun Jun  4 18:47:06 2023 kern.info kernel: [    7.334776] hub 1-1:1.0: 4 ports detected
Sun Jun  4 18:47:06 2023 kern.notice kernel: [    8.967815] F2FS-fs (mmcblk0p66): Mounted with checkpoint version = 6147d0f5
Sun Jun  4 18:47:06 2023 user.info kernel: [    8.976563] mount_root: loading kmods from internal overlay
Sun Jun  4 18:47:06 2023 user.info kernel: [    8.991021] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sun Jun  4 18:47:06 2023 kern.info kernel: [    9.002931] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Sun Jun  4 18:47:06 2023 kern.info kernel: [    9.010046] fsl-ehci: Freescale EHCI Host controller driver
Sun Jun  4 18:47:06 2023 kern.info kernel: [    9.016210] ehci-platform: EHCI generic platform driver
Sun Jun  4 18:47:06 2023 kern.info kernel: [    9.023157] usbcore: registered new interface driver usb-storage
Sun Jun  4 18:47:06 2023 user.info kernel: [    9.029347] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/*
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.160324] random: crng init done
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.401905] UBIFS (ubi0:5): Mounting in unauthenticated mode
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.407627] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" started, PID 690
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.910777] UBIFS (ubi0:5): UBIFS: mounted UBI device 0, volume 5, name "rootfs_data"
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.918593] UBIFS (ubi0:5): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.928488] UBIFS (ubi0:5): FS size: 90914816 bytes (86 MiB, 716 LEBs), max 727 LEBs, journal size 4571136 bytes (4 MiB, 36 LEBs)
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.940115] UBIFS (ubi0:5): reserved for root: 4294130 bytes (4193 KiB)
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   10.946713] UBIFS (ubi0:5): media format: w5/r0 (latest is w5/r0), UUID F19DF63B-9847-465B-8831-480BE62CE636, small LPT model
Sun Jun  4 18:47:06 2023 user.info kernel: [   10.965554] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
Sun Jun  4 18:47:06 2023 user.err kernel: [   10.992824] block: unable to load configuration (fstab: Entry not found)
Sun Jun  4 18:47:06 2023 user.info kernel: [   10.999544] block: attempting to load /tmp/ubifs_cfg/etc/config/fstab
Sun Jun  4 18:47:06 2023 user.err kernel: [   11.006010] block: unable to load configuration (fstab: Entry not found)
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.012722] block: attempting to load /etc/config/fstab
Sun Jun  4 18:47:06 2023 user.err kernel: [   11.017982] block: unable to load configuration (fstab: Entry not found)
Sun Jun  4 18:47:06 2023 user.err kernel: [   11.024687] block: no usable configuration
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   11.039651] UBIFS (ubi0:5): un-mount UBI device 0
Sun Jun  4 18:47:06 2023 kern.notice kernel: [   11.044358] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" stops
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.056417] mount_root: switching to f2fs overlay
Sun Jun  4 18:47:06 2023 kern.warn kernel: [   11.062544] overlayfs: null uuid detected in lower fs '/', falling back to xino=off,index=off,nfs_export=off.
Sun Jun  4 18:47:06 2023 user.debug kernel: [   11.074232] urandom-seed: Seeding with /etc/urandom.seed
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.096734] procd: - early -
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.099657] procd: - watchdog -
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.625609] procd: - watchdog -
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.629782] procd: - ubus -
Sun Jun  4 18:47:06 2023 user.info kernel: [   11.686416] procd: - init -

Thanks for your help

cimbaklm · June 5, 2023, 9:52am

Hello,
Can someone share a working extroot fstab configuration pease ?

I tried some things like preinit script to pre load nvme kernel module but none of them works..

Thanks

cimbaklm · June 5, 2023, 7:42pm

Hello,
After adding the kmod-nvme to the boot loaded modules, I have this error [

2111.087083] block: attempting to load /etc/config/fstab
[ 2111.142439] F2FS-fs (nvme0n1p1): Found nat_bits in checkpoint
[ 2111.443110] F2FS-fs (nvme0n1p1): Mounted with checkpoint version = 1c6df891
[ 2111.452304] block: extroot: unable to determine root device
[ 2111.479462] UBIFS (ubi0:5): un-mount UBI device 0
[ 2111.484178] UBIFS (ubi0:5): background thread "ubifs_bgt0_5" stops
[ 2111.495718] mount_root: rootfs_data:/dev/mmcblk0p66 is already mounted as /overlay

Can someone help me please ?
block: extroot: unable to determine root device

Thanks

VA1DER · June 5, 2023, 9:32pm

BPI-R3 stores its boot device configuration in the Linux device tree. You'll notice three entries later "/dev/mmcblk0p66 is already mounted as /overlay" - there is no actual setting for the overlay, that is in the device tree.

cimbaklm · June 6, 2023, 1:59pm

What do you mean by "there is no actual setting for the overlay, that is in the device tree"
How can I mount my extroot ?

Thanks

VA1DER · June 6, 2023, 8:08pm

There is no way to use extroot on a bpi-r3 right now. This is due to a bug in OpenWrt.

github.com/openwrt/openwrt

FS#2231 - fstools: PREINIT calling of block extroot doesn't acknowledge non-MTD rootfs overlays

opened 03:33PM - 09 Apr 19 UTC

openwrt-bot

flyspray

*knodel:* Extroot overlay mount fails on my ZyXEL NBG6817. I can confirm t…his bug on both the latest stable 18.06.2, and the snapshot from the 4th of April. When PREINIT calls 'block extroot', block fails to load the custom fstab from the eMMC ext4 overlay mounted at /tmp/overlay: ``` [...] Thu Apr 4 12:01:51 2019 user.info kernel: [ 3.508150] init: - preinit - Thu Apr 4 12:01:51 2019 kern.info kernel: [ 8.592387] EXT4-fs (loop0): recovery complete Thu Apr 4 12:01:51 2019 kern.info kernel: [ 8.595484] EXT4-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Thu Apr 4 12:01:51 2019 user.info kernel: [ 8.598875] mount_root: loading kmods from internal overlay Thu Apr 4 12:01:51 2019 user.info kernel: [ 8.649260] kmodloader: loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/* Thu Apr 4 12:01:51 2019 user.info kernel: [ 8.663223] kmodloader: done loading kernel modules from /tmp/overlay/upper/etc/modules-boot.d/* Thu Apr 4 12:01:51 2019 user.info kernel: [ 10.119972] block: attempting to load /etc/config/fstab Thu Apr 4 12:01:51 2019 user.err kernel: [ 10.120211] block: unable to load configuration (fstab: Entry not found) Thu Apr 4 12:01:51 2019 user.err kernel: [ 10.124238] block: no usable configuration Thu Apr 4 12:01:51 2019 user.info kernel: [ 10.132103] mount_root: switching to ext4 overlay Thu Apr 4 12:01:51 2019 user.info kernel: [ 10.292110] procd: - early - Thu Apr 4 12:01:51 2019 user.info kernel: [ 10.292264] procd: - watchdog - Thu Apr 4 12:01:51 2019 user.info kernel: [ 10.988093] procd: - watchdog - Thu Apr 4 12:01:51 2019 user.info kernel: [ 10.988475] procd: - ubus - Thu Apr 4 12:01:51 2019 user.info kernel: [ 11.046160] procd: - init - [...] ``` Unrelated dmesg entries omitted, full log here: https://gist.github.com/knuddelknoedel/2985ce7777a0263fbc22a02f8ef5307c Custom modules are loaded with the correct overlay /tmp/overlay/upper prefix by libfstools, however the forked 'block extroot' process behaves differently when searching for /etc/config/fstab configuration. Steps to reproduce: - Flash any current openwrt sysupgrade image on a device where the rootfs+overlay don't reside on MTD storage - Configure an appropriate /overlay uci fstab extroot entry as specified in the respective wiki documentation - Reboot Further notes: adding /etc/config/fstab with the desired /overlay entry to the sysupgrade squashfs image before flashing allows block to successfully find the uci fstab config, however the mounting of the therein configured /overlay mount still fails.

There is a fix, but it has not been committed to Master yet.

cimbaklm · June 8, 2023, 3:34pm

This bug is for Zyxtel hardware, is it the same issue for BPI ?

mk24 · June 8, 2023, 3:44pm

Rather than extroot, I would mount the large drive as a data disk. This will simplify upgrading.

cimbaklm · June 9, 2023, 8:17am

Yes but what I would like to do is to increase the usable space for package install
Mounting my NVMe as a data disk would not let me install packages on it.

cimbaklm · June 9, 2023, 2:58pm

Solved by using the imagebuilder and moduifying the overlay partition size. eMMC is large enough for package installations. Add partition on NVMe for datas

Thanks

Doom4535 · July 2, 2023, 12:23am

Wish I found your comment sooner, I've been going nuts trying to figure out how I didn't follow the wiki ext_root guide properly when setting up my generic x64 install with squash...

VA1DER · July 2, 2023, 3:14am

Heya Doom, welcome to the community. Sorry you didn't find it sooner.

For any x86-64 install you should be able to actually resize the overlay partition. How much actual space is on the unit's boot device?

Doom4535 · July 2, 2023, 4:33am

I ultimately just switched over to the non squash install and expanded the partition to fill the 40GB drive