Create/Define extroot in custom image

Grommish · May 22, 2019, 6:17am

First, I should say that I've gone through Extroot configuration and Overlay FS support (how to enable overlay for new device). They all deal with an individual image after it's flashed, not how to configure for one in development of the image.

I'm building OpenWrt from source on an unsupported device. Currently, everything is being baked into the image, but without an extroot, changes made on-device don't survive a reboot.

The image does not create an rootfs_data partition.
The image does not have a populated /etc/fstab

root@OpenWrt:/# cat /proc/mtd
dev:    size   erasesize  name
root@OpenWrt:/# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
root@OpenWrt:/#

Below is the fdisk output.

Disk /dev/mmcblk1: 3688 MB, 3867148288 bytes, 7553024 sectors
7553024 cylinders, 1 heads, 1 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device       Boot StartCHS    EndCHS        StartLBA     EndLBA    Sectors  Size Id Type
/dev/mmcblk1p1 *  0,0,1       0,0,1             2048    2099200    2097153 1024M  c Win95 FAT32 (LBA)
/dev/mmcblk1p2    0,0,1       1023,0,1       2101248    3842047    1740800  850M 83 Linux
/dev/mmcblk1p3    0,0,1       1023,0,1       3842048    5582847    1740800  850M 83 Linux
/dev/mmcblk1p4    0,0,1       1023,0,1       5582848    7323647    1740800  850M 83 Linux
Disk /dev/mmcblk1boot1: 2 MB, 2097152 bytes, 4096 sectors
64 cylinders, 4 heads, 16 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Disk /dev/mmcblk1boot1 doesn't contain a valid partition table
Disk /dev/mmcblk1boot0: 2 MB, 2097152 bytes, 4096 sectors
64 cylinders, 4 heads, 16 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Disk /dev/mmcblk1boot0 doesn't contain a valid partition table

I'm looking to host the /extroot on /dev/mmcblk1p3, and I've already formatted the partition as f2fs (I like the idea of the flash-friendly aspect given the onboard mmc (non-removable)

If anyone can walk me through this, or perhapse point me in the right direction, I'd appreciate the help!

jeff · May 22, 2019, 12:05pm

Extroot is to add an overlay.

Sounds like what you want is to modify the kernel to mount root from the MMC directly. The DTS for the device or boot loader args, if not munged / ignored, is where I’d make the change.

Grommish · May 22, 2019, 2:03pm

And here is where my ignorance of OpenWrt starts to cause problems.

Let me explain what I'm trying to accomplish.

I've got an itus Network's Shield, which was manufactured in 2015. The company went under in 2016, but they used a one-off OpenWrt image for the device. It is a IDS/IPS security device.

The device loads the .ELF image at boot from /dev/mmcblk1p1 as the rootfs (which .ELF file is determined by a front-panel switch)

It has 3 modes (router, bridge, gateway) and the image loads a specific /dev/mmcbpk1p[2-4] on boot as /extroot depending on the image that was loaded.

The rootfs is loaded into RAM, so nothing that is changed survives a reboot. From what I've been able to understand, the files were all stored in the /extroot mount.

[   33.474444] Freeing unused kernel memory: 48384K (ffffffff80930000 - ffffffff83870000)
Waiting for extroot dev : /dev/mmcblk0p2 : 0
Waiting for extroot dev : /dev/mmcblk0p2 : 1
Waiting for extroot dev : /dev/mmcblk0p2 : 2
Waiting for extroot dev : /dev/mmcblk0p2 : 3
Waiting for extroot dev : /dev/mmcblk0p2 : 4
Waiting for extroot dev : /dev/mmcblk0p2 : 5
Waiting for extroot dev : /dev/mmcblk0p2 : 6
Waiting for extroot dev : /dev/mmcblk0p2 : 7
Waiting for extroot dev : /dev/mmcblk0p2 : 8
Waiting for extroot dev : /dev/mmcblk0p2 : 9
Waiting for extroot dev : /dev/mmcblk0p2 : 10
Waiting for extroot dev : /dev/mmcblk0p2 : 11
Waiting for extroot dev : /dev/mmcblk0p2 : 12
Waiting for extroot dev : /dev/mmcblk0p2 : 13
Waiting for extroot dev : /dev/mmcblk0p2 : 14
Waiting for extroot dev : /dev/mmcblk0p2 : 15
[   49.921880] mmc1: BKOPS_EN bit is not set
[   49.942550] mmc1: new high speed DDR MMC card at address 0001
[   49.961016] mmcblk0: mmc1:0001 P1XXXX 3.60 GiB 
[   49.977989] mmcblk0boot0: mmc1:0001 P1XXXX partition 1 2.00 MiB
[   49.996347] mmcblk0boot1: mmc1:0001 P1XXXX partition 2 2.00 MiB
[   50.014699] mmcblk0rpmb: mmc1:0001 P1XXXX partition 3 128 KiB
[   50.036667]  mmcblk0: p1 p2 p3 p4
[   50.057602]  mmcblk0boot1: unknown partition table
[   50.079113]  mmcblk0boot0: unknown partition table
Waiting for extroot dev : /dev/mmcblk0p2 : 16
Found /sys/block/mmcblk0/mmcblk0p2 : 17
Creating /dev/mmcblk0p2
Mounting /dev/mmcblk0p2 on /extroot
[   50.571788] kjournald starting.  Commit interval 5 seconds
[   50.595535] EXT3-fs (mmcblk0p2): using internal journal
[   50.612905] EXT3-fs (mmcblk0p2): mounted filesystem with writeback data mode
Found /extroot/init

Yes, the OEM image calls it /dev/mmcblk0 and the OpenWrt build I made uses /dev/mmcblk1 (just for clarity so someone doesn't tell me I'm using the wrong number on it)

I may still need to make a device tree blob (which will be a whole other post, I'm sure), but I need to figure out

how to mount the /dev/mmcblk1 mount on boot as rootfs storage/overlay so changes are saved.
Populate the mounted device if it's empty (I figure I can script this)

I really appreciate the help.

jeff · May 22, 2019, 2:13pm

The mount of the root file system is performed in very low-level code by early init. See, for example, https://elixir.bootlin.com/linux/latest/source/init/do_mounts.c

OpenWrt often highly modifies the command line that is used to be able to accommodate "uncooperative" boot loaders that pass the "wrong" arguments to the kernel. Running make target/linux/{clean,prepare} will let you see the patched kernel in ./build_dir/target/linux-<board specific>/linux-4.xx/

Without the source code of the OEM's modifications, it's hard to know just what they're doing from here.

In addition to getting the right MMC device selected as the root, you'll likely need the appropriate file system compiled into the kernel, so you don't have a Catch-22 in needing to mount the file system to load the driver to mount the file system ...

Grommish · May 22, 2019, 2:22pm

Well, the bootloader is u-boot, but it's only is used in Stages 1 and 2 for the device.. Stage 3 loads the image file (created by OpenWrt as bin/targets/octeon/generic/openwrt-octeon-generic-initramfs-kernel.bin), so it loads correctly. Where in the kernel would an extroot be outlined? Is it something I can do from make kernel_menuconfig, or maybe in a baked in init script itself? If the init script for mounting the extroot was in the image, the script would be persistent even through reboots. Hmm, what are your thoughts on something like that? Putting an init.d script in the files/etc/init.d/boot or would that be too late in the boot?

Grommish · May 29, 2019, 12:21pm

I've run into an odd wall. I decided to see if I could just script the rootfs_data mount and handle it the way I need/want it to, but....

When I made the /etc/rc.d and /etc/init.d scripts, it seem during the ENTIRE boot process that /dev/mmcblk1 isn't available! I can't seem to figure out when it actually gets mounted.

I've set the scripts at S11, S99 and rc.local and nothing. I've posted over at StackExchange and while they've offered suggestions, I think it's my lack of OpenWrt knowledge that is holding me back.

Any suggestions?