Request for Best Practice for eMMC devices

I am in search of the easiest way to flash/upgrade of eMMC devices...

Is the PreInit problem with Overlay still actual with latest 21.02 ?

Is there any advice from advanced user of the best method for OpenWrt on eMMC devices ?

For Overlay : Is F2FS better for eMMC/SSD/USB ?
For Firmware : Is SquashFS better than ext4 ?
For partitions map : how can I manage a rootfs_data overlay on eMMC to easily manage FirstBoot and Upgrade of firmware ?
For auto-Upgrade : are the opkg extras sufficient to get a "full auto reinstall after upgrade" of user installed packages ?

I am very interested of any full feedback and user experiences usage about these specific subjects.
Also I may propose the study of a solution proposal as Wiki and/or script/packaging...

This subject is complexified by not so easy to understand OpenWrt processes...
More I read from wiki and forum, less I understand the better solution to get to !

I actually use an ext4 rootfs clone on a third partition of eMMC, it is the most crashless I have experiment but it still require manual actions for any upgrade !
I also recommend this solution around, but I want to be sure there is no better and simple one !?

Thanks in advance...
Hope we will get more simple and efficient solution with this topic ! :wink:

block: unable to load configuration (fstab: Entry not found)

https://bugs.openwrt.org/index.php?do=details&task_id=2231

Still got the issue in 21.02.0 ! :tired_face:

@vgaetera
I like the Opkg Extras and its hotplug companion !
The two works fine to get an auto restore after upgrade of user installed packages...
But I get a problem of "auto broke" when I am using it on my extroot system.

Do you have encounter this issue by yourself ?
Is there any tweak to workaround ?

@all
I have read some f2fs recommendations for SD and I think it is also better for eMMC.
I have only tried ext4 system and ext4 extroot, mainly because I never get success with squashfs system and f2fs overlay !

I will get some more tests, for factory and for sysupgrade, still on the EspressoBin v7 eMMC, and will report here...

some feedback...
The problem of overlay is mainly an issue of the squashfs firmware ! (may be now look at it to find the reason)
There is now problem with the ext4 firmware !

I just get this procedure working with EspressoBinUltra and OpenWrt 21.02.0 EXT4, enhanced with a F2FS overlay...

Here the steps followed :

  1. Install necessary package after firstboot and flash...
opkg update
opkg install f2fs-tools kmod-fs-f2fs block-mount fdisk e2fsprogs kmod-fs-ext4
  1. Prepare overlay
fdisk /dev/mmcblk0
mkfs.f2fs -l rootfs_data /dev/mmcblk0p3
  1. Config overlay
DEVICE="/dev/mmcblk0p3"

eval $(block info ${DEVICE} | grep -o -e "UUID=\S*")
uci -q delete fstab.overlay
uci set fstab.overlay="mount"
uci set fstab.overlay.uuid="${UUID}"
uci set fstab.overlay.target="/overlay"
uci commit fstab
  1. Unnecessary step (because there is no overlay on ext4 firmware... and it will be auto prepared at firstboot !)
mkdir -p /tmp/cproot
mount --bind /overlay /tmp/cproot
mount ${DEVICE} /mnt
tar -C /tmp/cproot -cvf - . | tar -C /mnt -xf -	
umount /tmp/cproot /mnt
sync
  1. Test
reboot
  1. Verify
root@OpenWrt:/# df -h                                                                                                                                            
Filesystem                Size      Used Available Use% Mounted on                                                                                               
/dev/root               102.4M     14.1M     86.2M  14% /rom                                                                                                     
tmpfs                   496.7M     52.0K    496.6M   0% /tmp                                                                                                     
/dev/mmcblk0p3            7.2G    412.8M      6.8G   6% /overlay                                                                                                 
overlayfs:/overlay        7.2G    412.8M      6.8G   6% /                                                                                                        
tmpfs                   512.0K         0    512.0K   0% /dev                                                                                                     
root@OpenWrt:/# mount                                                                                                                                            
/dev/root on /rom type ext4 (rw,noatime)                                                                                                                         
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)                                                                                                         
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)                                                                                                        
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)                                                                              
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)                                                                                                               
/dev/mmcblk0p3 on /overlay type f2fs (rw,lazytime,relatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)                                                       
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)                                                                                                 
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)                                                                                 
debugfs on /sys/kernel/debug type debugfs (rw,noatime)                                                                                                           
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)                                                                                           
root@OpenWrt:/# ls /overlay/ -hal                                                                                                                                
drwxr-xr-x    5 root     root        4.0K Jan  1  1970 .                                                                                                         
drwxr-xr-x    1 root     root        3.4K Jan  1  1970 ..                                                                                                        
drwxr-xr-x    2 root     root        3.4K Jan  1  1970 etc                                                                                                       
drwxr-xr-x    2 root     root        3.4K Jan  1  1970 upper                                                                                                     
drwxr-xr-x    3 root     root        3.4K Jan  1  1970 work                                                                                                      
root@OpenWrt:/# ls /overlay/etc/ -hal                                                                                                                            
drwxr-xr-x    2 root     root        3.4K Jan  1  1970 .                                                                                                         
drwxr-xr-x    5 root     root        4.0K Jan  1  1970 ..                                                                                                        
-rw-rw-rw-    1 root     root          36 Jan  1  1970 .extroot-uuid                                                                                             

Reference : https://openwrt.org/docs/guide-user/additional-software/extroot_configuration

With SquashFS, step 4 done, but no success !

[   95.833528] F2FS-fs (mmcblk0p3): Mounted with checkpoint version = 37a41e9c                                                                                   
[   95.841664] block: extroot: unable to determine root device                                                                                                   
[   95.855239] mount_root: overlay filesystem has not been fully initialized yet                                                                                 
[   95.862824] mount_root: switching to f2fs overlay                                                                                                             
[   95.867885] mount_root: switching to f2fs failed - fallback to ramoverlay

Always stuck with ram overlay !!

This is the only part that still need to be automated for sysupgrade !

Question :
What do you think about adding a short third rootfs_data partition to the default ext4 firmware, and adding only a script to auto full resize the rootfs_data at firstboot ?
It may work after a first installation and also after a sysupgrade !

Then the installation of opkg and hotplug extras of @vgaetera will do the rest... :partying_face:

I'm thinking to use a separate profile to store the state of the original overlay.
It makes possible to restore the original overlay and extroot separately.
The automatic restore function can first restore the original profile.

The above part is more or less clear, but the following is vague.
Creating extroot requires manual action to select the proper target device.
Using old extroot is also problematic due to possible ABI changes.

1 Like

Will be great !

Sorry...

Yes, sure.
I am looking to automate most parts.

I experiment mostly on EspressoBin which have a 4Gb eMMC which is not fully used by the firmware.
It is why I look to add a third partition for overlay, and look to had it automated, to be upgrade friendly.

Sure, I prefer a reset at upgrade, may be we can automate this.

1 Like

I updated Opkg extras, and now it supports RWM/overlay/extroot setup.
It relies on the RWM mount to identify the Opkg profile to restore.

Assuming the block-mount package is not preinstalled by default.
The RWM mount should be defined, but not mounted on the initial boot.

The Hotplug script is supposed to restore the RWM profile and reboot.
Then with block-mount available, the RWM should be mounted.

Now we can reconfigure extroot to be mounted upon the next boot.
At the same time, the Hotplug script restores the main Opkg profile.

1 Like

Look great…

May be the wiki need some correction in ordering.
Steps 2 relate of DEVICE which is not already used.

For the script, I will test as soon as possible, because it may get to what I search for long !
I feel that they will need some few tweaks in my own case of ext4+overlay.
May be some questions soon, but be sure of feedbacks !

Again thanks for these big extras for OpenWrt.

Thanks again @vgaetera

I have made a f2fs overlay (with a gap between end of root partition and rootfs_data)

after reboots, and the restore of config, I get stuck in a boot loop :

[    8.239939] /dev/mmcblk0p3: Can't open blockdev

The default firmware do not include f2fs, so the overlay is not mount nor reset ?
I need to install block-mount and kmod-fs-f2fs.

I also like to enable the enabled_fsck option in fstab.

I will do some more tests...
I will share my notes also.

In my case there is no overlay at first (the overlay is automatically created when /overlay is mounted for the first time...) so the rwm partition is the same than the overlay (/dev/mmcblk0p3) !

A test from failsafe (+mount_root):

root@(none):/# sh /etc/uci-defaults/90-extroot-restore                                                                                                                                              
[   47.165442] /dev/mmcblk0p3: Can't open blockdev                                                                                                                                                  
cp: '/overlay/.' and '/mnt/.' are the same file                                                                                                                                                     
reboot
root@(none):/# ls /overlay/                                                                                                                                                                         
bak.AGMPBf  bak.EHOieO  bak.KBCLPO  bak.engLbA  bak.jFLEDC  bak.pkgcko                                                                                                                              
bak.AahleM  bak.EKGaDc  bak.ccHclI  bak.fBlkOp  bak.kPEEbD  etc                                                                                                                                     
bak.CGjmlD  bak.FmJkld  bak.dNEdje  bak.fBlogc  bak.kjboGG  upper                                                                                                                                   
bak.CoIlJI  bak.IkoCcO  bak.dcefAn  bak.hgHEej  bak.ncKMfM  work                                                                                                                                    
bak.DLACCL  bak.JkpMai  bak.dlpMoP  bak.hoeHLg  bak.oClhKB                                                                                                                                          
bak.DePfMN  bak.JmjkIl  bak.ekPCNi  bak.ieeEia  bak.oDEbcF
1 Like

You need to include the required packages to the RWM profile before upgrading:

uci set opkg.rwm="opkg"
uci add_list opkg.rwm.ipkg="block-mount"
uci add_list opkg.rwm.ipkg="kmod-fs-f2fs"
uci commit opkg

I added a condition to skip the copy command if no overlay is mounted by default.

Assuming the extroot/overlay was created before upgrading, it should be defined in fstab.
Otherwise the fstab prerequisite fails and the extroot-restore script just exits doing nothing.

The extroot/overlay is typically located on a separate block device.
So, it is supposed to survive an upgrade/reflash and preserve its UUID/filesystem.
Could you clarify how upgrading affects the old extroot/overlay partition in your case?

1 Like

Here what I do after a "clean" install to try to prepare my extroot (overlay in f2fs) :

opkg update
opkg install f2fs-tools                                                                                                                                                             
opkg install kmod-fs-f2fs                                                                                                                                                           
opkg install fdisk
opkg install block-mount

Then I prepare the rootfs_data

fdisk a third partition far from the end of the second (rootfs)...

Then I format and add the overlay

mkfs.f2fs -l rootfs_data /dev/mmcblk0p3
DEVICE="/dev/mmcblk0p3"                                                                                                                                                              
eval $(block info ${DEVICE} | grep -o -e "UUID=\S*")                                                                                                                                 
uci -q delete fstab.overlay                                                                                                                                                                         
uci set fstab.overlay="mount"                                                                                                                                                                       
uci set fstab.overlay.uuid="${UUID}"                                                                                                                                                                
uci set fstab.overlay.target="/overlay"                                                                                                                                                             
uci commit fstab

Add extras (from @vgaetera ;-))

opkg update
opkg install libustream-mbedtls

uclient-fetch -O opkg-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/opkg_extras?codeblock=0"
. ./opkg-extras.sh

uclient-fetch -O uci-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/uci_extras?codeblock=0"
. ./uci-extras.sh

uclient-fetch -O hotplug-extras.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/hotplug_extras?codeblock=0"
. ./hotplug-extras.sh

Add extroot-restore :

cat << "EOF" > /etc/uci-defaults/90-extroot-restore
if [ ! -e /etc/extroot-restore ] \
&& lock -n /var/lock/extroot-restore \
&& uci -q get fstab.overlay > /dev/null \
&& block info > /dev/null
then
OVR_UUID="$(uci -q get fstab.overlay.uuid)"
OVR_DEV="$(block info | sed -n -e "/${OVR_UUID}/s/:.*$//p")"
mount "${OVR_DEV}" /mnt
OVR_BAK="$(mktemp -d -p /mnt -t bak.XXXXXX)"
mv -f /mnt/etc /mnt/upper "${OVR_BAK}"
cp -f -a /overlay/. /mnt
umount /mnt
rm -f /etc/opkg-restore
touch /etc/extroot-restore
lock -u /var/lock/extroot-restore
reboot
fi
exit 1
EOF
cat << "EOF" >> /etc/sysupgrade.conf
/etc/uci-defaults/90-extroot-restore
EOF

perfect !

I have done this :

uci set opkg.rwm="opkg"
uci add_list opkg.rwm.ipkg="block-mount"
uci add_list opkg.rwm.ipkg="kmod-fs-f2fs"
uci add_list opkg.rwm.ipkg="f2fs-tools"
uci commit opkg
reboot !

And use LuCI to upgrade (for testing...)

I just modify the extroot-restore to fit my need (because I have no rwm partition)

root@OpenWrt:/# cat /etc/hotplug.d/online/50-opkg-restore

if [ ! -e /etc/opkg-restore ] \                                                                                                                                                                     
&& lock -n /var/lock/opkg-restore \                                                                                                                                                                 
&& opkg update                                                                                                                                                                                      
then . /etc/profile.d/opkg.sh                                                                                                                                                                       
if uci -q get fstab.overlay > /dev/null \                                                                                                                                                           
&& ! grep -q -e "\s/overlay\s" /etc/mtab                                                                                                                                                            
then opkg restore rwm                                                                                                                                                                               
else opkg restore                                                                                                                                                                                   
fi                                                                                                                                                                                                  
touch /etc/opkg-restore                                                                                                                                                                             
lock -u /var/lock/opkg-restore                                                                                                                                                                      
reboot                                                                                                                                                                                              
fi

Then I reboot and reflash from LuCI !
After a reboot for upgrade and another for restore :

root@OpenWrt:/# block info                                                                                                                                                                                                
/dev/mmcblk0p1: UUID="84173db5-fa99-e35a-95c6-28613cc79ea9" LABEL="kernel" VERSION="1.0" TYPE="ext4"                                                                                                
/dev/mmcblk0p2: UUID="ff313567-e9f1-5a5d-9895-3ba130b4a864" LABEL="rootfs" VERSION="1.0" MOUNT="/rom" TYPE="ext4"                                                                                   
/dev/mmcblk0p3: UUID="f563e894-354b-4248-b8f3-09d0578c1dd1" LABEL="rootfs_data" VERSION="1.14" MOUNT="/overlay" TYPE="f2fs"
root@OpenWrt:/# mount                                                                                                                                                                               
/dev/root on /rom type ext4 (rw,noatime)                                                                                                                                                            
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)                                                                                                                                            
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)                                                                                                                                           
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)                                                                                                                 
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)                                                                                                                                                  
/dev/mmcblk0p3 on /overlay type f2fs (rw,lazytime,relatime,background_gc=on,discard,no_heap,user_xattr,inline_xattr,inline_data,inline_dentry,flush_merge,extent_cache,mode=adaptive,active_logs=6,)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)                                                                                          
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)                                                                                                                                    
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)                                                                                                                    
debugfs on /sys/kernel/debug type debugfs (rw,noatime)                                                                                                                                              
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
root@OpenWrt:/# df -h                                                                                                                                                                               
Filesystem                Size      Used Available Use% Mounted on                                                                                                                                  
/dev/root               102.4M     12.9M     87.3M  13% /rom                                                                                                                                        
tmpfs                   496.7M     56.0K    496.6M   0% /tmp                                                                                                                                        
/dev/mmcblk0p3            6.8G    397.7M      6.4G   6% /overlay                                                                                                                                    
overlayfs:/overlay        6.8G    397.7M      6.4G   6% /                                                                                                                                           
tmpfs                   512.0K         0    512.0K   0% /dev
root@OpenWrt:/# ls /overlay/                                                                                                                                                                        
bak.pbbahk  etc         upper       work

Need some tweak, maybe, but already a great step !
Thanks...
With a first init script, we get close to user-friendly autoupgrade !!!

The third partition will survive also...
As I do no more reformat it, the UUID do not change.
The Filesystem also.
I labelled it "rootfs_data", which look to help the system to prepare it !
I also use now the F2FS filesystem...
But I have never had to clone the original one, because there is no one, and also because the first auto mount by block-mount, will do the job...
Then the extroot-restore, modified to handle overlay instead of rwm, will works fine...

Just have to configure the required package in the rwm profile, as you proposed !

:+1: :partying_face:

1 Like

I give a try to include all extras and get a first install overlay auto prepared :

extroot-restore and extroot-init get stuck...
I suppose I have to manage or take care of each locks...
I suppose also to have a bad lock handle in my extroot-init, I do not touch the /etc lock file after an upgrade, when the overlay is already prepared, but still unmounted !

Or, better, put extroot-init and extroot-restore all in one script...

1 Like

looks like to be OK simply by adding a test lock for opkg update in opkg_restore !...

I have to do some more tests...

@vgaetera If you have time to look at the script...
What do you want as (C) mention in header ?

What do you think of a packaging of this ?
After more debug... more tests...

1 Like