EDIT:
I've given the solution in the ExtRoot wiki page.
Read on if you are interested in my dull story of how I came to that result.

ORIGINAL STORY:
I did prepare a miniSDHC for the "pivot-overlay" type of ExtRoot as per instructions.
The card sits in a slot of an USB-modem (Huawei E3131).

I tried to mount this device as per instructions above - from the /etc/config/fstab - but it failed. Increasing the delay_root global option up to 60 seconds does not help.

Then I followed Max Hopper's advise which has helped me to some extent (no need to sleep 10s in a script).

But still I do not see any volume increase on the /:

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                     896       244       652  27% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    14708        64     14644   0% /tmp
/dev/mtdblock6         7759872    477328   7221104   6% /overlay
overlayfs:/overlay         896       244       652  27% /
tmpfs                      512         0       512   0% /dev
/dev/sda1              7759872    477328   7221104   6% /overlay
/dev/sdb1              7759872    477328   7221104   6% /mnt/sda1
root@OpenWrt:~# 

May be I'm wrong, but I did expect to obtain the same 6% of occupancy on the /.
Without my "extrooting" I get the same 27% at /overlay (/dev/mountblock6), so it sort of "works" but not quite.

root@OpenWrt:~# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mtdblock6 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
/dev/sda1 on /overlay type f2fs (rw,relatime,background_gc=on,user_xattr,active_logs=6)
/dev/sdb1 on /mnt/sda1 type f2fs (rw,relatime,background_gc=on,user_xattr,active_logs=6)
root@OpenWrt:~# 

I did try to format the card to ext4 with the same (apparently failing) result.

Please comment.
Thank you in advance.

(Last edited by booBot on 21 Apr 2016, 23:05)

Please post your /etc/config/fstab.

Some observations:

• the router has 2X storage devices -

  • /dev/sda1

  • /dev/sdb1

• the mount point for /dev/sdb1 is (/mnt/sda1)

• rootfs is present (indicates extroot failed)

Boot the router and post the dmesg buffer.

Yo!
Thanks for the /dev/sdb1 observation!
I'm SO tired with this issue - I did not see the fact that the modem has a CDROM emulation as well.
But I'm sure it never appeared before.

I'll re-do all the preparations and post my results.

Thank you again!

The /dev/sdb1 there is an unexplainable one-off glitch I could not replicate any more.

1) Without any extroot my router shows:

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                     896       244       652  27% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    14708        64     14644   0% /tmp
/dev/mtdblock6             896       244       652  27% /overlay
overlayfs:/overlay         896       244       652  27% /
tmpfs                      512         0       512   0% /dev
/dev/sda1              7759872    477328   7221104   6% /mnt
root@OpenWrt:~# 

2) My best ever attempt to extroot gives:

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                     896       244       652  27% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    14708        64     14644   0% /tmp
/dev/mtdblock6         7759872    477328   7221104   6% /overlay
overlayfs:/overlay         896       244       652  27% /
tmpfs                      512         0       512   0% /dev
/dev/sda1              7759872    477328   7221104   6% /overlay
root@OpenWrt:~# 

This looks like exactly the example in the Step #6 of the pivot-overlay HowTo I'm following.
The problem is that I can't make any use of that half-baked extroot: opkg install always writes to JFFS2 partition - and always fails with "no more space left" error.

3) But today - by a happy accident - I let the router to boot into the failsafe mode with USB-modem/miniSDHC attached.
I telnetted in, typed mount_root, and lo and behold! I see this:

root@OpenWrt:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 7759872    477328   7221104   6% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    14708        64     14644   0% /tmp
/dev/mtdblock6         7759872    477328   7221104   6% /overlay
overlayfs:/overlay     7759872    477328   7221104   6% /
tmpfs                      512         0       512   0% /dev
/dev/sda1              7759872    477328   7221104   6% /overlay
root@OpenWrt:~# 

That's how I always imagined a really-working extroot would look like. See the 6% / at the rootfs line there.

I think I can just pause here - I can install all the additional packages I need for my project while in this failsafe+extroot, and then let the router to boot normally just mounting /dev/sda1 to /overlay with Max's advise.

Please comment if the Step #6 of the pivot-overlay HowTo shows a misleading df output as if the extroot is really working. As to me - it does not.

(Last edited by booBot on 6 Apr 2016, 21:55)

Hello,
I can smell a space for contribution to the page https://forum.openwrt.org/viewtopic.php?id=63891.
Nicely done!

(Last edited by mucha on 6 Apr 2016, 22:21)

mucha, far too early to say it's "nicely done" - my effort is not completed yet.
I've updated the instructions page, please have a look at it.

Comments and suggestions are much welcome!

Being almost ready to give up with extroot attempts - I decided to try it with a plain 2GB USB-flash.

I've formatted it to ext4, copied the /overlay there, adjusted the /etc/config/fstab to mount it to /overlay.
Rebooted my router.
Hoorray! I came up with the normally working pivot-overlay type of ExtRoot: I see all 2GB available on / and /overlay.

Now I tried to format that USB-flash to f2fs - and repeated the ExtRoot attempt - it failed.

Then I tried to format the miniSDHC in the USB-modem slot to ext4 - just as before - it did not work...

Right now I can conclude two things:

1. ExtRoot requires ext4 filesystem on the external storage;

2. a miniSDHC in a slot of an USB device is not exactly the same thing as a plain USB-storage - and because of that ExtRoot does not support such things "out of the box".

I do not know how to proceed further.

May be I should try an USB hub to connect the USB-modem (without a miniSDHC in it) and a plain USB-flash disk?

(Last edited by booBot on 13 Apr 2016, 11:53)

I think that usb_modeswitch is the one messing with USB modem's builtin SDcard reader to work.

one thing that I had done back then is I sent some AT commands to force my usb modem to be in modem mode and sdcard mode. so usb_modeswitch wont trigger. with that I had my modem in modem mode right away and Sd card accessible.

by default, usb modems is in cdrom and sdcard mode and modem mode is deactivated since the drivers/dashboard is not installed. once the drivers is installed, the modem needs to reset everytime to access modem mode everytime you plug it in or restart your computer, and this affects also the memory card access and cdrom to be like unplugeed and replugeed (virtually).

Incorrect.  USB FSDs used here are ext2 format for /overlay (journalling contributes to 'wearing').

I doubt it is the usb-modeswitch's fault here.
If I set option enabled 0 - when I can get into my router - and I can see my 8GB sda1 device being discovered during the boot time (in the logread) after which there is the "extroot not enabled, configuring jffs2 for overlay" message.

Max, I meant that F2FS is NOT supported in ExtRoot properly. And this comes from somewhere deeper than the block utility alone - which I have patched to accept F2FS-formated storages.
May be this is because F2FS is not yet supported by linux in general (at least not in Lubuntu 14.04.4).

Created a ticket#22217.

Are you mounting your overlay partition by UUID or device? If you have more than one device, you should be mounting your overlay partition by UUID only as is shown in the Wiki since you can not guarantee which device will be assigned /dev/sda and which device will be assigned /dev/sdb.

Journalling file systems recovery times are much quicker after a power outage which is why I use ext4 on my travel routers. I have yet to wear out an SSD or SD card using ext4. Of course I don't do a lot kernel builds and other activities which result in a lot of writes.

Really. ±2 seconds is too long to wait? The AP itself requires ±15 seconds to initialise.

And whilst 'battling the road' you experience power cuts how often? And further, what class of critical data are you concerned with (on a travel router)?

I tried it either way - nothing works for me.

By the way - that ghostly /dev/sdb1 appeared exactly from that - my referencing of the device by its UUID (in the fstab) and then trying to mount it by its name manually.

Now I wonder if an USB-hub could help me to have the pivot-overlay to a plain USB-flash and to have my USB-modem available simultaneously? I do not like that idea (if it works at all) though...

Since ext2 does not contain a journal, fsck must traverse the entire file system after a power interrupt. The fsck time becomes dependant on cpu, disk capacity etc. On my 486 DX2 machine with an ext2 formatted 20GB hard drive, fsck  takes over 15 minutes after a power interrupt.

One of my travel routers experiences a power interrupt at least twice a day. The travel router's power is connected to my car ignition switch. The travel router is in my car's ash tray. It serves up music to my phone which then sends it to my car's bluetooth. It servers up music, tv shows, movies and books to passengers.

This is what works for me with my a5-v11 device: My extroot device is a 128GB Sandisk Fit partitioned as follows:

# fdisk -l /dev/sda

Disk /dev/sda: 115.7 GiB, 124218507264 bytes, 242614272 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x000bc81c

Device     Boot   Start       End   Sectors   Size Id Type
/dev/sda1          2048   4196351   4194304     2G 83 Linux
/dev/sda2       4196352   4327423    131072    64M 82 Linux swap / Solaris
/dev/sda3       4327424 242614271 238286848 113.6G 83 Linux

/dev/sda1 and /dev/sda3 are formatted at ext4 and /dev/sda2 is formatted as swap. I don't know if partition order matters.

My /etc/config/fstab on the device's 4MB flash is as follows.

cat /etc/config/fstab
config 'global'
    option    anon_swap    '0'
    option    anon_mount    '0'
    option    auto_swap    '1'
    option    auto_mount    '1'
    option    delay_root    '5'
    option    check_fs    '0'

config 'mount'
    option    target    '/overlay'
    option    uuid    'd48a14b2-6016-4cdf-bdda-6d7a1840a21a'
    option    enabled    '1'

My /etc/config/fstab on my extroot device, the 128GB Sandisk Fit is as follows:

 cat /etc/config/fstab 
config 'global'
    option    anon_swap    '0'
    option    anon_mount    '0'
    option    auto_swap    '1'
    option    auto_mount    '1'
    option    delay_root    '5'
    option    check_fs    '0'

config 'swap'
    option    uuid    '233ce3f0-cdd4-4e9c-9d7c-56fcd1547118'
    option    enabled    '1'

config 'mount'
    option    target    '/data'
    option    uuid    'b9514cec-e6d6-4d36-9555-1e00a39b304d'
    option    enabled    '1'

After boot up, the output of the df and mount command are as follows:

# mount
rootfs on / type rootfs (rw)
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/sda1 on /overlay type ext4 (rw,relatime,data=ordered)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600)
/dev/sda3 on /data type ext4 (rw,relatime,data=ordered)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)


# df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                 1998672     17148   1860284   1% /
/dev/root                 2048      2048         0 100% /rom
tmpfs                    14612        84     14528   1% /tmp
/dev/sda1              1998672     17148   1860284   1% /overlay
overlayfs:/overlay     1998672     17148   1860284   1% /
tmpfs                      512         0       512   0% /dev
/dev/sda3            117142908  87317172  23852184  79% /data

I assume you also followed the steps in the Wiki to copy all files in the devices /overlay directory to the extroot device.

Good Luck!

(Last edited by vernonjvs on 15 Apr 2016, 20:00)

Thanks for the support, vernonjvs.

Yes, if I extroot to a plain USB-flash storage - it works perfectly.
My problem is that extroot breaks on a miniSDHC sitting in a slot of an USB-modem. Seems. it does matter - a card in a reader behaves not exactly as USB-flash memory...

(Last edited by booBot on 15 Apr 2016, 21:58)

Sorry I was not able to help. I only have one idea left. It is not clear to me how much current your USB-Modem is drawing. According to here, it may me close to 500ma. If your OpenWrt device only has a USB 2.0 port, it may be getting close to its power delivery capability. This could be causing resets which may be why you wound up with a phantom /dev/sdb block device. You may be able to alleviate this with a power supply with more current capacity for your OpenWrt device. I know it is a lot of IFs, but its the only thing left I can think of.

Good Luck!

Thank you very much for the suggestion.

My router is Asus WL-330N3G which is designed to use USB-modems - they are its only WAN option.

Need for speed:
SDHC
FSD (A-mark devices are fastest)

Max, I don't think the [lack of] speed is THE problem here.
As I can see - the extroot IS initiated but it cannot completely set itself up properly.
The settings and configurations from the overlayed storage DO appear and ARE applied. Not all of them, unfortunately - that is the problem.

(Last edited by booBot on 16 Apr 2016, 11:00)

You are absolutely right!

I'm fighting it (the usb-modeswitch's messing with storages and file systems) - and hope to get to the solution. I'll post my results.

Without the usb-modeswitch I now get a properly working pivot overlay to a miniSDHC (formatted to ext4 - this is vitally important!) sitting in a modem's slot.
As soon as I try starting the usb-modeswitch - it breaks the already successfully mounted and working ExtRoot - even if I start it from the overlay itself!

(Last edited by booBot on 20 Apr 2016, 12:44)

Just closed that ticket because it has nothing to do with the ExtRoot and a card in a modem slot.

Right now I'm trying to work-around the usb-modeswitch issue: I've put my modem in the NCM mode which supposedly does not depend on usb-modeswitch.