Question on extroot configuration

I found the Extroot Configuration page to be confusing and jargon-laden. I took a stab at rewriting it, but have no idea if it's correct. So I appeal to the wisdom of our readers for guidance...

Here's the first part of the description. Is it correct as far as it goes? Thanks.

Updates:
5Feb2019 based on first set of comments. This is great! Please keep adding comments!


Extroot configuration

The procedure below describes how to use a storage device (USB stick, SATA drive, SD card, or other) to expand the root filesystem of your OpenWrt device. This external-memory root filesystem ("extroot") can be as large as the storage device you add. This allows you to install more packages than would otherwise be possible.

You will need to connect an external storage device to your router. The easiest way to do this is to connect a USB stick with at least 1GByte 256Mbytes.

Note: This procedure is designed to work with OpenWrt 17.01.5 or newer.

Note: Although many devices with 4MBytes of Flash can usually be expanded using this extroot procedure, the memory requirements for the base OpenWrt plus the additional packages required may exceed the 4Mbytes.

Note: Devices with 4MBytes of Flash frequently have only 32MBytes of RAM, which may still be too little for a useful, functional router. You will have to determine if this procedure will benefit you, or whether a router with > 4MBytes Flash/32 MBytes RAM would serve you better.

Background Info

In OpenWrt devices, the firmware supports two partitions of internal storage.

  • OpenWrt's "root" partition is a highly-compressed read-only partition, stored in the Flash memory from the original manufacturer. Since it's the root of the filesystem, it's stored at /.
  • OpenWrt supports a second (usually removable) "overlay" partition that is writable. It is at /overlay in the filesystem.

The overlay partition contains a full copy of the root filesystem. Because the overlay partition's device is larger than the original Flash memory, it also has space available for more packages.

When the router boots and detects an overlay partition, it mounts that overlay partition in place of the (former) root partition. Since it contains a copy of the original root partition, the router operates exactly as before, with the addition of any other packages that are present in the overlay partition.

This approach also allows easy fallback in case the external storage device is removed. The OpenWrt device still has its original root partition and will load its firmware and configuration from there.

Step 1

You need to install several packages to create an ext4 filesystem on the (new) overlay partition.

  1. Connect the USB stick/device to the router.

  2. SSH into the router and enter the following commands into the SSH window.

    opkg update && opkg install block-mount e2fsprogs kmod-fs-ext4 kmod-usb-storage kmod-usb2
    

    If you see any error messages, you may not have sufficient internal memory to support an extroot (true?) See ??? for more information on troubleshooting this.

Step 2

Discover the current set of devices on the router. Enter block info and you will see something like this:

root@OpenWrt:~# block info
/dev/mtdblock2: UUID="1748d829-b790e1c0-4aac514d-aa6fd187" VERSION="4.0" MOUNT="/rom" TYPE="squashfs"
/dev/mtdblock3: MOUNT="/overlay" TYPE="jffs2"
/dev/sda: UUID="b44cad51-40cd-4f96-b79e-2cdfbaf88d90" VERSION="1.0" MOUNT="/opt" TYPE="ext4"
/dev/sdb: UUID="2015-11-03-03-23-01-00" LABEL="TEST - 20151103" TYPE="iso9660"
/dev/sdb1: UUID="2015-11-03-03-23-01-00" LABEL="TEST - 20151103" TYPE="iso9660"
  • /dev/mtdblock devices are the internal partitions of the internal Flash memory
  • /dev/sdX devices are external USB devices. /dev/sda is the first USB device; /dev/sdb is a second device.
  • /dev/sda1 is the first partition on the /dev/sda device; /dev/sda2 is the second partition, etc.

That's as far as I got - correct so far? Thanks.

Why that? Where does that come from?

I just did this yesterday on a Tp-Link TL-WDR4300 that was just updated to 18.06.2 and here is what I installed.

opkg update && opkg install block-mount kmod-fs-ext4 kmod-usb-storage e2fsprogs kmod-usb-ohci kmod-usb-uhci fdisk nano

looking at what is installed.

root@OpenWrt:~# opkg list-installed | grep usb
kmod-usb-core - 4.9.152-1
kmod-usb-ehci - 4.9.152-1
kmod-usb-ledtrig-usbport - 4.9.152-1
kmod-usb-ohci - 4.9.152-1
kmod-usb-storage - 4.9.152-1
kmod-usb-uhci - 4.9.152-1
kmod-usb2 - 4.9.152-1

root@OpenWrt:~# opkg list-installed | grep fs
e2fsprogs - 1.44.1-1
fstools - 2018-12-28-af93f4b8-3
kmod-fs-ext4 - 4.9.152-1
libext2fs - 1.44.1-1

Mine worked fine, Good luck with your Tutorial and also any USB drive over a ~200 meg should be fine.

The caveat I would add is that modulo the size of your flash storage, ensure that you have the minimal set of tools required to debug your connection on your 'internal root' before you switch to the extroot.

In an upgrade/failure scenario you won't have the extroot - and you may still want to both get online and diagnose why you can't get online in order to pull further upgrades down.

@tmomas, @sml156, @cbz - Thanks for the comments - I updated the original post.

Changelog:

  • Changed to recommend 256MByte USB stick (down from 1GByte)
  • I'm looking for the minimal set of packages that will enable someone to create an extroot. Adding additional packages that are not required may exceed the internal storage.
  • Consequently, removed kmod-usb3 from the set of packages. A device that needs extroot won't have USB3...
  • Added a note to the opkg install section that talks about errors. Is the reason correct?
  • And if people have troubles installing, is there a good place to send people for further info?
  • Anything else?

Please keep reviewing this. Thanks!

Leads me to think that a note about when "extroot" is appropriate and when one might want to just mount an external drive (logging, packet capture, anything that repeatedly writes to "disk") may be valuable.

Given how blindly people follow the walk-throughs, it might also be valuable to include something like "While most devices that would benefit from extroot configuration do not support USB 3, if your device has hardware support for USB 3, install the kmod-usb3 module and its dependencies."

2 Likes

Is that actually true? I routinely make backups of /overlay and these are decidedly do not contain complete copies of all files in the rootfs, but only those that were actually changed. Now I believe that overlayfs will transparently reach into the underlay filesystem it sits on top of to deliver those files that have not been changed to be different from their version in the underlay. Effectively this means that overlayfs will contain the most recent version of the filesystem, in that sense I can see the "copy of the full root filesystem", but I am not sure whether that is the correct way to describe this, as IMHO "copy of" implies that a copy of /overlay should contain the most recent state of everything, no?
I probably misunderstand this completely though....

2 Likes

Even more good comments! I'm learning so much!

Warning: I just write the simplest possible explanation of what I have read on the wiki, etc. I'm just makin' up the rest - hence my plea for help!

@jeff - good thought about extroot vs "just needing extra space" - there's a quick start for adding a USB drive at: https://openwrt.org/playground/richb/quickstartusb (latest version)
a) perhaps the extroot should steer people who "just want more space" to that article, and vice versa
b) that quick start article needs careful review, as well.

@moeller0 - See? It really helps to have smart friends. Thank you for the additional insights - it makes the notion of "overlay" make more sense.

I'll let this thread collect more comments, then update my original post to showcase your wisdom.

Thanks again.

The overlay-over-overlay-over-ROM concept had my head spinning the first time I tried extroot. The "gotchas" for me were:

  • The on-flash overlay / ROM combination needs to:
    • Be self-sufficient for boot when the external drive is unavailable
    • Have all the tooling required to mount and make changes to the contents of the external drive
    • Have sufficient tooling available to repair the extroot, or at least be able to install them if needed
  • Changes to the on-flash overlay that happen as a result of, for examples, an upgrade or "fresh" reconfiguration, that have been overwritten (modified) in the external overlay are not reflected once the external overlay mounts (see second bullet, above)

I recall sysupgrade being something of a nightmare with an extroot configuration, but didn't find any of my notes on it very insightful.

Edit: Same goes for changes on the extroot as they aren't reflected back to the flash-based overlay, such as ISP configuration, LAN configuration, passwords, ....

1 Like

Oh, jeez... I hadn't even considered the sysupgrade problem...

I briefly I wondered if I am working at cross-purposes to one of my other (not very strongly-held) goals of discouraging the use of 4/32 devices, especially in the face of sub US$40 8/64 devices. Supporting those older devices had likely consumed a fair amount of developer time.

But 17.01.6 is unlikely to receive further updates, so I guess that I'd rather have clear and correct documentation for extroot so those older devices can remain useful instead of landfill.

as a note, assume someone have another linux machine to partion/format usb for router, we can remove e2fsprogs(for formating) or fdisk (partioning) if it needs opkg it may matters as each uses 500 KB, total 1 MB

Doing a sysupgrade on a extroot-ed device is dead easy... just boot without the USB drive, do a normal sysupgrade over the basic configuration, and start over with an empty USB drive... dead easy.

Changing the clemanture may help....

The word root is a bit like infinity ;).....

What say all references to extroot ..... were something like "extroot-overlay".

That and a big diagram near the top of the page breaking down core operation would help I reckon..... Happy to have a stab at a diagram if it helps.....

There is substantial variation on this.... def scope for a "helper" script.... if anything just to inventory and spit back at someone the modules they need.....

1 Like

Yes - and you don't even need to reboot without the USB - by default the sysupgrade will only write to the ROM, so just backup config (and package list), do a sysupgrade and then reboot, at this point you'll be back in the un-extrooted situation with the new image.

Then step through the ext-root process again including formatting the USB disc - as the previous contents won't be compatible with the new image. [I'd recommend also installing a minimum of stuff required for troubleshooting before you introduce the USB device, to cater for the case where it fails later].

1 Like

There is one thing I would add to your tutorial.

To make it easier to find the new drive name have the user run

block info

before inserting the drive into the router.

Then after inserting the drive run

block info

again --- and note the changes.

I am not sure if you need to reboot the router for the new drive to show up I cannot test this at this time, Maybe tell the user to copy paste the first result if a reboot is indeed required.

And even more great comments. I got buried on other projects, but will update the description when I get more time (within a week, I hope). Thanks again!

I followed your tutorial that you posted on the playground (~~?-feb 5-- 2019) and every SSH command worked as expected to get extroot working properly.

https://openwrt.org/playground/richb/extroot

I think that you should take this tut out of the playground and publish it.

That's great news - I'm glad it worked as desired. (In fact, the article in the playground is an exact copy of the current public writeup - I haven't got 'round to changing it.)

My reason for attacking the writeup, though, is that it's simply a cookbook: type this, type that, you're done.

But a) I wanted to know what I was doing, and b) there's not a lot of help/background/understanding of what to do if something goes wrong.

I envision the original post (with a few updates) in this thread replacing the beginning of the wiki article. Then (since I understand a lot more about how all this works), I would update the second half with the actual steps.

Thanks for reading - I'll update the original post above, then let the world know about it.