Booting OpenWrt from USB?

I am trying to do something a little bit different: I'm trying to boot OpenWrt not off its flash memory, but rather off USB.

My candidate is a My Book Live Duo that usually boots off one of the two SATA drives since it has quite insufficient flash memory.

So far I had success booting an initramfs image from USB (kernel.img is the initramfs kernel image, kernel.dtb is obviously the device tree file):

usb reset
fatload usb 0:1 1000000 /kernel.img
fatload usb 0:1 1800000 /kernel.dtb
bootm 1000000 - 1800000

This successfully loads and boots the initramfs build from a FAT formatted USB stick.

But of course, an initramfs image is not suitable for regular operation since it would not retain configuration. Booting a full OpenWrt uImage fails. I'm thinking it needs to be compiled in a special way to work with USB to carry rootfs/overlayfs. (Maybe you can tell I'm not completely familiar with the innards of OpenWrt.)

Searching the OpenWrt/LEDE pages and forums and the rest of the web, I came up empty. Is there anyone who can shed a little light on this? I'm thinking this would be beneficial to other people and devices, too.


1 Like

I would probably try an initramfs kernel with a preconfigured extroot setup (block-mount + /etc/config/fstab already baked into the image).

@jow: Thank you, that was actually one of the things I considered.

Am I correct in assuming this requires a full build from source? There is no way to build an initramfs image using the Image Generator/Builder, right?

Right, as far as I know it isn't directly possible. The ImageBuilder might ship enough bits and pieces to do it manually somehow (didn't check) but there certainly is no official "make initramfs" command or similar.

Alright, progress. To my own amazement, I managed to build an initramfs image containing block-mount (by default) and all the usb-related kernel modules. In uboot it loads (excruciatingly slow, the 4.4 MB image takes almost two minutes to transfer in through ext2load) and boots.

But I hit a snag hooking in the extroot.

Manually, I can mount an usb partition just fine, both to /mnt/foo and to /overlay, and block mount will also mount it to /mnt/foo. However, block mount will not mount it to /overlay, neither on boot nor when block mount is manually run from the fully booted system.

I have no idea on how even to start to debug this. Has anyone made similar experiences?

Block should print some extroot diagnostics into dmesg during boot - do you see some logs there?

Nope, nothing related to block. (Interestingly, not even if I manually run block mount.)

Here's a dump of dmesg, boot log, block info and /etc/config/fstab ... I can't see any error. /etc/rc.d/S40fstab is present. Mounting /dev/sda2 works if I try to mount it anywhere else but /overlay, both manually and through block mount, so the drive itself is also not the issue.

Edit: When I compare mountpoints between the initramfs-booted to a regularly booted up system, there is no overlayfs mounted. Is that maybe the reason mount would not pivot /overlay to the new destination?

Nah, not sure what to say. The initramfs images for the MBLs are meant as a recovery tool. For example the user got locked out or a sysupgrade went wrong and the system is no longer booting.

Anyway, the initramfs is a special construct, if you want to know more about it you could checkout the Documentation that comes with the kernel.
As it explains some of the shortcomings:

- When switching another root device, initrd would pivot_root and then
  umount the ramdisk.  But initramfs is rootfs: you can neither pivot_root
  rootfs, nor unmount it.  Instead delete everything out of rootfs to
  free up the space (find -xdev / -exec rm '{}' ';'), overmount rootfs
  with the new root (cd /newmount; mount --move . /; chroot .), attach
  stdin/stdout/stderr to the new /dev/console, and exec the new init.

Alright, so that possibility goes out the window.

What I find rather interesting is that the MBL is supposed to boot off an external disk. I would love to know what exactly goes wrong when booting a regular image off USB. But on the Duo, early in the boot process of OpenWrt the serial console stops working.

Pretty much :slight_smile: ... Unless you are desperate :frowning_face: ... nobody else is looking or cares :man_shrugging: .

That "serial console stops working" is because of the (missing) bootargs. You could either edit the apollo3g.dtsi to include stdout-path in the chose-node or try "run addtty" in the uboot shell (before the bootm)

If you are willing to invest some time, I think you can get a custom OpenWrt image to boot off a usb-stick. I can't really test it, since my MyBook Live Duo died due a faulty powerbrick... So no: This isn't a PoC. no guarantees! (And there can be gaps/jumps or errors).

If you still want to give it a shot: you should try the following:

  1. add the following to the target/linux/apm821xx/sata/config-default:
    (at the end is fine)

    # CONFIG_USB_EHCI_HCD is not set

    This will add support for the dwc2-based usb port, usb-storage into the kernel.

  2. edit target/linux/apm821xx/image/mbl_boot.scr to look something like (untested):

    setenv boot_args 'setenv bootargs root=/dev/sdc2 rw rootfstype=ext4'
    setenv load_part 'ext2load usb 0:1 ${kernel_addr_r} /boot/uImage; ext2load usb 0:1 ${fdt_addr_r} /boot/apollo3g.dtb'
    setenv boot_usb 'run load_part; run boot_args addtty; bootm ${kernel_addr_r} - ${fdt_addr_r}'
    run boot_usb

    Note: the bootargs "root=/dev/sdc2" is fragile (if you know what you are doing you should use UUIDs here.) The device detection is inherently racy so this could be sda2 (if no HDD gets detected before the usb enumeration happens), sdb2 (if one HDD is detected) or sdc2 (if two HDD are detected). So watch out!

  3. now compile

    Probably best to use make V=s

  4. unpack openwrt-apm821xx-sata-wd_mybooklive-duo-ext4-rootfs.img.gz and
    dd the extracted data to the usb-stick.
    (so yes: this should be almost a "normal" installation image. Furthermore sysupgrade might just work as well... but again no guarantees)

  5. interrupt the bootloader and make it load the boot/boot.scr script from the usb-stick

    to run a script.scr in u-boot simply load the boot/boot.scr via ext2load to a $memoryaddr and use

           source $memoryaddr


1 Like

Honestly, I am really surprised that this isn't a huge topic, but rather
a) hasn't been discussed before, anywhere, and
b) is of no interest to anyone but three people (two helpful devs and me asking stupid questions)

I mean, really, any device with a usb port and uboot to support fatload/ext2load would really benefit from it. Switching configurations and testing new firmware by switching/unplugging a usb stick seems a no-brainer. Heck, even devices with spotty flash memory could benefit from it. The mind boggles.

Ah! For testing, I manually loaded kernel/dtb and bootm'ed from uboot console and missed the bootargs part.

Thanks for all your advice, seems straightforward enough. I will try it ASAP and report back.

1 Like

I think this really depends on the setup and test procedure. I can see why having a bootable usb-stick for the MyBook Live Duo can make sense (since you don't need to waste space for the boot+rootfs on the HDDs at all). However most other devices that OpenWrt supports are network-only. So the most common way to testdrive new patches/settings/... is via tftp(server) and initramfs route.

yeah, 2 minutes is a long time... too long. tftp should be much quicker to transfer the image. If you need pointers look at the commit that added the mbl initramfs images:

This ain't official.... but it's where I ended up at in your situation.....

  1. Use the buildroot (stable)
  2. ./scripts/feeds update -a; ./scripts/feeds install -a
  3. make menuconfig ( set ARCH / TARGET .... some other stuff if you like )
  4. Enable atags in boot - scsi v4
  5. make V=s ( possible IGNORE_ERRORS=1 if fails on random pkg )

( this is the gist but this step doesn't work for me )
6. make kernel_menuconfig

-usb-storage ( maybe uas )
-probably a few of ehci / xhci / ohci
-ext4 - mbcache - nls ( msdos maybe )
-block something ( sg )
-missing one or two here

( note: what you need is obtainable from dmesg / /etc/modules(-boot).d )

  1. save > make defconfig

7.1 Find your .dts. ... usually in target/....lin...../ARCH../....... files....... somewhere
( find devicedts )

7.2 Remove the line under chosen labelled "root-block"

  1. make V=s again

Now it probably wont work...... but your close....... to debug.......

  1. dmesg ..... see if your root=/dev/sda2 ( etc ) uboot parameter is sticking in the early output.....

  2. cat build_..../targ...../lin....../lin-4....../.config.override | grep USB
    ( find .confg.override )

If you see =m next to things like usb storage...... you have to search here to find the best way to force them to be built in..... there are at least 3 ( documented ways )

One day hope to figure out an easy way / whats zapping the official methods..... ( )

1 Like

I'm sorry to revive this long dead thread, but I just saw it on the MBLD device page that boot from USB is now possible. WOW! Excellent work! Thank you very much whoever made it happen!

I just tested it, I have some questions/remarks:

  • according to block info and mount /dev/sda1 is mounted as /boot (not /dev/sdc1 on the USB drive). It is an OpenWrt boot partition indeed, but from another version. I presume it is because of some UUID clash.
  • It boots extremely slowly, as it was mentioned earlier it took 2 minutes to load the ~4 MB boot image. Is there any way to speed it up?
  • Are there plans to put it in the stable release?
  • Is there another thread where the progress is recorded?

I'm not using my MBLD anymore, I've moved to a more capable NAS, I just kept it for nostalgic reasons. So I can use it for test and can give feedback on the results.

I actually never tried, but it's possible that /root is always mounted on sata 0:1 if it's present there, even if the MBLD boots from sata 1:1, or now from USB.

The kernel and dtb are loaded by the bootloader, so I'm fairly sure it doesn't matter for regular operation where /boot is mounted. The correct location would be relevant for sysupgrading, though, so it's probably a bit more than a cosmetic issue.

Short of replacing the bootloader with a more capable one I don't see how. It's not a huge deal, once the devices are running they are running ... occasionally for years. But I agree that it makes configuration and debugging a bit tiresome.