How do I boot off of large nvme ssd?

Installing and Using OpenWrt
1

I've got a QNAP QHora-322 / iEi Puzzle-M902 box which I'd like to set up
as a router. It's a standard (but somewhat muscular) kind of router box:
Arm processor, 4 GB eMMC flash, runs OpenWRT, but it also has an M.2
socket for nvme ssd. I can put a big, roomy 4TB drive in that M.2 slot.

That's appealing to me. It means I can use the box to do light service
duties, like TimeMachine backups, and serving my music collection. This
makes me want to punt the whole squashfs / overlay / whiteout stuff that
is engineered for small-flash routers, along with busybox, and run a
large install with plenty of packages to make admin easier.

But I don't know how to do that. One, I'm pretty new to u-boot; I'm
currently reading the docs. I have no problem connecting to a
serial-port console with a serial-to-USB adapter via screen or
miniterm. Two, I'm also not sure if there are OpenWRT firmware installs
that don't use the squashfs / overlay setup, suitable for throwing onto
a large partition. In fact, my prior experience with partition / filesys
setups on x86 linux distros is very different, and I'm not sure if it's
workable in the OpenWRT world. On >2TB drives, I will typically do
the following:

  1. EFI partition
  2. "Microsoft reserved" -- render unto Bill the 16MB that is Bill's.
  3. Root -- This is /, but just the parts that don't mutate much.
  4. Swap
  5. SysRW -- /var, with subdirectories rebound as follows
    /var/rebinds/root-tmp -> /tmp
    /var/rebinds/root-root -> /root
    This is system data that mutates a lot.
  6. HomePart -- this is "user" data -- /home.

OK, I left out two more partitions that I use when circumstances force
me to boot into Windows. But the above is the total linux picture. It
has the advantage that if some write operation corrupts a filesystem,
the damage is contained, improving my odds that I can get the system
to stagger back to its feet so I can try to repair it. I'm paranoid
that way.

This is all tuned for using large drives, running a large linux distro
like Debian, Ubuntu, Fedora or Arch, and booting off grub2.

Is there a straightforward way to install an OpenWRT setup onto a
large drive, and boot it off of the flash's u-boot, with the setup
I've sketched out above (or something roughly like that, but tuned for
OpenWRT), that takes advantage of not having to torque things around
to handle small storage?

I've poked around on the OpenWRT site looking for documentation that
would help me do this, but didn't find what I was seeking. (Which
doesn't mean it's not there; there are a lot of thoughtful howtos.
I might have just missed the one I needed.)

It seems to me that as flash sizes get larger & larger over time, and
more routers come with M.2 slots, this kind of use case is going to
become more & more common.

If anyone could advise me -- either explanations, or just pointers to the
right pre-existing web pages -- I'd greatly appreciate it.

Thanks.
-EKH

2

Not familiar with your specific device but why not leave the eMMC as your boot (kernel) and system (rootfs) partitions and just add the M.2 drive for data.

3 Likes
3

The simple answer is, you can't<fullstop />.
This is an embedded device, not a generic x86 system that asks you to insert a boot floppy into drive a:, it won't boot from USB, CD/DVD-ROM, SATA, M.2/ PCIe or removable media, it boots from its internal flash - or not at all (and secure boot might be another topic).

Furthermore it's a router, not a general purpose system. Overloading it with non-routing tasks is already not a smart choice to begin with (the aim would be to reduce the attack surface, not to broaden it - no, spare cycles is not a good argument). While I may understand (but not share) a desire to add some server tasks, it's still not a match for interactive usage (and the consideration for a /home/ partition goes 180° into the wrong direction).

If you disregard common sense of keeping a spade to be a spade, at least follow darksky's advice to disentangle optional (data-) mountpoints from the main firmware.

1 Like
4

Good question. The reason is that it's nice to have an OS install that isn't space constrained. You can have a real shell like bash and actual utilities, instead of busybox. You can throw on packages that make your life easier. The standard architecture of OpenWRT is carefully designed to be very parsimonious with persistent storage. I can run OpenWRT on a Netgear WNDR3700. It has eight megabytes of flash -- it's 15 years old. And OpenWRT just... instals and works. That's impressive engineering. But I can't add on much in the way of extra packages.

Now I've got this new router. I can put on literally a million times as much persistent storage. That is a qualitative shift. I don't want to put up with the strictures imposed by the details of what the OpenWRT do to make an 8 MB system work. I want u-boot on my flash, and the kernel, root filesys and user filesys to come off my big ssd.

It would also be nice if I could have an independent setup on the router's eMMC flash, just as an emergency backup. So... I would be cool with having just one kernel, on the flash, and then have it start up with root on the ssd, retaining an emergency-root partition on the flash, or something.

Does that answer your question as to motivation? I'm not a wizard at OpenWRT. What I'm really comfortable with is the Debian life -- frequent package upgrades, booting off of grub2 instead of u-boot, apt instead of opk, x86 instead of Arm. I am coming up to speed on OpenWRT, but, again, with all this storage on my box, I don't have to constrain myself quite so much and am trying to figure out how to run in a manner more like what I already know.

-EKH
5

The simple answer is, you can't.

This is an embedded device, not a generic x86 system that asks you to insert a boot floppy into drive a:, it won't boot from USB, CD/DVD-ROM, SATA, M.2/ PCIe or removable media, it boots from its internal flash - or not at all (and secure boot might be another topic).

You seem to be saying that u-boot can't boot a kernel off an M.2 ssd, which doesn't seem right. Or that u-boot can't boot a kernel off of flash with a root filesystem on an M.2 ssd, which, again, doesn't seem right.

I'm perfectly cool with the kernel living on flash, btw. I just want to be able to put the OpenWRT root filesys on a partition that lives on the M.2 drive. Then I will not be constrained by size limits on the number of packages I can install. Secondarily, I'd love to know if it's possible to punt the squashfs / overlay / whiteout stuff that OpenWRT uses to reduce the flash footprint, when I'm putting the system on a partition that is literally almost a million times larger than what's available on, say, an old Netgear WNDR3700. But I haven't been able to figure out, yet, if there is an easily installable version of OpenWRT that doesn't use the overlay-root setup.

Furthermore it's a router, not a general purpose system. Overloading it with non-routing tasks is already not a smart choice to begin with (the aim would be to reduce the attack surface, not to broaden it - no, spare cycles is not a good argument). While I may understand (but not share) a desire to add some server tasks, it's still not a match for interactive usage (and the consideration for a /home/ partition goes 180° into the wrong direction).

I can see that I didn't explain my intention clearly enough. I don't want to use this router box as a "daily driver" system on which I perform my usual interactive tasks. This is a box that will sit in my basement and manage my home net.

So it's not about "spare cycles." It's not about being a "match for interactive usage." And when I said "home partition," I meant that as shorthand for "non-OS data." E.g., the music collection that is served to the stereo systems scattered through my home, or occasional TimeMachine backups.

I'd prefer not to have to administer another box to run, e.g., my home's CUPS, or UPS monitoring. These are small services, but they have to run on something. They aren't interactive; I set them up and then don't mess with them. They do not have ports exposed to external WAN access.

And, while I'm administering this box, I'd prefer it if I was using bash, not busybox; sshd, not dropbear; avahi, not umdns; a full less(1) that gave me the -cem options that I'm used to having on every other system I use; and so forth. These things are all available in OpenWRT as standard packages, but they have to be installed and they take up space. No problem: space is what I've got.

If you disregard common sense of keeping a spade to be a spade, at least follow darksky's advice to disentangle optional (data-) mountpoints from the main firmware.

??? This was my stated intent from my first post. I carefully segregate the persistent bits into multiple partitions for multiple reasons; I laid out the way I do partitions on my interactive PCs to give the rough picture on how I tend to set up a linux system in general. I've been dealing with multi-partition unix setups since I had to do it on a PDP-11/45 in 1979.

I will, however, endeavor to avoid using boot floppies -- I've checked this QNAP router box pretty carefully and haven't been able to find the slot where they are supposed to get fed into the thing. Still looking...

EKH

6

The typical OEM u-boot can't.
This isn't a devboard where you can easily replace the OEM u-boot with a fully featured custom one - nor can you just plug in a new sdhc card with a working u-boot, if (when) you break it.

That's the difference between an open system (x86_64 or to some -lesser- extent even the classic RPi/ sunxi/ rockchip devboards) and an embedded system, like the typical routers supported by OpenWrt. OpenWrt typically doesn't touch the OEM u-boot and has to cope with its feature set and limitations, if you want to replace it, the effort will be yours and you're in a very high-risk situation, with a significant chance to brick the system for good.

OEM u-boot tends to be ancient and patched to hell.
Mainline u-boot is unlikely to support your device.
It's not going to be fun to find a middle ground here, to port the required patches to mainline u-boot (while staying compatible with OpenWrt, which expects the OEM u-boot), all that without bricking the device for good.

7

slh-

Thank you very much for that clear explanation. I appreciate your taking the time.

The typical OEM u-boot can't. This isn't a devboard where you can
easily replace the OEM u-boot with a fully featured custom one - nor
can you just plug in a new sdhc card with a working u-boot, if
(when) you break it.

OK, but here's my dodge. The OpenWRT people have to deal with all this
lossage, because they are taking on the difficult task of making the
software work across all the systems. Or as many as possible, anyway.

A specific user like me, however, can choose to buy a specific router.
Something modern, with a u-boot that isn't locked down, and that has
an M.2 socket. I look around and I see some very nice platforms, such
as the OpenWRT One & Two, or the GL.inet Flint 3. I would expect the
OpenWRT One & Two routers to have pretty full-featured u-boots. No?

I don't really want to have u-boot load the kernel off an SSD drive.
I'm perfectly happy for it to load the kernel off on-board flash,
where it has been loading the kernel from day one. I just want to have
that from-the-flash kernel come up using a root filesystem that lives
on a partition on the SSD. That seems much more straightforward, no?
U-boot is not really part of the picture, beyond using it to pass command-line
arguments to the kernel.

I look around and I see these new routers, with M.2 nvme sockets, and
it's clear to me that these systems are not as space-constrained as in
the past. This changes some of the tradeoffs. I can run real-deal
software, instead of cutdown lookalikes that give most of the
functionality but are tuned for size -- e.g., busybox. I can set aside
a separate rescue partition in case I bork my main root partition,
without having to mess with overlays. I could update packages more in
the style of standard linux distros. Etc.

So it would be nice if there was a variation of the firmware that was
structured for this kind of setup. Hence my original question.

Anyway -- I'm new to u-boot, embedded systems and OpenWRT. But I don't think I can ever go back to commercial router firmware. Thank God for Unix and open source.

EKH

8

What's the basis for these assumptions?

Relatively speaking, yes.
Absolutely speaking, no.

Does it, really?

You can't.

Indeed, and that leads to wrong assumptions.

If you want a general purpose linux distribution (arch, debian, fedora, gentoo, mageia, opensuse, ubuntu, …) on x86_64 (or maaaybe RPi, sunxi, rockchip), then get one of those. Configuring those as a router (as long as you let the wireless AP aspects be handled by more sensible hardware, different topic, the forum search will tell you) will be easier, than the amount of changes you'd have to do to OpenWrt and the underlying bootloader considerations for a fraction of the expectations you seem to take for granted. Unless you have absolutely positive documentation from a trusted source (and not some ai hallucination) for either of the features you're looking for to be supported on your specific hardware, you really shouldn't expect them to be present, nor easily achievable - that's the difference between "embedded" and generic/ general purpose hardware. Your off-the-mill plastic router isn't an x86_64 system, nor an SBC/ devboard with a vibrant community around it, realizing this would save you a lot of painful experiences.

OpenWrt is focused to work on heavily resource constrained systems, which requires accepting quite a number of tradeoffs to make it possible to run on a 8/64 or 16/128 system (and high-end/ mid-three-figure wifi 6/7 routers aren't as far away from those figures than you seem to assume), tradeoff causing rather fundamental paradigms that don't magically go away if you run it on x86_64 hardware either. OpenWrt is not yet another general purpose (desktop-) linux distribution, nor does it try to compete in that arena - it can't, without losing the ability to run on the kind of systems it is primarily targeting. At the same time you won't get to boot a general purpose linux distribution on a bog standard mips- or ARM 'plastic router', the hardware is far away from being ARM SBSA (UEFI && ACPI) compliant, main storage and bootloader constraints providing additional hurdles.

Conventional wisdom and best practices is to keep 'router' and 'server' tasks separate, each on dedicated hardware, both have different requirements, a different upgrade cadence (and approach) - as well as very orthogonal security requirements.

Yes, a lot of things 'could' be achieved, if you'd quit your job, concentrate on a limited subset of your desires and take on a new full-time hobby for the next couple of months++ to develop them for your specific hardware, make that years++ if your intent would be to make it a standard feature for 'OpenWrt'.

1 Like