Booting Kirkwood NASes directly from disk (was:Synology DS211j boot support)

I started working on support for the aforementioned NAS. My tree is here: https://github.com/Leo-PL/openwrt/tree/synology_ds211j
Specs:

CPU: Marvell Kirkwood 88F6281 A1 @ 1.2GHz
RAM: 128MB Samsung K4T1G164QE-HCF7 DDR2 (16-bit) @ 800MHz
Flash: 4MB ST M25P32V6P SPI-NOR
Storage: Two internal, fixed 3,5" SATA 3Gbps bays
Network: Single rear Gigabit Ethernet port through Marvell 88E1116R PHY
USB: Two rear and one front USB 2.0 ports through onboard Genesys Logic GL805G hub
RTC: Ablic S-35390A I2C RTC
Console: 3.3V TTL serial port at ttyS0
Embedded controller: Microchip PIC16F627A at ttyS1, running Synology firmmware.

The main difference from all other Kirkwood NASes I've encountered is, that this device uses only 4MB SPI-NOR Flash for booting, which holds U-boot, kernel and (stock) initramfs. Primary root filesystem lives on the HDDs on this line of devices. This 4MB is not enough to store OpenWrt installation by any means, and I'm trying to figure out the best way to boot the device.

There is upstream support in U-boot (under ds109 target) as well as in Linux, including a device tree. Stock U-boot can only boot the kernel from TFTP or SPI. Upstream U-boot boots the device from USB by default using fatload, however in OpenWrt, drivers for neither SATA controller nor USB storage are built into the kernel on kirkwood target. They are not included in DEFAULT_PACKAGES either, so initramfs won't be able to mount the storage either.

I'm trying to figure out how to proceed - currently, re-enabling one of those modules is the only thing that comes to my mind, as the initramfs has way different purpose in OpenWrt than in stock firmware, but maybe there is a better way? My initial idea was to borrow the booting scheme from imx target with the imx_sdcard method and perform installation on external USB drive, which would leave the internal HDDs free to use with any layout.

Currently, I think that whether upstream U-boot is chainloaded by stock one, or replaced entirely, is orthogonal to that and not an issue - if installation of the image is possible on SATA drive, the kernel should be loadable from the disk as well, to avoid the kernel size limitation in flash - there are 3328kB available for kernel and initramfs total in flash.

Ideas? Chiming in @CHKDSK88 as one of main kirkwood contributors.

Build your own image with the modules you need, grab the uimage and initramfs and put them in a fat partition on usb or start of sata drive 1 then set uboot to fatload the images.

For rootfs, dd it to a second partition and set the kernel parameters to access it there.

With an updated uboot kirkwoods are very easy to boot in any way you might want

For kirkwood it is not uncommon to replace the factory uboot with an OpenWrt specific patch to the upstream U-boot. For examples see package/boot/uboot-kirkwood/patches. You should be able to modify configuration of upstream uboot to suit your needs. You can also specify additional device packages to be included in your image (for example see the ctera_c200-v1 device).

dunno if this device is compatible with the other kirkwood hw/products, but the doozan forum is a gold mine for uboot stuff.

@jdwl1o1 Yeah, it's cool and all, but this solution is not upstreamable. I'm looking to create upstream support.
For upstream support, the driver for either kmod-usb-storage or kmod-ata-marvell-sata would need to built into the kernel once more, after @CHKDSK88 split it out into packages (and for a good reason).

@RaylynnKnight thanks - I've seen it when testing out Netgear RND2000 support, this will likely be needed, to supply PARTUUID to the kernel instead of currently hardcoded /dev/sda in the upstream. Just including extra modules in the image won't help in my case, though - they will need to be loaded from disk or USB, so it's a chicken-or-egg problem.
I'm currently leaning towards external USB storage, to free up the SATA disk partitioning for the user. Shouldn't be a much problem in the era of small USB drives that basically hide inside the port.

@frollic - thanks, I'll check it out. Since there is upstream support, somebody must have ran an "upstream distro" on one of those boxes.

Now, there is another question. Is it possible to perform normal boot retaining settings from the initramfs? Or maybe, create a small initramfs containing just kmod-usb-storage and kmod-ata-marvell-sata and bundle that to normal kernel, to avoid growing the kernel size for other devices.

Many devices rely on particular kmods to operate, you can specify which kmods the device requires in the device setup files.

And using an initramfs is the Debian boot strategy, if you want to go that route the doozan crew can assist with running straight Debian

As I wrote in the last post, this is a chicken-or-egg problem, and just enabling kmods for SATA or USB won't help. Please read it again, fully.

1 Like

Mate, I’ve built openwrt for multiple kirkwoods, it really is that easy.

Mod uboot, build image, load kernel, boot.

Try being more gracious to people who know what their doing.

I really don't want to go down that route. I maintain a Yocto-based embedded distro for quite a few years commercially, and know my ways around OpenWrt as well. If it is that easy, feel free to prove me wrong, the code is up on my Github. ITT: creating an additional initramfs to bundle just those kmods to normal kernel in Yocto would be trivial.

Back to the topic. Sure, I can alter the kernel config to include drivers for USB or SATA, and then load rootfs this way, but this will grow the kernel size for all kirkwoods, and for some of them, the space for kernel is tight. I don't want to undo the changes done by @CHKDSK88.
Let me reiterate: this device does not have NAND flash like all other NASes supported by OpenWrt so far. It only has a mere 4MB SPI-NOR to boot from, and loads OS from the disk.

The main thing is, that with just enabling kmods for the image, the kernel won't be able to load those very modules if it cannot mount the rootfs from external media, and those modules are required to do just that, so need to be provided to kernel in an other way. Unless we enable the modules to be built-in again, there is a circular dependency.

The WD mybooklive nas boots Openwrt directly from the sata drive. The target has a nand option and a sata option in the download section. Maybe something similar could be done for kirkwood?

https://downloads.openwrt.org/releases/21.02.3/targets/apm821xx/

Both subtargets of apm821xx have CONFIG_SATA_DWC enabled directly in kernel, which is what I'm trying to avoid. But maybe this is the only option. Let's wait for comment from @CHKDSK88 - sorry for pinging you that many times :wink:

Edit: even if we need to enable the drivers to be built-in again, it may be good idea to provide two variants of images for Diskstations, similar to MyBook Live, and leave the choice to the user.

@Leo-PL
Sorry for long time response. I was away.

SATA was disabled because Linksyses EA3500 and EA4500 have small kernel partition and there was no device with SATA images.

Limit still is very small (3072k):

We can't enlarge kernel. It could cause disabling Linksyses in future.

OpenWrt doesn't use Initrd images. I suggest create new subtarget for SATA-image devices. But I'm nod core-dev, @mans0n is.

@mans0n
What's Your oppinion?

I do believe we may need a sub-target for SATA-image devices as I have several that I would like to provide support for that will have the same issue as the DS211j. Many perfectly good Kirkwood NAS servers would benefit from using OpenWrt as their vendors no longer provide a reasonably current kernel.

1 Like

@CHKDSK88, @RaylynnKnight, that's the same conclusion that I came to, that re-enabling SATA and USB drivers can be done in different subtarget, and this is way to go - I'll implement that soon-ish.

My idea is to borrow the boot scheme from apm821xx or imx (the sdcard one), using PARTUUID and patching upstream U-boot to attempt booting both USB and SATA, so both media types can be used from the same factory image.

Now, one question remains: how to name new subtargets? My first idea was to use "nand" and "disk" for current and new devices, respectively, but "generic" and "sata" would also make sense - or combination of them. What do you think?

1 Like

Wondering if NAND and DISK might be better as those devices without NAND will have a small NOR flash to hold U-Boot and could boot the OS from either SATA, USB or in some cases SD card. Some examples I have contain as little as 512KB of NOR flash (i.e. LaCie and some Buffalo NAS devices and the Dell Kace M300).

I'm interested in creation of a new subtarget.
I'm working for adding support for I-O DATA HDL-A/HDL2-A series nas, which has no flash for os image.

Cool!

I wonder if the boot procedure for your device could be genericized a bit - is the stock loader expecting a particular partition layout?

Synology is holding kernel on SPI-flash, but I think I'd rather dump it in favour of /boot partition, the same as done on imx subtarget - please see my recent refactoring for imx-sdcard image type. Maybe this way the installation could be as easy as writing new U-boot (in case of Synology) and connecting a SATA or USB disk having the factory image just unpacked using dd.

Of course, due to Linksys' limitations, a new subtarget would be required :slight_smile:
I'll take a more detailed look later, for now - looks promising.

1 Like

Thanks, HDL-A/HDL2-A tries to load kernel from the first partition in SATA disk. On HDL2-A, loading kernel from secondary disk will also be tried.

default bootcmd:

hdlfixwol
mw.b 0x800000 0x00 0x800000
mw.b 0x2000000 0x00 0x800000
hdlwake
ide reset
hdlstart
if   ext2load ide 0:1 0x800000 /uImage.l2a && \
     ext2load ide 0:1 0x2000000 /initrd.l2a && \
     sbootm 0x800000 0x2000000; then  // <--- unsigned uImage can also be booted

elif ext2load ide 1:1 0x800000 /uImage.l2a && \
     ext2load ide 1:1 0x2000000 /initrd.l2a && \
     sbootm 0x800000 0x2000000; then

else 
     usb start;

     if   fatload usb 0:0 0x800000 l2a/uImage.l2a && \
          fatload usb 0:0 0x2000000 l2a/initrd.l2a && \
          sbootm 0x800000 0x2000000; then

     elif fatload usb 0:1 0x800000 l2a/uImage.l2a && \
          fatload usb 0:1 0x2000000 l2a/initrd.l2a && \
          sbootm 0x800000 0x2000000; then

     else
          hdlbtn  // <--- wait pushing POWER/RESET button

     fi
fi

note: dummy data can be used as initrd.l2a
(dd if=/dev/zero of=initrd.l2a bs=4 count=1)

Synology is holding kernel on SPI-flash, but I think I'd rather dump it in favour of /boot partition, the same as done on imx subtarget - please see my recent refactoring for imx-sdcard image type. Maybe this way the installation could be as easy as writing new U-boot (in case of Synology) and connecting a SATA or USB disk having the factory image just unpacked using dd .

Sounds great! :slight_smile:

Does the stock U-boot have writable environment? Maybe it would be enough to override the bootcmd, and have common image format.
Anyway, I think I'm reaching too far out. Maybe on the weekend I can allocate some time towards splitting out the "disk" subtarget, and then generating some kind of filesystem image - not necessarily in final form.

As for defining logic for booting DS211j, I plan to do more-or-less the same as in your post: try both HDDs, then fallback to USB, and then maybe even fallback to TFTP, because why not - this may save some headaches for end users.

I also updated the topic to cover other devices with similar issues.

HDL-A/HDL2-A have U-Boot environment partition and updated values can be stored. However, aborting of boot sequence is disabled by source. :cry:

HDL_PLATFORM_2008 and HDL2A are defined.

#ifdef	HDL_PLATFORM_2008
#if defined(HDL2A) && !defined(HDL_DEBUG)
	if (bootdelay >= 0 && s) {
#else
	if (bootdelay >= 0 && s && !abortboot (bootdelay)) {
#endif
#else
	if (bootdelay >= 0 && s && !abortboot (bootdelay)) {
#endif

I tried to write experimental disk image recipe and succeeded to boot on HDL2-A.
(fe0ff7e "tools: compile genext2fs on kirkwood target" ~)

log: