MikroTik rb450gx4 support

There is a single Spansion S34ML04G200TF100 NAND mounted as U902 and it looks like I/O0 - I/O7 (×8) is connected to the CPU.


❐Page size:
•×8:
1 Gb: (2048 + 64) bytes; 64-byte spare area
2 Gb / 4 Gb: (2048 + 128) bytes; 128-byte spare area

•×16:
1 Gb: (1024 + 32) words; 32-word spare area
2 Gb / 4 Gb (1024 + 64) words; 64-word spare area

❐Block size: 64 Pages
•×8:
1 Gb: 128 KB+ 4 KB
2 Gb / 4 Gb: 128 KB + 8 KB

•×16
1 Gb: 64k + 2k words
2 Gb / 4 Gb: 64k + 4k words

❐Plane size
•×8
1 Gb: 1024 blocks per plane or (128 MB + 4 MB
2 Gb: 1024 blocks per plane or (128 MB + 8 MB
4 Gb: 2048 blocks per plane or (256 MB + 16 MB

•×16
1 Gb: 1024 blocks per plane or (64M + 2M) words
2 Gb: 1024 Blocks per Plane or (64M + 4M) words
4 Gb: 2048 Blocks per Plane or (128M + 8M) words

Now that hAP ac2 is merged. Could it be possible to merge rb450gx4 in OpenWrt 21.02?
Thanks,

I doubt 21.02, but it needs to be rebased on the current master as the code is way out of sync.
I honestly have no idea when will I get to it.

I have been doing some testing with a netboot image. Started from @robimarko's work, and rebased to 2021 master 1.

I am not sure how we should partition the NAND. Please provide any advice or corrections.

OEM splits NAND to two mtd paritions:

  • 0x000000000000-0x000000800000 : "RouterBoard NAND 1 Boot"
    UBI volume "Kernel"
    contains the ELF kernel, and VERSION

  • 0x000000800000-0x000020000000 : "RouterBoard NAND 1 Main"
    UBI volume "RouterOS"
    contains bootimage (NPK), plus various directories & files

I guess RouterBoot loads the mtd0 UBI volume Kernel/kernel ELF, which loads the mtd1 UBI volume RouterOS/bootimage.

From what I can see, OpenWrt upgrade/nand.sh 2 is only designed to work with one UBI (mtd) partition, usually labelled "ubi" in DTB, with possibly multiple volumes under it:
local ubidev="$( nand_find_ubi "$CI_UBIPART" )"
With the image was using adron's loader, the kernel was packed into a ubi partition image. For tar sysupgrade, with an mtd partition labelled "kernel", nand_upgrade_tar for kernel only uses openwrt's mtd 3 to write to the mtd partition, which would lose the UBI erase counts?

Options?

  • Keep OEM mtd partitions:
    • extend or build custom upgrade script to install the self-extracting ELF kernel to mtd0 UBI partition Kernel/kernel:
      • Allow different CI_UBIPART for rootfs & kernel?
      • mount, and replace the file, or
      • ubiupdatevol?, and
    • ubiformat mtd1 UBI partition to Openwrt "ubi" volumes for rootfs

I have not tried, or looked into RouterBoot to check the limitations to see if we could remove OEM mtd partitions, and use single UBI partition. Might lose UBI erase counts in this process?

define the PoE output compat GPIO input as a gpio key (extcon-gpio does not have DTS bindings)?
define the PoE output control GPIO as a LED, or leave as sysfs / gpio-export?

Any feedback would be great.

Cheers,

I dont think you can unify the partitions as from my testing it is looking for a UBIFS packed kernel

I started working on hAP ac3, it also has the same NAND installation issue.
The layout is somewhat weird as they use UBI with volumes for Kernel and Rootfs, but both are then packed in XZ compressed UBIFS.

So, I am having issues in figuring a way to have a permanent flash with sysupgrade.

Ok, so I did some digging and testing.
It appears that RouterBoot does not require kernel and rootfs to be separated with partitions but only UBI volumes.
It also does not care if the kernel is named "kernel" or "Kernel" like in ROS images.
All it cares about is that the kernel is packed in UBIFS, without UBIFS it won't boot the kernel at all, so I manually packed the kernel in UBIFS per what stock kernel is packed and ubinized it and that boots.

So, we just gotta find a way to pack the kernel in UBIFS in OpenWrt buildroot directly.

Nice work. I have some make hackery to do the packing. It would be great if you tested to check if it works for hap ac3 with a single partition. Please let me know your RouterBOOT version as well. Would be really nice to disassamble RouterBOOT to find the ELF boot constraints, but I have not had any luck so far (can see truncated strings that are used in RouterBOOT console near the end of RouterBOOT fwf…).

For my rb450gx4, RouterBOOT 6.47.4 128k eraseblock NAND

  • ubinized image would not fit in NAND part1
  • did not work if I ubiformated NAND part1, and used a ubinized image in NAND part2
    loading kernel... kernel not found or data is corrupted
  • did work if I manually ubiformated NAND part1 with ubinized ELF self extracting kernel
  • did not work if I turned NAND part1 into NAND part1+part2 in DTS, and used ubinized image
    loading kernel... OK
    setting up elf image... not an elf header
    kernel loading failed
    
    I did check that UBI Kernel/kernel had ELF header, and the same sha256sum in build_dir as on device

From what you have found (single part worked), it may be that my packing had an error, or RouterBOOT did not like what ubinized.sh generated. It is the commented-out recipe in Device/mikrotik_nand here.

  • For my ubinized sysupgrade single part test:

    • used only NAND part1, with label ubi in DTS, and offset 0, with size that covered OEM part1 + part2
    • used the commented-out KERNEL, IMAGE/sysupgrade.bin, and UBINIZE_PARTS recipies
      This built a single ubi(inized) sysupgrade image, which the default sysupgrade_nand could then flash (with ubiformat)
      If this does not work, that is great, as a single partition NAND UBI might work for my 450gx4. Then we check the kernel & compare the UBIFS & UBI settings.
    • sysupgrade ubinized:
      . /lib/upgrade/nand.sh
      
      CI_UBIPART=ubi
      nand_do_upgrade
      
    • if Kernel == kernel volname case does not matter to RouterBOOT, can remove UBINIZE_PARTS, and use KERNEL_IN_UBI := 1 instead.
  • For my OEM NAND parts1+2 test:

    • This is the current recipe.
    • Ignore the custom upgrade script for now, it does not work yet (variable scope I guess, copy-paste stepping through it worked)
      It only matters if we care to keep UBI erasecounts in the first NAND part.
      Otherwise, name part1 kernel, and nand_upgrade_tar overwrites it with the non-UBI aware OpenWRT mtd flasher

Good luck!

I am using 6.46.6 as for me 6.47 and newer RouterBoot has broken DHCP.
Its probably even the same NAND on RB450Gx4 and hAP ac3 I have.
Mine has Winbond W29N01HV 128k blocks, 2048 pagesize.

I am also hitting the:

loading kernel... OK
setting up elf image... not an elf header
kernel loading failed

If ubiformating with an OpenWrt generated UBI factory image.
Weird thing is that I pulled the kernel from OpenWrt and then manually packed that into UBIFS and ubinized it, then simply ubiformated and RouterBoot booted that without any issues.
Naturally, rootfs was missing but it still proved that it does not care for naming.

Now I am going over the parameters I used in UBIFS packing manually and with OpenWrt.
I added a simple packer to image-commands.mk:

define Build/package-kernel-ubifs
	mkdir $@.kernelubifs
	cp $@ $@.kernelubifs/kernel
	$(STAGING_DIR_HOST)/bin/mkfs.ubifs \
		$(UBIFS_OPTS) \
		-r $@.kernelubifs $@
	rm -r $@.kernelubifs
endef

$(UBIFS_OPTS) used are: -m 2048 -e 124KiB -c 2048 -x none

So, this makes no sense to me.

I tested to see whether packer breaks the kernel image and no, after extracting that one boots just fine over DHCP.

Thanks. Good to see.

I did not save an initramfs with single NAND part (overwritten), so I will need to recompile for the following:

Next suggestions to test RouterBOOT not an elf header
With the single NAND part, with working kernel volume

  • Boot initramfs, then attach the UBI part, and ubimkvol:
    • -N rootfs -s $(size of build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-ipq40xx_mikrotik/root.squashfs)
    • -N rootfs_data -m (rest of space)
      If RouterBOOT still boots kernel, then boot initramfs again, and ubiupdatevol rootfs to the root.squashfs from build_dir

Otherwise manually ubinize kernel + rootfs + rootfs_data? Something like this?

[Kernel]
mode=ubi
vol_id=0
vol_type=dynamic
vol_name=Kernel
#vol_size=8MiB
image=build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-ipq40xx_mikrotik/mikrotik_rb450gx4-128k-kernel.bin.ubifs
[rootfs]
mode=ubi
vol_id=1
vol_type=dynamic
vol_name=rootfs
image=build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-ipq40xx_mikrotik/root.squashfs
#vol_size=2535424
[rootfs_data]
mode=ubi                                                                        
vol_id=2
vol_type=dynamic
vol_name=rootfs_data
vol_size=1MiB
vol_flags=autoresize

Okay, I think the issue is coming from append-ubi or ubinize.sh the rootfs image in the ubinized image.
Manually got a working single NAND partition system on rb450gx4. Thanks for the hint.
ubiformated with append-ubi image, removed vols rootfs & rootfs_data, recreated them (rootfs had a different size), and ubiupdatevol rootfs
Now to work out why.

My notes / log
  • single NAND partition + ubinized ubifs kernel
    with KERNEL_IN_UBI:=1 (need to cp $@.ubifs to $@ in from my earlier commit)

    • Bad: error:
      loading kernel... OK
      setting up elf image... not an elf header
      kernel loading failed
      
  • cut _rootfs_data volume

    ubirmvol /dev/ubi0 -N rootfs_data
    sync; reboot
    
    • Bad: same error
  • cut rootfs volume

     ubirmvol /dev/ubi0 -N rootfs
     sync; reboot
    
    • Good: Routerboot loads kernel fine
  • reverse process:

  • add rootfs vol

     ubimkvol /dev/ubi0 -N rootfs -s 10MiB
     sync; reboot
    
    • Good: Routerboot loads kernel fine
  • scp the rootfs:

    tar -C /tmp -xf /tmp/openwrt-ipq40xx-mikrotik-mikrotik_rb450gx4-128k-squashfs-sysupgrade.bin_OEM_NAND_parts
    ubiupdatevol /dev/ubi0_1 /tmp/sysupgrade-mikrotik_rb450gx4-128k/root
    sync; reboot
    
    • Good: Routerboot loads kernel fine; squashfs gets mounted
  • Recreate rootfs_data

    ubimkvol /dev/ubi0 -N rootfs_data -m
    sync; reboot
    
    • Good: Routerboot loads kernel fine; squashfs gets mounted; overlay gets used

Guess so far: RouterBOOT (for one NAND UBI partition) seems to have problems with any ubiformat flashed UBI image that has a ubinize stuffed root in it (squashfs, or ubifs).
squashfs.root UBI volume updated with ubiupdatevol on the device works fine

@robimarko I am leaning towards leaving the NAND partitioning as OEM had it? The first specifically for kernel.
So far, RouterBOOT is refusing (with ELF header error) to load the known-good kernel ELF, when ubinize has also packed a rootfs (as another volume, either squashfs or ubifs) into the UBI partition.

Hmm, those are some good findings.

Single partition is gotta work as that is what MikroTik uses on hAP ac3, they just have 2 UBIFS packed volumes.
One is for kernel and the other one is rootfs.

So, could it be that it freaks out with more than 2 volumes or requires empty space after the kernel?
Tried UBIFS rootfs as that reduces the UBI volumes to only 2 and same ELF error.

Hm, using sysupgrade makes things even worse as then we get the not found error.

I am really confused as stock FW uses the same layout and UBIFS for both there is not even empty PEB-s after the kernel.

Ok, looking at the generated ubifs factory image and NAND dump I can see one big difference.
OpenWrt generated one is identified as one image with 2 volumes, while stock has 2 images each with its own volume.
That points again to 2 partitions being used, so sorry for me misleading into a single partition.

@johnth I had some free time today, so I implemented UBI kernel support to flash the kernel in a separate partition but in UBI.

Its still hit and miss and worked for me on the first try but after couple more tries it will not boot the flashed kernel.

Maybe you can test it and spot a bug as it now makes even less sense that it sometimes works and sometimes not.

My branch is here: https://github.com/robimarko/openwrt/tree/mikrotik/hap-ac3

@robimarko Nice. Nothing stands out at first look.
I will test later.

If first sysupgrade works, then after some more sysupgrades, we see the RouterBOOT error not an elf header: As a test, maybe try forcing the kernel mtd ubi parition to be ubiformat every sysupgrade? To see that ubiupdatevol is not leaving old bits, or a state RouterBOOT cannot understand in the UBI partition.

To deal with later:

  • The UbiFit append-ubi factory image will not be any good to us, as kernel volume gets packed into the same partition as rootfs, instead will need a pre upgrade function to remove the kernel part Kernel vol, and data part RouterOS vol, or ubiformat both if one detected?

The thing is that it booted once fine after sysupgrade, I forgot to include the bootarg to tell it whats rootfs, added that, sysupgraded again and then it will find the kernel but get stuck on jumping.

I then booted initramfs again, ubiformated both kernel and rootfs partitions and sysupgraded and after that it will throw an error that no kernel is found, not even then ELF error.

We actually don't need the factory image at all, only the sysupgrade one so I am planning on dropping the UbiFit completely.
We can add a pre-upgrade function that will simply ubiformat or do something else.

But, we first gotta see what the hell is wrong still

@robimarko Ah. I might try to look at RouterBOOT again, or the netinstall (device initramfs) ELF to see how the NAND gets formatted and upgraded.

Thinking out loud:

Guessing there is a difference to RouterBOOT in a ubiformat -f ubinized.img vs ubimkvol & ubiupdatevol -f ubifs.img:

  • I have not tried to ubimkvol and/or ubiupdatevol kernel, only ubiformat -f ubinized_kernel.
  • With adron's loader, the ubinized kernel was directly written to NAND by mtd.
  • When I tried the single partition kernel + rootfs volumes ubiformat -f ubinized image, RouterBOOT would not work, with the setting up elf image... not an elf header
    When I then ubirmvol, ubimkvol, ubiupdatevol the rootfs, RouterBOOT then worked.
    Therefore, there is a difference to RouterBOOT in a volume made by ubimkvol vs included in a ubinized image.
    • Not sure that the packed kernel ELF worked then after the bootargs update.
    • With the rootfs containing UBI partition labelled ubi in DTS, it gets automatically attached in OpenWrt.

Perhaps (reaching, but trying to work it out):

  • The (kernel booted) first sysupgrade used the existing kernel volume (no ubimkvol)
  • The (failed at jumping) next sysupgrade had a bad linux kernel, so it did not start.
  • The (not found / corrupted) next, following manual ubiformat caused the next sysupgrade to ubimkvol, which confuses RouterBOOT

Will go testing again later.

I had been poking around trying to get a ubiformat -f ubinized_kernel, then cut kernel from tar, then normal try upgrade to work: https://github.com/john-tho/openwrt/tree/450gx4

Cheers

Update: I think I have working sysupgrade working on my 450gx4 branch, through using ubiformat -f kernel.ubi

If anyone is testing my branch from an earlier Openwrt image, you should netboot the initramfs from my branch (to get the correct NAND partition layout), then sysupgrade -n the sysupgrade to create a clean config. Otherwise, the UBI partition extents will probably mismatch, and cause issues.

Hm, yeah it looks like there is a difference for sure as I rebuilt everything, ubiformated and then ran sysupgrade.
I changed the custom rootfs partition name to ubi to avoid bootargs.

Now it throwing the ELF error and really pissing me off.

If you still want to try to use ubiupdatevol for kernel, you could try ubiformat -f blank_kernel.ubi when the kernel ubi part gets formatted, instead of ubiformat followed by ubimkvol?
Not sure if it would work though.
Something like this for the ubinize config :

[kernel]
mode=ubi
vol_id=0
vol_name=kernel
vol_type=dynamic
vol_alignment=1
vol_size=8MiB

But shouldnt ubiformat do the same as ubiformating with an empty UBI?

Turns out there is quite a bit of overhead for a UBIFS (1.6MiB for an empty test directory).
Then, UBI reserves PEBs on each UBI device, depending on NAND size: http://linux-mtd.infradead.org/doc/ubi.html#L_max_beb

My 450gx4 kernel.ubifs is 4464KiB, for a 2847KiB kernel.

This was the only way I could get ubimkvol & ubiupdatevol to work on my small kernel UBI partition, on the big NAND device:

source /lib/upgrade/nand.sh
ubiformat /dev/mtd0
ubiattach -m 0 --max-beb-per1024=4
# otherwise, no room left for mkvol
ubirmvol "/dev/$(nand_find_ubi ubi_kernel)" -N kernel || true
ubimkvol "/dev/$(nand_find_ubi ubi_kernel)" -N kernel -m
ubiupdatevol "/dev/$(nand_find_volume $(nand_find_ubi ubi_kernel) kernel)" /tmp/mikrotik_rb450gx4-128k-kernel.bin.ubifs

This then booted from RouterBOOT, but I am using the OEM partition layout.
My ubiformat ubinized kernel sysupgrade method was simpler, as no custom ubiattach needed: https://github.com/openwrt/openwrt/compare/master...john-tho:450gx4

On the 128MiB NAND hapac3 flash UBI would reserve less space, but just to be sure: Have you checked your flashed kernel image volume mounts and validates, after ubiformat & sysupgrade, from the netboot initramfs image? I also saw you were using a much bigger -c, --max-leb-cnt for your mkfs.ubifs than I am.

After testing nanddumped ubiformat ubinize images versus ubimkvol with the nandsim kernel module on 512MiB NAND:

  • The name of the test volume (part of the UBI volume layout information; even though RouterBOOT probably does not use volume name) can move around
    • appears early (0x1010) for ubinized image
    • or late (0x10021010) for ubimkvol, or after ubimkvol is run after the ubiformated image, or after ubiattach with an autoresize flag in ubinized image

A guess: If RouterBOOT (to save boot time) limits the amount of NAND it will scan (to something like the OEM kernel partition size), it may miss the kernel if OpenWRT's UBI stores whatever it is that RouterBOOT is searching for beyond this limit (whether it is UBI volumes table, or UBIFS, or something else).

nandsim testing on 5.8 x86_64
# setup
sudo rmmod nandsim
sudo modprobe nandsim first_id_byte=0x20 second_id_byte=0xac third_id_byte=0x00 fourth_id_byte=0x15
# 512MiB, 2048, 128k
# https://elixir.bootlin.com/linux/v5.11.15/source/drivers/mtd/nand/raw/nandsim.c
sudo modprobe ubi
sudo modprobe mtd

sudo mtdinfo /dev/mtd1 --ubi-info


# ubimkvol

sudo ubiformat --sub-page-size=2048 /dev/mtd1
sudo ubiattach --vid-hdr-offset=2048 --dev-path=/dev/mtd1
#sudo nanddump /dev/mtd1 > nand.img
sudo ubimkvol /dev/ubi0 -N kernel -s 20MiB
sudo nanddump /dev/mtd1 > nand.img
binwalk --raw='kernel' nand.img
sudo ubidetach --dev-path=/dev/mtd1


#Dumping data starting at 0x00000000 and ending at 0x20000000...

#DECIMAL       HEXADECIMAL     DESCRIPTION
#--------------------------------------------------------------------------------
#268570640     0x10021010      Raw signature (kernel)
#268701712     0x10041010      Raw signature (kernel)


# ubinized

cat <<EOF > ubi.ini
[kernel]
mode=ubi
vol_id=0
vol_name=kernel
vol_type=dynamic
vol_alignment=1
vol_size=20MiB
[rootfs]
mode=ubi
vol_id=1
vol_name=rootfs
vol_type=dynamic
vol_alignment=1
vol_size=20MiB
[rootfs_data]
mode=ubi
vol_id=2
vol_name=rootfs_data
vol_type=dynamic
vol_alignment=1
vol_size=1MiB
#vol_flags=autoresize
EOF

ubinize --peb-size=128KiB --min-io-size=2048 --output=test.ubi ubi.ini

sudo ubiformat --sub-page-size=2048 /dev/mtd1 --flash-image=test.ubi
sudo nanddump /dev/mtd1 > nand.img
binwalk --raw='kernel' --raw='rootfs' nand.img


#DECIMAL       HEXADECIMAL     DESCRIPTION
#--------------------------------------------------------------------------------
#4112          0x1010          Raw signature (kernel)
#4284          0x10BC          Raw signature (rootfs)
#4456          0x1168          Raw signature (rootfs)
#135184        0x21010         Raw signature (kernel)
#135356        0x210BC         Raw signature (rootfs)
#135528        0x21168         Raw signature (rootfs)


# after attach

sudo ubiattach --vid-hdr-offset=2048 --dev-path=/dev/mtd1
sudo ubidetach --dev-path=/dev/mtd1
sudo nanddump /dev/mtd1 > nand.img
binwalk --raw='kernel' --raw='rootfs' nand.img


#DECIMAL       HEXADECIMAL     DESCRIPTION
#--------------------------------------------------------------------------------
#4112          0x1010          Raw signature (kernel)
#4284          0x10BC          Raw signature (rootfs)
#4456          0x1168          Raw signature (rootfs)
#135184        0x21010         Raw signature (kernel)
#135356        0x210BC         Raw signature (rootfs)
#135528        0x21168         Raw signature (rootfs)

# rmvol & mkvol

sudo ubiattach --vid-hdr-offset=2048 --dev-path=/dev/mtd1
sudo ubirmvol /dev/ubi0 -N rootfs_data
#sudo ubirmvol /dev/ubi0 -N rootfs
#sudo ubimkvol /dev/ubi0 -N rootfs -s 20MiB
#sudo ubirmvol /dev/ubi0 -N rootfs_data -s 20MiB
sudo ubidetach --dev-path=/dev/mtd1
sudo nanddump /dev/mtd1 > nand.img
binwalk --raw='kernel' --raw='rootfs' nand.img


#DECIMAL       HEXADECIMAL     DESCRIPTION
#--------------------------------------------------------------------------------
#269619216     0x10121010      Raw signature (kernel)
#269619388     0x101210BC      Raw signature (rootfs)
#269750288     0x10141010      Raw signature (kernel)
#269750460     0x101410BC      Raw signature (rootfs)