MikroTik rb450gx4 support

vester_11 · October 31, 2020, 7:57pm

@robimarko @bm42 Do You have working image with Winbond NAND?

joergschne · January 11, 2021, 3:58pm

@robimarko Thanks for everything. Works great with the exception of the latest generation of the RB450Gx4 with RouterBoot 6.45.9 and AMD Flash. The RAM image will still work, but the device will not start after sysupgrading the standard image you provided.

KR

RouterBOOT booter 6.45.9

RB450Gx4

CPU frequency: 716 MHz
  Memory size: 1024 MiB
    NAND size: 512 MiB

Press any key within 2 seconds to enter setup..

loading kernel... OK
setting up elf image... OK
jumping to kernel code

OpenWrt kernel loader for Qualcomm IPQ-4XXX/IPQ-806X
Copyright (C) 2019  Sergey Sergeev <adron@mstnt.com>


Op ret = -97
Auto reboot in 5 sec

[    0.786943] nand: device found, Manufacturer ID: 0x01, Chip ID: 0xdc
[    0.787783] nand: AMD/Spansion S34ML04G2
[    0.794437] nand: 512 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 128

robimarko · January 11, 2021, 4:01pm

Its because of block and page sizes

joergschne · January 11, 2021, 4:11pm

Thanks for the quick reply. Would a donated device help fix the problem? If so, then I need a postal address from you.

KR

robimarko · January 11, 2021, 4:53pm

I have an RB450Gx4 with the older NAND, I know the issue with the never, and it simply requires different block/page sizes.

RB450Gx4 support is on hold until hAP ac2 is merged as it introduces all of the stuff needed for any other Mikrotik board.

xback · January 11, 2021, 8:32pm

Any chance of opening up the device and providing the exact details on the nand chip?

I ran into a similar issue on a rb2011 and had to patch the full chip description in the kernel as described in thread marked below.
It looks like it gets detected, but the it's probably badly detected.

See this thread

robimarko · January 11, 2021, 9:00pm

Its detected properly, but MikroTik has 2 revisions of the board.
One NAND uses 128k blocks while other one uses 256k blocks.

joergschne · January 12, 2021, 2:51pm

There is a single Spansion S34ML04G200TF100 NAND mounted as U902 and it looks like I/O0 - I/O7 (×8) is connected to the CPU.

❐Page size:
•×8:
1 Gb: (2048 + 64) bytes; 64-byte spare area
2 Gb / 4 Gb: (2048 + 128) bytes; 128-byte spare area

•×16:
1 Gb: (1024 + 32) words; 32-word spare area
2 Gb / 4 Gb (1024 + 64) words; 64-word spare area

❐Block size: 64 Pages
•×8:
1 Gb: 128 KB+ 4 KB
2 Gb / 4 Gb: 128 KB + 8 KB

•×16
1 Gb: 64k + 2k words
2 Gb / 4 Gb: 64k + 4k words

❐Plane size
•×8
1 Gb: 1024 blocks per plane or (128 MB + 4 MB
2 Gb: 1024 blocks per plane or (128 MB + 8 MB
4 Gb: 2048 blocks per plane or (256 MB + 16 MB

•×16
1 Gb: 1024 blocks per plane or (64M + 2M) words
2 Gb: 1024 Blocks per Plane or (64M + 4M) words
4 Gb: 2048 Blocks per Plane or (128M + 8M) words

ppcharli · April 6, 2021, 5:47am

Now that hAP ac2 is merged. Could it be possible to merge rb450gx4 in OpenWrt 21.02?
Thanks,

robimarko · April 6, 2021, 8:29pm

I doubt 21.02, but it needs to be rebased on the current master as the code is way out of sync.
I honestly have no idea when will I get to it.

johnth · April 15, 2021, 8:54am

I have been doing some testing with a netboot image. Started from @robimarko's work, and rebased to 2021 master 1.

I am not sure how we should partition the NAND. Please provide any advice or corrections.

OEM splits NAND to two mtd paritions:

0x000000000000-0x000000800000 : "RouterBoard NAND 1 Boot"
UBI volume "Kernel"
contains the ELF kernel, and VERSION
0x000000800000-0x000020000000 : "RouterBoard NAND 1 Main"
UBI volume "RouterOS"
contains bootimage (NPK), plus various directories & files

I guess RouterBoot loads the mtd0 UBI volume Kernel/kernel ELF, which loads the mtd1 UBI volume RouterOS/bootimage.

From what I can see, OpenWrt upgrade/nand.sh 2 is only designed to work with one UBI (mtd) partition, usually labelled "ubi" in DTB, with possibly multiple volumes under it:
local ubidev="$( nand_find_ubi "$CI_UBIPART" )"
With the image was using adron's loader, the kernel was packed into a ubi partition image. For tar sysupgrade, with an mtd partition labelled "kernel", nand_upgrade_tar for kernel only uses openwrt's mtd 3 to write to the mtd partition, which would lose the UBI erase counts?

Options?

Keep OEM mtd partitions:
- extend or build custom upgrade script to install the self-extracting ELF kernel to mtd0 UBI partition Kernel/kernel:
  - Allow different CI_UBIPART for rootfs & kernel?
  - mount, and replace the file, or
  - ubiupdatevol?, and
- ubiformat mtd1 UBI partition to Openwrt "ubi" volumes for rootfs

I have not tried, or looked into RouterBoot to check the limitations to see if we could remove OEM mtd partitions, and use single UBI partition. Might lose UBI erase counts in this process?

define the PoE output compat GPIO input as a gpio key (extcon-gpio does not have DTS bindings)?
define the PoE output control GPIO as a LED, or leave as sysfs / gpio-export?

Any feedback would be great.

Cheers,

robimarko · April 15, 2021, 11:23am

I dont think you can unify the partitions as from my testing it is looking for a UBIFS packed kernel

robimarko · April 17, 2021, 5:56pm

I started working on hAP ac3, it also has the same NAND installation issue.
The layout is somewhat weird as they use UBI with volumes for Kernel and Rootfs, but both are then packed in XZ compressed UBIFS.

So, I am having issues in figuring a way to have a permanent flash with sysupgrade.

robimarko · April 17, 2021, 7:00pm

Ok, so I did some digging and testing.
It appears that RouterBoot does not require kernel and rootfs to be separated with partitions but only UBI volumes.
It also does not care if the kernel is named "kernel" or "Kernel" like in ROS images.
All it cares about is that the kernel is packed in UBIFS, without UBIFS it won't boot the kernel at all, so I manually packed the kernel in UBIFS per what stock kernel is packed and ubinized it and that boots.

So, we just gotta find a way to pack the kernel in UBIFS in OpenWrt buildroot directly.

johnth · April 17, 2021, 8:43pm

Nice work. I have some make hackery to do the packing. It would be great if you tested to check if it works for hap ac3 with a single partition. Please let me know your RouterBOOT version as well. Would be really nice to disassamble RouterBOOT to find the ELF boot constraints, but I have not had any luck so far (can see truncated strings that are used in RouterBOOT console near the end of RouterBOOT fwf…).

For my rb450gx4, RouterBOOT 6.47.4 128k eraseblock NAND

ubinized image would not fit in NAND part1
did not work if I ubiformated NAND part1, and used a ubinized image in NAND part2
loading kernel... kernel not found or data is corrupted
did work if I manually ubiformated NAND part1 with ubinized ELF self extracting kernel
did not work if I turned NAND part1 into NAND part1+part2 in DTS, and used ubinized image
```
loading kernel... OK
setting up elf image... not an elf header
kernel loading failed
```
I did check that UBI Kernel/kernel had ELF header, and the same sha256sum in build_dir as on device

From what you have found (single part worked), it may be that my packing had an error, or RouterBOOT did not like what ubinized.sh generated. It is the commented-out recipe in Device/mikrotik_nand here.

For my ubinized sysupgrade single part test:
- used only NAND part1, with label ubi in DTS, and offset 0, with size that covered OEM part1 + part2
- used the commented-out KERNEL, IMAGE/sysupgrade.bin, and UBINIZE_PARTS recipies
  This built a single ubi(inized) sysupgrade image, which the default sysupgrade_nand could then flash (with ubiformat)
  If this does not work, that is great, as a single partition NAND UBI might work for my 450gx4. Then we check the kernel & compare the UBIFS & UBI settings.
- sysupgrade ubinized:
```
. /lib/upgrade/nand.sh

CI_UBIPART=ubi
nand_do_upgrade
```
- if Kernel == kernel volname case does not matter to RouterBOOT, can remove UBINIZE_PARTS, and use KERNEL_IN_UBI := 1 instead.
For my OEM NAND parts1+2 test:
- This is the current recipe.
- Ignore the custom upgrade script for now, it does not work yet (variable scope I guess, copy-paste stepping through it worked)
  It only matters if we care to keep UBI erasecounts in the first NAND part.
  Otherwise, name part1 kernel, and nand_upgrade_tar overwrites it with the non-UBI aware OpenWRT mtd flasher

Good luck!

robimarko · April 17, 2021, 9:21pm

I am using 6.46.6 as for me 6.47 and newer RouterBoot has broken DHCP.
Its probably even the same NAND on RB450Gx4 and hAP ac3 I have.
Mine has Winbond W29N01HV 128k blocks, 2048 pagesize.

I am also hitting the:

loading kernel... OK
setting up elf image... not an elf header
kernel loading failed

If ubiformating with an OpenWrt generated UBI factory image.
Weird thing is that I pulled the kernel from OpenWrt and then manually packed that into UBIFS and ubinized it, then simply ubiformated and RouterBoot booted that without any issues.
Naturally, rootfs was missing but it still proved that it does not care for naming.

Now I am going over the parameters I used in UBIFS packing manually and with OpenWrt.
I added a simple packer to image-commands.mk:

define Build/package-kernel-ubifs
	mkdir $@.kernelubifs
	cp $@ $@.kernelubifs/kernel
	$(STAGING_DIR_HOST)/bin/mkfs.ubifs \
		$(UBIFS_OPTS) \
		-r $@.kernelubifs $@
	rm -r $@.kernelubifs
endef

$(UBIFS_OPTS) used are: -m 2048 -e 124KiB -c 2048 -x none

So, this makes no sense to me.

I tested to see whether packer breaks the kernel image and no, after extracting that one boots just fine over DHCP.

johnth · April 17, 2021, 10:03pm

Thanks. Good to see.

I did not save an initramfs with single NAND part (overwritten), so I will need to recompile for the following:

Next suggestions to test RouterBOOT not an elf header
With the single NAND part, with working kernel volume

Boot initramfs, then attach the UBI part, and ubimkvol:
- -N rootfs -s $(size of build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-ipq40xx_mikrotik/root.squashfs)
- -N rootfs_data -m (rest of space)
  If RouterBOOT still boots kernel, then boot initramfs again, and ubiupdatevol rootfs to the root.squashfs from build_dir

Otherwise manually ubinize kernel + rootfs + rootfs_data? Something like this?

[Kernel]
mode=ubi
vol_id=0
vol_type=dynamic
vol_name=Kernel
#vol_size=8MiB
image=build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-ipq40xx_mikrotik/mikrotik_rb450gx4-128k-kernel.bin.ubifs
[rootfs]
mode=ubi
vol_id=1
vol_type=dynamic
vol_name=rootfs
image=build_dir/target-arm_cortex-a7+neon-vfpv4_musl_eabi/linux-ipq40xx_mikrotik/root.squashfs
#vol_size=2535424
[rootfs_data]
mode=ubi                                                                        
vol_id=2
vol_type=dynamic
vol_name=rootfs_data
vol_size=1MiB
vol_flags=autoresize

johnth · April 18, 2021, 12:03am

Okay, I think the issue is coming from ~~append-ubi or ubinize.sh~~ the rootfs image in the ubinized image.
Manually got a working single NAND partition system on rb450gx4. Thanks for the hint.
ubiformated with append-ubi image, removed vols rootfs & rootfs_data, recreated them (rootfs had a different size), and ubiupdatevol rootfs
Now to work out why.

My notes / log

single NAND partition + ubinized ubifs kernel
with KERNEL_IN_UBI:=1 (need to cp $@.ubifs to $@ in from my earlier commit)
- Bad: error:
```
loading kernel... OK
setting up elf image... not an elf header
kernel loading failed
```

cut _rootfs_data volume

ubirmvol /dev/ubi0 -N rootfs_data
sync; reboot

Bad: same error

cut rootfs volume
```
 ubirmvol /dev/ubi0 -N rootfs
 sync; reboot
```
- Good: Routerboot loads kernel fine
reverse process:

add rootfs vol

 ubimkvol /dev/ubi0 -N rootfs -s 10MiB
 sync; reboot

Good: Routerboot loads kernel fine

scp the rootfs:

tar -C /tmp -xf /tmp/openwrt-ipq40xx-mikrotik-mikrotik_rb450gx4-128k-squashfs-sysupgrade.bin_OEM_NAND_parts
ubiupdatevol /dev/ubi0_1 /tmp/sysupgrade-mikrotik_rb450gx4-128k/root
sync; reboot

Good: Routerboot loads kernel fine; squashfs gets mounted

Recreate rootfs_data
```
ubimkvol /dev/ubi0 -N rootfs_data -m
sync; reboot
```
- Good: Routerboot loads kernel fine; squashfs gets mounted; overlay gets used

Guess so far: RouterBOOT (for one NAND UBI partition) seems to have problems with any ubiformat flashed UBI image that has a ubinize stuffed root in it (squashfs, or ubifs).
squashfs.root UBI volume updated with ubiupdatevol on the device works fine

@robimarko I am leaning towards leaving the NAND partitioning as OEM had it? The first specifically for kernel.
So far, RouterBOOT is refusing (with ELF header error) to load the known-good kernel ELF, when ubinize has also packed a rootfs (as another volume, either squashfs or ubifs) into the UBI partition.

robimarko · April 18, 2021, 9:11am

Hmm, those are some good findings.

Single partition is gotta work as that is what MikroTik uses on hAP ac3, they just have 2 UBIFS packed volumes.
One is for kernel and the other one is rootfs.

So, could it be that it freaks out with more than 2 volumes or requires empty space after the kernel?
Tried UBIFS rootfs as that reduces the UBI volumes to only 2 and same ELF error.

Hm, using sysupgrade makes things even worse as then we get the not found error.

I am really confused as stock FW uses the same layout and UBIFS for both there is not even empty PEB-s after the kernel.

Ok, looking at the generated ubifs factory image and NAND dump I can see one big difference.
OpenWrt generated one is identified as one image with 2 volumes, while stock has 2 images each with its own volume.
That points again to 2 partitions being used, so sorry for me misleading into a single partition.

robimarko · April 18, 2021, 5:10pm

@johnth I had some free time today, so I implemented UBI kernel support to flash the kernel in a separate partition but in UBI.

Its still hit and miss and worked for me on the first try but after couple more tries it will not boot the flashed kernel.

Maybe you can test it and spot a bug as it now makes even less sense that it sometimes works and sometimes not.

My branch is here: https://github.com/robimarko/openwrt/tree/mikrotik/hap-ac3