Resizing kernel images for Edgerouter X

For Developers
1

I noticed builds for Edgerouter X have been disabled in commit due to 3MB kernel size limits with the OEM layout.

I thought I would take a look at resolving this issue so this device can continue to work with newer kenels. However wondering if anyone has any hints on the best way forward to resize the partitions.

Can't simply resize in dts as some other device have done recently, since the kernel partitions are consecutive here.

I guess that leaves a couple of options

  • Convert layout to a single kernel allowing 6MB and disable the code that sets kernel flag on flash upgrades.
  • Resize both kernel partitions and move ubi start offset, subsequently override load addresses within uboot.

Any hints on the best option would be appreciate or let me know if there is a better way. I do have some experience working with uboot, but never anything as ancient as the uboot Ubuitiy choose!
OEM layout

root@OpenWrt:/proc# cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00080000 00020000 "u-boot"
mtd1: 00060000 00020000 "u-boot-env"
mtd2: 00060000 00020000 "factory"
mtd3: 00300000 00020000 "kernel1"
mtd4: 00300000 00020000 "kernel2"
mtd5: 0f7c0000 00020000 "ubi"
root@OpenWrt:/proc# ubinfo -a
UBI version:                    1
Count of UBI devices:           1
UBI control device major/minor: 10:127
Present UBI devices:            ubi0

ubi0
Volumes count:                           2
Logical eraseblock size:                 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks:     1972 (250396672 bytes, 238.7 MiB)
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       10
Count of reserved physical eraseblocks:  30
Current maximum erase counter value:     3
Minimum input/output unit size:          2048 bytes
Character device major/minor:            250:0
Present volumes:                         0, 1

Volume ID:   0 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        22 LEBs (2793472 bytes, 2.6 MiB)
State:       OK
Name:        rootfs
Character device major/minor: 250:1
-----------------------------------
Volume ID:   1 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        1916 LEBs (243286016 bytes, 232.0 MiB)
State:       OK
Name:        rootfs_data
Character device major/minor: 250:2

uboot env, although this seems to be for recovery, so I guess uboot is configured dynamically via the menu system.

oot@OpenWrt:/proc# fw_printenv
bootcmd=tftp
bootdelay=1
baudrate=57600
stdin=serial
stdout=serial
stderr=serial
ethaddr=ac:8b:a9:a4:45:42
bootargs=console=ttyS1,57600n8 ubi.mtd=7 root=ubi0_0 rootfstype=ubifs rootsqimg=squashfs.img rootsqwdir=w rw model=e50
BootType=3
filesize=44ee27
fileaddr=80A00000
ipaddr=172.16.3.211
serverip=172.16.3.210
autostart=no
bootfile=vme50
4 Likes
2

This seems like the most obvious thing.

The addresses for kernel1 and kernel2 are not being kept in the u-boot env variables. I do not see an obvious way to change them.

Then there is the problem of flashing this bigger image to existing 3MiB slot. I am not sure but I assume that dd in nand.sh respects the size limit of existing mtd partitions. So how do we upgrade from smaller kernel slots to combined one? One way is to boot using an initramfs image. Another option is via kernel v6.1 snapshop image, which seems to fit in 3MiB still, but then all device owners would have to be really alert to update when main is still in 6.1. Moreover, thanks to kernel slot toggling, you'd have to booted using kernel2 in order to get new image to flash in kernel1.

Tricky.

3

yeh I suspect they are being set dynamically in the boot script

I dont think this is the case, some other boards resize kernel just by moving the rootfs start address in dts, so that the kernel itself can overrun the kernel mtd slot. Also in nand.sh, kernel length seems to defined based on file size not mtd size.

Kernel toggling is handled in custom implementation specific to edgerouter (X and X-sfp), seems easy enough to override this to always flash to kernel1. However I presume there must be some failover logic in uboot also that flips the kernel to point after boot failures?

4

I was thinking about the case that how does the existing sysupgrade in branches 22.x and 23.x handle the new image with updated DTS. They are aware of two slots only and will install the new kernel+dtb accordingly in one of them. For the next boot it would have to boot from where kernel1 was, right? Yes, no?

I am thinking about using the DEVICE_COMPAT_VERSION and lay out the way for combined kernel slot while main is still in 6.1.

5

So as a first attempt, I made the following changes to Edgerouter X build:

  • patch dts to make kernel1 6MB and drop kernel2
  • patch mt7621.mk to set kernel size to 6MB
  • patch ubnt.sh to always flash to kernel1 (and update kernel index flag to match)

Then using serial console

  • select <1> from uboot menu
  • tftpboot the new initramfs
  • sysupgrade to the new sysupgrade.bin

The system rebooted successfully into new build from flash with kernel 6.1 (size ~3.8MB) and new layout:

root@OpenWrt:/# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00080000 00020000 "u-boot"
mtd1: 00060000 00020000 "u-boot-env"
mtd2: 00060000 00020000 "factory"
mtd3: 00600000 00020000 "kernel1"
mtd4: 0f7c0000 00020000 "ubi"

Obviously all this is dependent on having a serial console.

Not sure that flashing factory file from OEM OS would work, given its now over 3MB? and how would select correct kernel slot in the case.

Perhaps it might be possible to do a direct upgrade to the new layout from existing openwrt install, by overwriting ubnt.sh before sysupgrade?

6

maybe replacing ubnt.sh on the live filesystem before running sysupgrade might work? that way can force the new kernel to flash into kernel1 slot and ensure this slot is active. I will roll back to 23.05 and see if that works.

EDIT: The above does work, by updating ./lib/upgrade/ubnt.sh prior to sysupgrade from 23.05 and reboots into new layout.

Is the use of COMPAT_VERSION to force the new layout?

7

Ok good. I may have been too hasty in thinking it must land on kernel1. U-boot is not aware of mtd partitions and just loads what it finds at given address, right?

Suppose we introduce a patch where we update DTS to join slots kernel1 and
kernel2 and we also modify ubnt.sh to always use kernel1 slot.

Should this kernel+dtb land on kernel2, u-boot would load it as normal and proceed without problems. Then we would have a situation where kernel is being loaded from middle of mtd3 and we'd have the updated ubnt.sh in place. A subsequent sysupgrade would drop the new kernel in place of kernel1 and allow 6MiB kernel size from there onwards.

(Also, I think we can go back and forth between releases as rolling back from 24.x to 23.x or earlier should not be a problem. It'll just land on kernel1 and then it would start toggling again.

Is my logic sound enough? Dunno if we need any COMPAT_VERSION.)

Edit: on a second thought, I am inclined to think that we'll have to introduce the new DTS and ubnt.sh also in 22.x and 23.x so that they'll be able to update a kernel that's bigger than 3MiB, which will be the situation in 24.x. And for that we'll might need COMPAT_VERSION message telling users to update latest of 22.x or 23.x first.
How does it sound now?

8

U-boot selects kernel parition to boot based on flag stored in factory mtd.

MT7621 # ubntw dump
Reading NAND flash factory partition 0x000E0000 size 393216
UBNT BD type=e50, mac=AC8BA9A44542, s/n=AC8BA9A44542, mrev=22, k_idx=1

The patched ubnt.sh always sets the flag to boot kernel1

With the patches applied on a existing stable release where kernel is < 3MB this would work as you describe. 6.1 is probably already over 3MB hence why the builds have been disabled in master.

Yes, I can confirm that rolling back to 23.x works fine without any special steps.

Yes I think that would be best, if we can backport this to stable releases, they will get the new layout on subsequent sysupgrade with no other interaction required.

EDIT: not sure if we should or need to backport the DTS change to stable releases. Just having the updated ubnt.sh there should be enough. Flashed kernel image gets truncated at 3MB if new dts is not loaded

I have created a draft PR with changes for main:

And a branch for backports here:

4 Likes
9

Tested:
OpenWrt 23.x -> main with 6MB slot. It gave a warning as expected.
OpenWrt 23.x -> 23.x with 6MB slot landing on 'kernel2'. Works as expected.
23.x with 6MB slot -> main (6.6) with 6MB slot. Works as expected
main with 6MB slot -> OpenWrt 23.x. Forced, but was able to downgrade. Worked as expected.

4 Likes
10

Is there any chance that what you are trying to do and the resulting OpenWrt images would be able to be used for an initilal OpenWrt install via the stock edgeOS or perhaps via the bootlader of an Edgerouter X?

Currently, as far as I can tell, it is still necessary to use a special, customized, years old, unofficial OpenWrt image first because of the size limit.

I am very new to OpenWrt and not experienced with the command line, so having to use that old unofficial OpenWrt image, as well as reading several posts about problems with that install process, makes me hesitant to try it. I'd happily wait if there is a chance for a simpler and easier initial install method.

Thanks.

11

I think you could also use an old official factory image for initial install then upgrade freely.

Loading and booting an intramfs through serial means the image is never stored in flash, so it is free of the size limit. I have found this the simplest way to install.

1 Like
12

No there is nothing here that fixes the factory images. Once my PR's are merged and there are images available you would have two options.

Web based with multiple steps:

  • Install the old custom 22.03 factory image through edgeos
  • Sysupgrade (via webui) to a 23.05 image containing the new changes for kernel size
  • Sysupgrade to snapshot or future 24.x release

Serial (as mk24 mentions):

  • Boot 24.x initramfs via serial console + tftp
  • sysupgrade directly to the new release
13

darkxst and mk24, thanks for the replies.

For the first method via edgeOS:

  1. Are you sure there will be a webUI after installation of the custom image? I understood that luci is not installed there, though I might well be wrong.
  2. Do the current 23.05 images already include the kernel size changes and if not, how would I be able to know when they do?

As for the serial installation method, it looks even more intimidating to me than the other one, though I do like that it only uses the current official OpenWrt images.

I'll think about which method, if any, I might try. In any case, I appreciate the time and effort that you and others put into making OpenWrt work.

14

Yes the builds from the stman repo include Luci

Not yet, my PR's still need to be merged, but then hopefully they make it into the next official release i.e 23.05.4 later in the year.

15

I wonder if an issue needs to be created for 23.05 saying that it's not possible to upgrade to 24.x as it looks now. I didn't find one yet.

16

Probably at some point, but perhaps it should wait until the changes are in main first?

17

OK. Thanks for the information.

18

So it seems like patches to sysupgrade wont be accepted for backport to 23.05 :frowning:

This PR now instead provides migration script that can be run on 23.05 (or older) and will install the new layout.

Hoping this can be accepted for backport.

1 Like
19

Done

root@OpenWrt:~# cat /proc/mtd
dev: size erasesize name
mtd0: 00080000 00020000 "u-boot"
mtd1: 00060000 00020000 "u-boot-env"
mtd2: 00060000 00020000 "factory"
mtd3: 00600000 00020000 "kernel"
mtd4: 0f7c0000 00020000 "ubi"

I didn't find snapshot with kernel 6.6
Do you know where I can download it?

Thanks

20

Hi Deto,
Snapshot builds are disabled until my PR can be merged, until that happens you will need to build your own images if you want to test with Linux 6.6.

2 Likes