Snapshot not available for certain supported devices recently

Hi,

One specific example is:
https://lede-project.org/toh/hwdata/belkin/belkin_f9k1115v2_v2

The firmware for this device is available in LEDE release but missing in snapshot. I did a diff of the list of image available in release VS those in snapshot and there were quite some other devices missing as well.

At least for f9k1115v2 (basically an archer c7 from belkin) the snapshots used to be available as recent as october 19 but it cease to be the case.

Why is it? Where can we check if buildbot compilation for certain device are failing?

http://phase1.builds.lede-project.org/builders/ar71xx%2Fgeneric/builds/516/steps/images/logs/stdio

It appears that the kernel partition for the Belkin F9K1115 v2 is limited to 1408 KB, which is no longer enough for kernel >= 4.9 (currently 1465.3 KB are required, but this will grow a bit more for kernel 4.14; this was apparently 1239.94 kB for kernel 4.4 in lede-17.01). Several other devices with these issues were able to lift this restriction by slightly changing the mtd partitioning. If similar approaches are possible for the F9K1115 depends on the partition order, how the bootloader accesses the kernel and if the OEM firmware updater accepts a different split between kernel and rootfs (for the factory images) - and it will obviously need experimenting and confirmation (plus a pull request) by a user of this device.

Thanks for the info.

Could you point me to one commit that adjust the mtd partitioning?

Currently the mtd partitioning table for the f9k115v2 looks like this:

dev: size erasesize name
mtd0: 00040000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00e20000 00010000 "rootfs"
mtd3: 00be0000 00010000 "rootfs_data"
mtd4: 00160000 00010000 "kernel"
mtd5: 00010000 00010000 "nvram"
mtd6: 00010000 00010000 "envram"
mtd7: 00010000 00010000 "art"
mtd8: 00f80000 00010000 "firmware"

so looks like kernel (mtd4) sits after rootfs_data (mtd3). We can probably squeeze couple hundred kilobytes there.

As for compatibility with stock firmware...Would it be possible to have users install pre kernel 4.9 version image then upgrade to latest version? Going back to stock may leverage the same pre kernel 4.9 version to adjust the partition layout back to stock. I also hardly believe anyone would want to go back to stock which has not been updated for a long time as a result vulnerable to the many wifi vulnerability from recent months...

That device might not be easy as the kernel is after the rootfs, and I expect that the bootloader expects to find kernel are a certain address. changing kernel start location on flash would possibly require u-boot changes. Your device's partition layout is rather non-standard.

Some relevant commits where kernel end location was expanded to enable enable compilation again. Note that these devices had kernel before rootfs, so it was easy to just expand kernel and decrease rootfs. Kernel start location stayed the same.

https://git.lede-project.org/?p=openwrt/openwrt.git;a=commitdiff;h=ca6002c0efab2e847be91b13d558e5c38184b50a;hp=3f96e57aed69fcd88a8d4f06bbf97688f67ca343

https://git.lede-project.org/?p=openwrt/openwrt.git;a=commitdiff;h=55c77b3d3c7a85a2bbf891437e0f6aee4021fc96;hp=a4bb13e720315c9f4688248fc11d1727ef53846c

1 Like

Exactly that, it will be rather difficult to come up with a fix for your device. You won't be able to avoid reverse engineering your OEM firmware format a bit more (reading the GPL source might be required) and experimenting quite a bit (serial console pretty much required, external SPI-NOR flasher and full-flash very strongly recommended).

Food for thoughts, u-boot-env is a clearly labelled -dedicated- partition, you may be able to use uboot-envtools, query fw_printenv and change it from the installed system. That might allow you to change the bootloader mtd partitioning (warning, this is dangerous and might prevent you from reverting back to the OEM firmware) - as at least rootfs, rootfs_data and kernel are behind each other, without holes inbetween. If you are very, very lucky, the OEM firmware format might have means to change u-boot-env as part of the upgrade process (careful not to break potential tftp recovery procedures or bootloader based factory resets), if not, you need an intermediate step (e.g. initramfs image) - but whatever possible, this will make reverting back to the OEM firmware pretty difficult.

Hi hnyman,

Thanks for the pointers.

I checked out the initial commit that enabled support for f9k1115v2, and seems that the generated image is patched with a edimax firmware header patcher.

Initial commit:
https://github.com/lede-project/source/commit/211cac98d559fdcb11fd0825aaf165da15c87b71

The make file for this device is already moved to legacy:

edimax_fw_header is invoked and seems like it is possible for it to accept a "start" and "end" address. This is a wild guess, but from your experience do you think it is possible that uboot "reprogram" itself with new boot arguments (specifically, kernel / rootfs partition offset) by reading headers of an image as part of the flashing process?

HI slh,

I downloaded the source code for f9k11115v2 and looked in the uboot source code and found the following which seems to correspond to reading headers generated by the edimax header patcher:

int eraseall_func(uint8_t *upg_addr){
	uint8_t hcrc=0, dcrc=0;
	unsigned long start=0, end=0, size=0, data=0;
	fw_header_s hdr, *phdr=NULL;
	uint8_t *p=NULL;
	p=(uint8_t *)upg_addr;
	data=(unsigned long)(upg_addr+sizeof(fw_header_s));
	memset(&hdr, 0, sizeof(fw_header_s));
	memcpy(&hdr, p, sizeof(fw_header_s));
	phdr=(fw_header_s *)p;
	printf("######################### Eraseall %01x ########################\n", eraseall_count);
  if(strcmp(phdr->magic, "eDiMaXsUpEr")!=0 && strcmp(phdr->magic, MAGIC_STRING)!=0) goto run_cmd_err; 					//check header "super magic" and "magic" string
	hdr.hcrc=0;
	hcrc=checksum((char *)&hdr, sizeof(fw_header_s));
	dcrc=checksum((char *)(p+sizeof(fw_header_s)), hdr.size);
	start=ntohl(phdr->start_addr);
	size=ntohl(phdr->size);
	if(!strcmp(phdr->mtd_name, "u-boot")){
		start+=0x9f000000; 
	}
	if(!strcmp(phdr->mtd_name, "uImage")){
		start+=0x9fe70000; 
	}
	if(!strcmp(phdr->mtd_name, "rootfs")){
		start+=0x9f050000; 
	}
	end=start+size-1;
	printf("Header Data: \n");
	printf("Model Name:     %.*s\n", 32, phdr->model);
	printf("Data Size:      %ld Bytes\n", size);
	printf("Start Address:  0x%08X\n", start);
	printf("End Address:    0x%08X\n", end);
	printf("MTD Name:       %.*s\n", 16, phdr->mtd_name);
	printf("Version:        %.*s\n", 14, phdr->version);
	if(hcrc!=ntohl(phdr->hcrc)) goto hcrc_err; 
	if(dcrc!=ntohl(phdr->dcrc)) goto dcrc_err; 
	current_upg_addr+=sizeof(fw_header_s)+size;
	//if(uboot_run_cmd("protect off %lx %lx", start, end)<0) goto run_cmd_err; 
	if(uboot_run_cmd("protect off all")<0) goto run_cmd_err; 
	if(uboot_run_cmd("erase %lx +%lx", start, size)<0) goto run_cmd_err;  
	if(uboot_run_cmd("cp.b %x %lx %lx", data, start, size)<0) goto run_cmd_err;
	return 1; 
run_cmd_err:
	printf("run_cmd error!!\n"); 
	goto enderr;
magic_err:
	printf("magic error!!\n"); 
	goto enderr;
hcrc_err:
	printf("hcrc error!!\n"); 
	goto enderr;
dcrc_err:
	printf("dcrc error!!\n"); 
enderr:
	printf("#############################################################\n");
	return 0; 
}

It doesn't seem like this is doing anything to adjust boot argument...

However, the mtd partition layout from the source code for this board (board955x) seems to be the following:

boot\u-boot\include\configs\board955x.h:178:# define MTDPARTS_DEFAULT "mtdparts=ath-nor0:256k(u-boot),64k(u-boot-env),14464k(rootfs),1536k(uImage),64k(ART),16320k@0x0(all)"

Which is curiously different from the lede one:

f9k1115v2_mtdlayout=mtdparts=spi0.0:256k(u-boot)ro,64k(u-boot-env),14464k(rootfs),1408k(kernel),64k(nvram)ro,64k(envram)ro,64k(art)ro,15872k@0x50000(firmware)

I am not sure what to make out of this - is it possible to get rid of nvram and envram and merge them into kernel partition?

On another note, in this commit:
https://github.com/lede-project/source/commit/f7a6fd31539be54d14d7c52b491b40b26bf8f740

Hauke Mehrtens mentioned

When CONFIG_KERNEL_KALLSYMS is not set it could be that the kernel will
fit onto the board again, this is the case for release images.

Do you know how much it shrinks the kernel? Are there other known safe ways to shrink the kernel - in which case I can happily compile custom builds for this router (and hopefully it is also generally applicable to other devices with similar size of kernel partition)

--edit--It might be moved to the tiny target. I don't know what that throws away other than saving some kernel memory. I'd guess LuCI and opkg because when I did a build for a 4M router it was that or internationalization and ipv6.--edit--

I was thinking of getting one of these since they're so cheap because the stock FW is such garbage. It's sounding like a lot of work though...

Per the kconfig file, dropping the symbols might save 300kB. I don't know if that's before or after compression, or just ram after loading, though. After that, your best bet is to make sure everything is built as a module, except for what's required to load the squashfs. But the kernel is going to keep growing. Eventually you have to start doing stuff like this reducing utility.

Nvram is the bootloader configuration. You can't get rid of it; you only might be able to move it if you build a custom uboot and copy the previous data. This isn't worth it.

Like others said, you might be able to change the nvram to rearrange the partitions, but in my experience that's often compiled directly into the bootloader. Even if you could do this, you would have to figure out how to flash a new image with a different location while the kernel thinks the partitions are in the old spots and it'd be difficult to do it the right way where you put the kernel start where the rootfs is so it's easy to change later. You might have to then tftp boot (if the bootloader supports it) an image with a ramdisk root so you can get the system up with the new partitions to flash the real image. If the bootloader has fancier recovery, you might be able to use that, but it might not understand the new layout.

If the partition locations aren't nvram variables, then you need to build a new bootloader with the new layout and use its recovery to get the image in.

I wouldn't touch any of the partitioning without serial and verifying that the bootloader has some sort of recovery. If you need a new bootloader, I'd make sure I can read/write the flash from an external device and get a backup beforehand.

I already solved the issue for myself - by compiling a custom firmware disabling kernel debug symbol.

Getting rid of luci opkg wouldn't help with the kernel image size. It would reduce the overall firmware size but kernel image remains the same and it's on a separate partition anyways.

There's a PR moving f9k1115v2 to tiny target but it may take a while (or never) getting merged until perhaps when we get close to next official release.
https://github.com/openwrt/openwrt/pull/784

1 Like

I know getting rid of those won't help the kernel issue here, but the tiny target was originally setup for 4MB flash devices, not devices with plenty of memory but a crappy flash layout. So I'm theorizing what else they're getting rid of on that target to include things that wouldn't help this router, but still be good for their original goal.