[SOLVED] Ubnt Unifi AC Pro partition problems - "kernel0" vs "kernel1"

EDIT2:

And then I noticed the "kmod-mtd-rw" package - all problems have been solved before. I should know that by now...

EDIT:

Describing the problem often helps. I figured out a solution after finding this hint:
http://electronics.stackexchange.com/questions/67955/arm-linux-and-uboot-can-i-make-a-read-only-mtd-writable

Bulit this module for the old kernel:

#include <linux/module.h>
#include <linux/mtd/mtd.h>

static char *part;
module_param(part, charp, 0400);
MODULE_PARM_DESC(part, "MTD partition to change");


/*
 * Copyright 2017  Bjørn Mork <bjorn@mork.no>
 *
 * stole the idea from
 * http://electronics.stackexchange.com/questions/67955/arm-linux-and-uboot-can-i-make-a-read-only-mtd-writable
 */

static int __init mtd_force_rw_init(void)
{
	struct mtd_info *mtd;

        printk(KERN_ALERT "Hello World!\n");
	if (!part)
		return -ENODEV;

        mtd = get_mtd_device_nm(part);
        if (IS_ERR(mtd))
                return -ENODEV;

	printk(KERN_ALERT "mtd part '%s' flags: %#x\n", part, mtd->flags);
	mtd->flags |= MTD_WRITEABLE;
	put_mtd_device(mtd);
        return 0;
}

static void __exit  mtd_force_rw_exit(void)
{
        printk(KERN_ALERT "Goodbye cruel world!\n");
}

module_init(mtd_force_rw_init);
module_exit(mtd_force_rw_exit);

MODULE_DESCRIPTION("MTD read-only fix");
MODULE_LICENSE("GPL");

I initially intended to make the "ubnt-airos" partition writable, but looking at a few boot logs I figured that the Ubnt boot loader image selection magic must be stored in the "bs" partition. And it looks like a simple 4 byte number and 4 byte magic:

00000000  80 00 00 00 a3 4d e8 2b  00 00 00 00 00 00 00 00  |.....M.+........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00020000

Changing this to

00000000  00 00 00 00 a3 4d e8 2b  00 00 00 00 00 00 00 00  |.....M.+........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00020000

made the bootloader load "kernel0" instead, matching the LEDE expectations. So after rerunning sysupgrade, I finally got LEDE v17.01 running.

I guess this info should be stream lined and put into the wiki....

ORIGINAL PROBLEM:

Hello,

I am beginning to realize that I have painted myself info a corner... Quite embarassing.

I installed a LEDE snapshot on a Unifi AC Pro in November, using the procedure documented on https://wiki.openwrt.org/toh/ubiquiti/unifiac . The only issue was that it still booted into the original firmware after ther first attempt to write LEDE to the "kernel0" partition. So I went on and wrote LEDE to the "kernel1" partition too, as instructed. Rebooted and got LEDE running. Everything seemed fine and I was happy.

Until today, when I wanted to test out the v17.01 release. So I sysupgraded, rebooted and got... the old kernel and a new root file system.

Looked a bit around and finally understood what has happend: The boot loader is booting the second kernel partion ("kernel1" in the original firmware or "ubnt-airos" in LEDE). This is still the kernel I flashed in November. The kernel finds the root file system inside the "kernel0" partition. Which worked fine as long as both "kernel0" and "kernel1" contained the same LEDE firmware.

root@unifiac:~# cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 00150000 00010000 "kernel"
mtd4: 00640000 00010000 "rootfs"
mtd5: 00410000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "EEPROM"
root@unifiac:~# file -s /dev/mtd6ro 
/dev/mtd6ro: u-boot legacy uImage, MIPS LEDE Linux-4.4.30, Linux/MIPS, OS Kernel Image (lzma), 1327599 bytes, Wed Nov  9 11:17:52 2016, Load Address: 0x80060000, Entry Point: 0x80060000, Header CRC: 0xF302D0BE, Data CRC: 0x2788620E
root@unifiac:~# cat /proc/cmdline 
 board=UBNT-UF-AC-PRO mtdparts=spi0.0:384k(u-boot)ro,64k(u-boot-env)ro,7744k(firmware),7744k(ubnt-airos)ro,128k(bs)ro,256k(cfg)ro,64k(EEPROM)ro console=ttyS0,115200 rootfstype=squashfs,jffs2 noinitrd
root@unifiac:~# cat /proc/mounts 
/dev/root /rom squashfs ro,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,noatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,noatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev,noatime 0 0
/dev/mtdblock5 /overlay jffs2 rw,noatime 0 0
overlayfs:/overlay / overlay rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work 0 0
tmpfs /dev tmpfs rw,nosuid,relatime,size=512k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,mode=600 0 0
debugfs /sys/kernel/debug debugfs rw,noatime 0 0

The problem is that there is no way to change the kernel inside the "ubnt-airos" partition, since it is made read-only by the embedded command line (why???). And I cannot find any way to make the boot loader boot from the "firmware" partition. I have no serial console. To make everything even worse, this access point is mounted in such a way that it is very hard to even get to the reset button. Yes, stupid.

But I am still hoping that there is some way to either make the Ubnt boot loader select the first partition, or - failing that - some way to rewrite the "ubnt-airos" partition without having to press the button.

Any hints? Or should I just bite the lemon and go mount a serial console? That will be difficult, given the stupid mounting...

For now I have reverted to the old snapshot, so that file system and kernel is matching. Which works OK. But I'd really like to have some way to upgrade in the future.

Hi! I have the same problem but i'm a beginner and i can't understand all your post: how can I write to the "bs" partition and boot the "kernel0" partition (installing "kmod-mtd-rw" package is enough to make it writable?) ? which program do you use to write hex to "bs" partiton? Thanks!!

EDIT: Ok i manage somehow to follow your fantastic post (install "xxd" and "kmod-mtd-rw" and using vi if someone like me with no experience....) Many, many, many thanks for your work that saved my unifi ac lite!!!!

See https://github.com/jclehner/mtd-rw for instructions on how to use the "kmod-mtd-rw" package. Note that this will enable you to brick your AP, since it makes the boot loader partitions writable. Be careful not to write to the wrong partition!

As for editing the "bs" partition, I didn't bother to make it more complicated than necessary. It's a very small partition, so I just made a copy of it, transferred the file to a PC, hexedited it there keeping a copy of the original, and copied the modified file back, before writing it with mtd.

Something like:

# dd if=/dev/mtd7ro of=/tmp/bs.orig bs=64k
# scp /tmp/bs.orig bjorn@mypc:
mypc:~$ cp bs.orig bs.modified
mypc:~$ hexedit bs.modified
# scp bjorn@mypc:bs.modified /tmp/
# mtd -e bs write /tmp/bs.modified bs

I also used hexdump along the way to visually verify the files.and partition contents. Note that AFAIK, only the first byte can/should be changed. Mine was 0x80 and I changed it to 0x00. The second 32bit block is a magic number. Googling for boot logs showed the Ubnt booloader refer to this exact magic number during partition selection. I assume this means that it will do something sane if the magic doesn't match, but better not experiment with that....

There are obviously many other variants that would work here. But no matter how you do it, I recommend keeping a copy of the original "bs" partition contents on some external storage. This makes it more likely that you can fix any errors

No guarantees given. I've only tried this once, and I haven't seen Ubnt U-Boot code.

Let us know how things goes.

I think I figured out how I ended up with the second firmware partition ("kernel1", aka "ubnt-airos") being default in case anyone is interested. My UAP AC PRO came with v3.4.14 firmware by default, having the u-boot modification preventing OpenWrt/LEDE installation. So I downgraded to v3.4.7 before installing LEDE. Looking over the changes made by the downgrade (luckily I saved a copy of all partitions before and after this downgrade), I see that it changed only 3 of the partitions.

These were the changes during the v3.4.14 => v3.4.7 downgrade:

mtd0 "u-boot" - got the downgraded u-boot I wanted
mtd3 "kernel1" - got the v3.4.7 firmware
mtd4 "bs" - changed the first byte from 0x00 to 0x80

Note in particular that partition mtd2 "kernel0" was unmodified by the firmware downgrade, keeping the v3.4.14 firmware intact! So it appears that the Ubnt firmware downgrade process simply updated the inactive firmware partition and switched active partiions by modifying "bs". I believe all this adds up pretty nicely.

I guess the initial LEDE installation instructions for the UAP AC PRO should include a part making sure "bs" points to "kernel0" before proceding. I wonder if the original firmware comes with a tool to read and modify "bs"? I guess it probably does?