Old Kernel after sysupgrade (ath79 unifi-ac_mesh)

I used sysupgrade -n to flash an unifi-ac_mesh from an old LEDE based firmware to an OpenWRT 19.07 based build, but the kernel is not getting upgraded, causing several issues.

uname -a
Linux ap-openwisp-2 4.4.61 #0 Mon May 15 20:43:33 2017 mips GNU/Linux

While looking in the forum, I found this thread: Upgrading fully working Unifi AC Mesh from 18.06.1 to 18.06.2 - kernel stays at 4.9.120 / no WiFi.

Since I do not have physical access to this device, before trying solutions which could harm the device, I would like to ask if anyone can clarify what is happening exactly wit this hardware model?
It's still not clear to me why everything is getting upgraded except the kernel.

PS: found also FS#662 - Sysupgrade messes up Ubiquiti Unifi AP AC Lite.

PPS:

cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 001a0000 00010000 "kernel"
mtd4: 005f0000 00010000 "rootfs"
mtd5: 00150000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "EEPROM"
mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /tmp/root type tmpfs (rw,noatime,mode=755)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
/dev/mtdblock5 on /overlay type jffs2 (rw,noatime)
overlayfs:/overlay on / type overlay (rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.8M      4.8M         0 100% /rom
tmpfs                    61.2M    108.0K     61.1M   0% /tmp
tmpfs                    61.2M     72.0K     61.1M   0% /tmp/root
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock5            1.3M    268.0K      1.1M  20% /overlay
overlayfs:/overlay        1.3M    268.0K      1.1M  20% /

dmesg output:

[    0.000000] Linux version 4.4.61 (OpenWRT@Builder) (gcc version 5.4.0 (LEDE GCC 5.4.0 r4140-43a7a1a) ) #0 Mon May 15 20:43:33 2017
[    0.000000] MyLoader: sysp=77191367, boardp=312332ff, parts=1362b1de
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019750 (MIPS 74Kc)
[    0.000000] SoC: Qualcomm Atheros QCA956X ver 1 rev 0
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 08000000 @ 00000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] No valid device tree found, continuing without
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] On node 0 totalpages: 32768
[    0.000000] free_area_init_node: node 0, pgdat 804163f0, node_mem_map 81000000
[    0.000000]   Normal zone: 256 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32768 pages, LIFO batch:7
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
[    0.000000] Kernel command line:  board=UBNT-UF-AC-LITE mtdparts=spi0.0:384k(u-boot)ro,64k(u-boot-env)ro,7744k(firmware),7744k(ubnt-airos)ro,128k(bs)ro,256k(cfg)ro,64k(EEPROM)ro console=ttyS0,115200 rootfstype=squashfs,jffs2 noinitrd
[    0.000000] PID hash table entries: 512 (order: -1, 2048 bytes)
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] Memory: 125076K/131072K available (3008K kernel code, 160K rwdata, 756K rodata, 292K init, 200K bss, 5996K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS:51
[    0.000000] Clocks: CPU:775.000MHz, DDR:650.000MHz, AHB:258.333MHz, Ref:25.000MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 4932285024 ns
[    0.000007] sched_clock: 32 bits at 387MHz, resolution 2ns, wraps every 5541893118ns
[    0.008198] Calibrating delay loop... 385.84 BogoMIPS (lpj=1929216)
[    0.071008] pid_max: default: 32768 minimum: 301
[    0.075966] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.082939] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.092504] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.102918] futex hash table entries: 256 (order: -1, 3072 bytes)
[    0.109893] NET: Registered protocol family 16
[    0.115826] MIPS: machine is Ubiquiti UniFi-AC-LITE/MESH
[    0.339242] registering PCI controller with io_map_base unset
[    0.353372] PCI host bridge to bus 0000:00
[    0.357739] pci_bus 0000:00: root bus resource [mem 0x12000000-0x13ffffff]
[    0.364978] pci_bus 0000:00: root bus resource [io  0x0001]
[    0.370854] pci_bus 0000:00: root bus resource [??? 0x00000000 flags 0x0]
[    0.378000] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    0.386387] pci 0000:00:00.0: [168c:003c] type 00 class 0x028000
[    0.386414] pci 0000:00:00.0: invalid calibration data
[    0.391842] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[    0.391893] pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    0.391950] pci 0000:00:00.0: supports D1
[    0.391962] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    0.392148] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[    0.392176] pci 0000:00:00.0: BAR 0: assigned [mem 0x12000000-0x121fffff 64bit]
[    0.399921] pci 0000:00:00.0: BAR 6: assigned [mem 0x12200000-0x1220ffff pref]
[    0.407535] pci 0000:00:00.0: using irq 40 for pin 1
[    0.413430] clocksource: Switched to clocksource MIPS
[    0.419841] NET: Registered protocol family 2
[    0.425109] TCP established hash table entries: 1024 (order: 0, 4096 bytes)
[    0.432462] TCP bind hash table entries: 1024 (order: 0, 4096 bytes)
[    0.439202] TCP: Hash tables configured (established 1024 bind 1024)
[    0.445983] UDP hash table entries: 256 (order: 0, 4096 bytes)
[    0.452154] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[    0.459032] NET: Registered protocol family 1
[    0.463722] PCI: CLS 0 bytes, default 32
[    0.467511] Crashlog allocated RAM at address 0x3f00000
[    0.484427] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.490582] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.503024] io scheduler noop registered
[    0.507201] io scheduler deadline registered (default)
[    0.512857] Serial: 8250/16550 driver, 1 ports, IRQ sharing disabled
[    0.519921] console [ttyS0] disabled
[    0.543767] serial8250.0: ttyS0 at MMIO 0x18020000 (irq = 11, base_baud = 1562500) is a 16550A
[    0.552858] console [ttyS0] enabled
[    0.560412] bootconsole [early0] disabled
[    0.571650] m25p80 spi0.0: mx25l12805d (16384 Kbytes)
[    0.576943] 7 cmdlinepart partitions found on MTD device spi0.0
[    0.583051] Creating 7 MTD partitions on "spi0.0":
[    0.588017] 0x000000000000-0x000000060000 : "u-boot"
[    0.595094] 0x000000060000-0x000000070000 : "u-boot-env"
[    0.601931] 0x000000070000-0x000000800000 : "firmware"
[    0.621312] 2 uimage-fw partitions found on MTD device firmware
[    0.627475] 0x000000070000-0x000000210000 : "kernel"
[    0.633745] 0x000000210000-0x000000800000 : "rootfs"
[    0.640163] mtd: device 4 (rootfs) set to be root filesystem
[    0.646090] 1 squashfs-split partitions found on MTD device rootfs
[    0.652472] 0x0000006b0000-0x000000800000 : "rootfs_data"
[    0.659403] 0x000000800000-0x000000f90000 : "ubnt-airos"
[    0.666255] 0x000000f90000-0x000000fb0000 : "bs"
[    0.672315] 0x000000fb0000-0x000000ff0000 : "cfg"
[    0.678567] 0x000000ff0000-0x000001000000 : "EEPROM"
[    0.691774] libphy: ag71xx_mdio: probed
[    1.364970] ag71xx ag71xx.0: connected to PHY at ag71xx-mdio.0:04 [uid=004dd074, driver=Atheros 8031/8033 ethernet]
[    1.376323] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode:SGMII
[    1.384207] NET: Registered protocol family 10
[    1.391890] NET: Registered protocol family 17
[    1.396604] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    1.409730] 8021q: 802.1Q VLAN Support v1.8
[    1.420251] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.429174] Freeing unused kernel memory: 292K (80437000 - 80480000)
[    2.437252] init: Console is alive
[    2.440943] init: - watchdog -
[    2.454859] init: - preinit -
[    3.533493] random: jshn: uninitialized urandom read (4 bytes read, 5 bits of entropy available)
[    3.782803] random: jshn: uninitialized urandom read (4 bytes read, 5 bits of entropy available)
[    3.820786] random: jshn: uninitialized urandom read (4 bytes read, 5 bits of entropy available)
[    3.869054] random: jshn: uninitialized urandom read (4 bytes read, 6 bits of entropy available)
[    3.896318] random: jshn: uninitialized urandom read (4 bytes read, 6 bits of entropy available)
[    3.945099] random: jshn: uninitialized urandom read (4 bytes read, 6 bits of entropy available)
[    3.980476] random: jshn: uninitialized urandom read (4 bytes read, 6 bits of entropy available)
[    4.398019] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[    7.415726] eth0: link up (1000Mbps/Full duplex)
[    7.420522] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.639781] mount_root: jffs2 not ready yet, using temporary tmpfs overlay
[    7.658729] urandom-seed: Seed file not found (/etc/urandom.seed)
[    7.729992] eth0: link down
[    7.742173] procd: - early -
[    7.745809] procd: - watchdog -
[    8.348019] procd: - watchdog -
[    8.351514] procd: - ubus -
[    8.418474] random: ubusd: uninitialized urandom read (4 bytes read, 13 bits of entropy available)
[    8.504399] random: ubusd: uninitialized urandom read (4 bytes read, 13 bits of entropy available)
[    8.514112] random: ubusd: uninitialized urandom read (4 bytes read, 13 bits of entropy available)
[    8.524252] procd: - init -
[    9.318124] urngd: v1.0.2 started.
[   28.631826] device eth0 entered promiscuous mode
[   28.745433] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   31.535712] eth0: link up (1000Mbps/Full duplex)
[   31.540522] br-lan: port 1(eth0) entered forwarding state
[   31.546166] br-lan: port 1(eth0) entered forwarding state
[   31.553099] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   33.543434] br-lan: port 1(eth0) entered forwarding state
[   40.673454] random: nonblocking pool is initialized
[   45.928934] jffs2_scan_eraseblock(): End of filesystem marker found at 0x0
[   45.937569] jffs2_build_filesystem(): unlocking the mtd device... done.
[   45.944466] jffs2_build_filesystem(): erasing all blocks after the end marker... done.
[   51.824788] jffs2: notice: (1306) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   88.981986] IPv6: ADDRCONF(NETDEV_UP): br-msh: link is not ready

Yes, your problem does sound like a symptom of the problems described on those links.

And the fix is pretty simple: Just write a single 0 byte to the start of the the "bs" partition, like the solution on the first link says.

If you want to verify that this is indeed the problem, then you can do that by dumping the contents of /dev/mtd7`

It should look like this example from one of my Unifi AP AC Pros:

root@unifiac2:~# hexdump -C /dev/mtd7
00000000  00 00 00 00 a3 4d e8 2b  00 00 00 00 00 00 00 00  |.....M.+........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00020000

If your first byte is 01 instead of 00 then you have the same problem as described in those links, and the solution should work:

dd if=/dev/zero bs=1 count=1 of=/dev/mtd7
1 Like

Thanks for your help @bmork!
The bs partition does indeed look similar if not identical:

00000000  80 00 00 00 a3 4d e8 2b  00 00 00 00 00 00 00 00  |.....M.+........|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*

However, writing to that partition doesn't seem to be possible:

dd if=/dev/zero bs=1 count=1 of=/dev/mtd7
dd: can't open '/dev/mtd7': Permission denied

Looks like is mounting as read only:

[ 0.000000] Kernel command line: board=UBNT-UF-AC-LITE mtdparts=spi0.0:384k(u-boot)ro,64k(u-boot-env)ro,7744k(firmware),7744k(ubnt-airos)ro,128k(bs)ro,256k(cfg)ro,64k(EEPROM)ro console=ttyS0,115200 rootfstype=squashfs,jffs2 noinitrd

How can I make it writable?

PS:

fw_printenv
Cannot parse config file '/etc/fw_env.config': No such file or directory

Right. I forgot about that annoying thing, making this problem even harder to fix for no good reason.

Install the kmod-mtd-rw package. It will make all your partitions writable. Remove it when you're finished.

I'll try as soon as I get access to the device again and will update this thread. Thanks.

I could not install the kmod-mtd-rw package because the system says a specific kernel is required, however, for some absurd reason I don't know, dd if=/dev/zero bs=1 count=1 of=/dev/mtd7 this time worked out and we were able to solve the issue.

In this service we have other devices which most probably are in the same situation, we'll try again with another model and report here so hopefully other people going through the same issue can find more information.

Thanks a lot for your help @bmork.

I think it would be great if Someone(tm) with enough interest in this took the time to add some cleanup scripts to the OpenWrt sysupgrade for these devices. Fixing all the buggy installation instructions all over the net is impossible by now. But sysupgrade could

  • verify that "bs" points to the correct partition, and
  • try to fix it if not, and/or
  • write a new kernel to both partitions if possible

But this will most likely involve overriding the read-only flag on these partitions. Which makes it kind of hard to do automatically. Maybe better if sysupgrade just bailed out early and pointed to instructions on how to fix the problem?

We should also remove the completely unnecessary and harmful "read-only" attribute on the "bs" partition. But that's not gonna help anyone with this issue since it's a catch22. You can't update the booting kernel (and DT) without writing to either "bs" or the second firmware partition.

1 Like

Testing on another device, I do not have the same luck and dd if=/dev/zero bs=1 count=1 of=/dev/mtd7 is failing with: dd: can't open '/dev/mtd7': Permission denied and kmod-mtd-rw fails because it says it needs kernel 4.14:

opkg install kmod-mtd-rw
Installing kmod-mtd-rw (4.14.229+git-20160214-1) to root...
Downloading http://downloads.openwrt.org/releases/19.07-SNAPSHOT/targets/ath79/generic/packages/kmod-mtd-rw_4.14.229%2bgit-20160214-1_mips_24kc.ipk
Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-mtd-rw:
 * 	kernel (= 4.14.229-1-b84a5a29b1d5ae1dc33ccf9ba292ca1d)
 * opkg_install_cmd: Cannot install package kmod-mtd-rw.

What is an alternative solution I can use?

Should I mtd another partition with the firmware image?

Or can I install an older kmod-mtd-rw compiled for another version of OpenWRT which used this kernel (4.4.14)?

Maybe try to download and unpack manually to avoid the dependency mismatch? Select a version as close as possible to your currently running kernel (which is different from what opkg will tell you since that is part of the problem) and do something like the untested

cd /tmp/
wget http://downloads.openwrt.org/releases/19.07-SNAPSHOT/targets/ath79/generic/packages/kmod-mtd-rw_4.14.229%2bgit-20160214-1_mips_24kc.ipk
tar zxvf kmod-mtd*
tar zxvf data.tar.gz
insmod lib/modules/*/mtd-rw.ko i_want_a_brick=1 

Maybe it works, maybe not...

What about using mtd to flash the other partition from which the old kernel is booted?
Wouldn't that work?

I believe that partition is read-only too. Otherwise it would have been a solution.

Do you know which partition is the other one exactly?

I believe it is mtd6, but you should verify this before attempting to write to it. From one of my uap ac pros:

root@unifiac:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 00200000 00010000 "kernel"
mtd4: 00590000 00010000 "rootfs"
mtd5: 00190000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "kernel1"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "art"

I have this:

cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 001a0000 00010000 "kernel"
mtd4: 005f0000 00010000 "rootfs"
mtd5: 00150000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "EEPROM"

It could be ubnt-airos but how do I know which one is exactly? Any hint on this?

Yes, it is. You can see that it has the same size as your firmware partition

On some systems mtd is not available and in that case the only solution we have found is to rollback to stock firmware and then use mtd to flash both partitions.

Is there any solution to this? I bought a Pi 3b+, installed 23.05.1 and all was ok. Now I upgraded to 23.05.2 while keeping configurations, and I have the same issue of old kernel. Luci says I have new kernel "5.15.137", but when I execute opkg list-installed kernel I get "kernel - 5.15.134-1-2f29f5d52148751b163e722c9eaead1e". I cannot install any kmod package.

How to get out of this?

Edit: I went back to original firmware, and now everything works. But I guess there is an issue updating the kernel when doing upgrade via luci for some devices (squashfs).