Ubiquiti UniFi AP AC PRO - readonly system partition?

Hi, I'm building myself several fw from latest code, yet for AC Pro I get images, which seems to be "read only" partition - no chances get saved. I'm attaching config I'm using. Do you see there anything problematic please?

Glad for any help.

https://pastebin.com/1fDLjkPi

Does it happen with a bare minimum .config? ( no ./files or other fancy stuff! )

Posting a 5332 line .config file is not so exciting for those who may help you;

  • Give an overview of what modifications you've done
  • Provide "df; mount; free; dmesg | grep -C5 'mount'; logread" type diagnostics
1 Like

Thank you for reply. It doesn't happen at least with auto build snapshots so indeed there is something wrong with config, I pasted. And by flashing latest fw I can revert this state, so this is where I'm currently, I'm fool I didn't check the mount itself, sorry for that.
.
For the diagnostic you asked to provide and I fully understand reasons behind, it would mean for me to flash it back to broken fw, with necessity for me to take out AP from the Attic, bring it down to the router, connect, set laptop to 192.168.x.x ... and then provide the logs and then back flashing to latest fw, and move back up...simply kind of pain. So that's why I was hoping someone might see directly something wrong with the config itself, things which are apparently wrong with the config generated by the "make menuconfig" and custom modifications. Experienced person would for sure spot error immediately, at least that was my assumption.

I'm not using ./files additional configs, at least not for this case.

Striped .config if that would help : https://pastebin.com/L22PyQve

Side note / sorry not expert here : is it possible to have on 10.x.x.x local subnet reachable as well 192.16.x.x. (eg. for fw recoveries?) ? Would that work to have my main router configured two IPs on LAN for both subnet ? So I can avoid "exercise" described above ? (I have Edgerouter in which I have plugged in via LAN two APs)

If I will get time again, I will try to provide the diag you asked for.

1 Like

A very convenient way: execute "scripts/diffconfig.sh" in your build-dir to only display your configuration on top of base config.

3 Likes

I remember I got a similar issue on AC PRO as I was not aware it has a dual boot system designed for stock firmware. Maybe you should read the guidlines on how to force the bootloader to only boot kernel0 and forget about kernel1.

I read those guidelines, yet it never happen with snapshot upgrade and it always happen when I upgrade Openwrt with my own compiled fw. However there is note from March 2020 : This did not work anymore (March 2020) so not sure what exactly shouldn't be working.

My flash layout on Unify AC PRO with Openwrt currently looks like this :

dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 001e0000 00010000 "kernel"
mtd4: 005b0000 00010000 "rootfs"
mtd5: 00330000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "art"

So I shall really now AGAIN write a nullbyte to the bs partition ?

Thanks for reminding me about diffconfig, that's what I'm actually using after updating the tree.

Diffconfig results : wrong (another config C7v2) -> https://pastebin.com/ZdE75rFP
Correct diffconfig results : https://pastebin.com/HJRqtt3d

1 Like

If you've been following the directions you should have erased the ubnt-airos partition to be sure that nothing boots from there.

Then look at the bootlog where the file systems are mounted and runtime commands like df and mount.

If you have not reverted to stock fw in between the nullbyte in bs partion should never change.
I have different flash layout on my AC PRO devices (home build - OpenWrt SNAPSHOT, r12719-84f4a783c6):

dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 001c0000 00010000 "kernel"
mtd4: 005d0000 00010000 "rootfs"
mtd5: 00210000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "art"

My diffconfig shows following:

CONFIG_TARGET_ath79=y
CONFIG_TARGET_ath79_generic=y
CONFIG_TARGET_ath79_generic_DEVICE_ubnt_unifiac-pro=y
CONFIG_OPENSSL_ENGINE=y
CONFIG_OPENSSL_PREFER_CHACHA_OVER_GCM=y
CONFIG_OPENSSL_WITH_ASM=y
CONFIG_OPENSSL_WITH_CHACHA_POLY1305=y
CONFIG_OPENSSL_WITH_CMS=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_ERROR_MESSAGES=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_WITH_SRP=y
CONFIG_OPENSSL_WITH_TLS13=y
# CONFIG_PACKAGE_ath10k-firmware-qca988x-ct is not set
CONFIG_PACKAGE_ath10k-firmware-qca988x-ct-htt=y
CONFIG_PACKAGE_cgi-io=y
CONFIG_PACKAGE_hostapd=y
CONFIG_PACKAGE_kmod-ledtrig-default-on=y
CONFIG_PACKAGE_kmod-ledtrig-heartbeat=y
CONFIG_PACKAGE_kmod-ledtrig-netdev=y
CONFIG_PACKAGE_kmod-ledtrig-timer=y
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblucihttp=y
CONFIG_PACKAGE_liblucihttp-lua=y
CONFIG_PACKAGE_libmbedtls=y
CONFIG_PACKAGE_libopenssl=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libustream-mbedtls=y
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_luci=y
CONFIG_PACKAGE_luci-app-firewall=y
CONFIG_PACKAGE_luci-app-opkg=y
CONFIG_PACKAGE_luci-base=y
CONFIG_PACKAGE_luci-lib-ip=y
CONFIG_PACKAGE_luci-lib-jsonc=y
CONFIG_PACKAGE_luci-lib-nixio=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-mod-network=y
CONFIG_PACKAGE_luci-mod-status=y
CONFIG_PACKAGE_luci-mod-system=y
CONFIG_PACKAGE_luci-proto-ipv6=y
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_luci-ssl=y
CONFIG_PACKAGE_luci-theme-bootstrap=y
CONFIG_PACKAGE_muninlite=y
CONFIG_PACKAGE_px5g-mbedtls=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-luci=y
CONFIG_PACKAGE_rpcd-mod-rrdns=y
CONFIG_PACKAGE_uhttpd=y
CONFIG_PACKAGE_uhttpd-mod-ubus=y
# CONFIG_PACKAGE_wpad-basic is not set
CONFIG_PACKAGE_xinetd=y
# CONFIG_TARGET_ROOTFS_INITRAMFS is not set

In your diffconfig I see line 3:

CONFIG_TARGET_ath79_generic_DEVICE_tplink_archer-c7-v2=y

This might be the culprit.

1 Like

I must apologies because I might have mixed provided config with C7 v2 I'm using as well (with kids on my backs, it's not so easy :slight_smile: ) . But thank you for providing your config, I will compare it with the correct one I'm using for UC Pro, I appreciate your help.

So I really mixed up the configs, here is the correct one - https://pastebin.com/HJRqtt3d and indeed I don't see anything important that would differs.
So the culprit? could be different flash layout but then I don't get why the custom build fw is the only problem.
When I get the time, I'm going to flash new fw that I just compiled and let's see. Thank you for your details.

If anyone can comment on different flash layout, if that could be indeed a problem, I would appreciate.

Maybe one additional information - prior to switching to the Openwrt, I had installed the most latest version on vendor fw, which I had to downgrade to "supported" version. This could potentially result into different flash layout.

Mine

dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
--> mtd3: 001e0000 00010000 "kernel"
--> mtd4: 005b0000 00010000 "rootfs"
--> mtd5: 00330000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "art"

Yours

dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
--> mtd3: 001c0000 00010000 "kernel"
--> mtd4: 005d0000 00010000 "rootfs"
--> mtd5: 00210000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "art"

I run recent snapshot versions (kernel 4.19), maybe you run on stable branch (kernel 4.14)? That could explain the difference, no need to worry. I remember that increasing the size of kernel and rootfs was necessary in order to fit the growing size as I found in this commit.
Usually I check my build images using the binwalk tool. My latest build does show a problem when I extract (-e) that build (OpenWrt SNAPSHOT r12834-d24e43a7f6):

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             uImage header, header size: 64 bytes, header CRC: 0x9F825224, created: 2020-04-04 15:12:43, image size: 1810329 bytes, Data Address: 0x80060000, Entry Point: 0x80060000, data CRC: 0x21794ACC, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "MIPS OpenWrt Linux-4.19.108"
64            0x40            LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 5792374 bytes

WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -le -d 'squashfs-root-0' '%e'': [Errno 2] No such file or directory: 'sasquatch': 'sasquatch', 'sasquatch -p 1 -le -d 'squashfs-root-0' '%e'' might not be installed correctly

WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -be -d 'squashfs-root-0' '%e'': [Errno 2] No such file or directory: 'sasquatch': 'sasquatch', 'sasquatch -p 1 -be -d 'squashfs-root-0' '%e'' might not be installed correctly
1835008       0x1C0000        Squashfs filesystem, little endian, version 4.0, compression:xz, size: 3909474 bytes, 1243 inodes, blocksize: 262144 bytes, created: 2020-04-04 15:12:43

A good build has following result (OpenWrt SNAPSHOT r12719-84f4a783c6):

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             uImage header, header size: 64 bytes, header CRC: 0x2B56E68A, created: 2020-03-27 09:18:41, image size: 1810159 bytes, Data Address: 0x80060000, Entry Point: 0x80060000, data CRC: 0x9E78886F, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "MIPS OpenWrt Linux-4.19.108"
64            0x40            LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 5792374 bytes
1835008       0x1C0000        Squashfs filesystem, little endian, version 4.0, compression:xz, size: 3908698 bytes, 1243 inodes, blocksize: 262144 bytes, created: 2020-03-27 09:18:41

It might help you as well trying to find the culprit.

Side note / sorry not expert here : is it possible to have on 10.x.x.x local subnet reachable as well 192.16.x.x. (eg. for fw recoveries?) ? Would that work to have my main router configured two IPs on LAN for both subnet ? So I can avoid "exercise" described above ? (I have Edgerouter in which I have plugged in via LAN two APs)

Yes you can: usually I put a secondary IP on the interface of network facing to management of AP something like 192.168.1.55 (not .1 of course). Just log on to that router and telnet to 192.168.1.1 after having reverted to default config.

1 Like

Yeah, binwalk - I was reading about it already so many times, never installed it though. Thank you for reminding it to me ! :wink:

Today compiled one :

Scan Time:     2020-04-05 13:27:11
Target File:   /home/station/lede/lede_test/openwrt-ath79-generic-ubnt_unifiac-pro-squashfs-sysupgrade.bin
MD5 Checksum:  409185cbbc35edce71abe4dbd948ba77
Signatures:    404

DECIMAL       HEXADECIMAL     DESCRIPTION
--------------------------------------------------------------------------------
0             0x0             uImage header, header size: 64 bytes, header CRC: 0x1E9700F, created: 2020-04-03 15:59:56, image size: 1810135 bytes, Data Address: 0x80060000, Entry Point: 0x80060000, data CRC: 0xEC9796D8, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "MIPS OpenWrt Linux-4.19.108"
64            0x40            LZMA compressed data, properties: 0x6D, dictionary size: 8388608 bytes, uncompressed size: 5791566 bytes
1835008       0x1C0000        Squashfs filesystem, little endian, version 4.0, compression:xz, size: 5967698 bytes, 1693 inodes, blocksize: 262144 bytes, created: 2020-04-03 15:59:56

When I tested the old fw which failed in the past with binwalk...successful as well.
So - I tried flashing new fw again. Now to be sure once more writing null byte to bs partition prior to the flashing. After reboot,

BusyBox v1.31.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r3012+9786-0d1b329914
 -----------------------------------------------------
=== WARNING! =====================================
There is no root password defined on this device!
Use the "passwd" command to set up a new password
in order to prevent unauthorized SSH logins.
--------------------------------------------------

So posting diagnostic as suggested :

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.8M      5.8M         0 100% /rom
tmpfs                    60.4M    264.0K     60.1M   0% /tmp
tmpfs                    60.4M     84.0K     60.3M   0% /tmp/root
overlayfs:/tmp/root      60.4M     84.0K     60.3M   0% /
tmpfs                   512.0K         0    512.0K   0% /dev

root@OpenWrt:~# mount
/dev/root on /rom type squashfs (ro,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup on /sys/fs/cgroup type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,pids)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /tmp/root type tmpfs (rw,noatime,mode=755)
overlayfs:/tmp/root on / type overlay (rw,noatime,lowerdir=/,upperdir=/tmp/root/upper,workdir=/tmp/root/work)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
root@OpenWrt:~# free
              total        used        free      shared  buff/cache   available
Mem:         123668       16832       92252         348       14584       75776
Swap:         61436           0       61436

root@OpenWrt:~# dmesg | grep -C5 'mount'
[    6.136834] random: procd: uninitialized urandom read (4 bytes read)
[    7.135072] eth0: link up (1000Mbps/Full duplex)
[    7.139957] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    7.146775] IPv6: ADDRCONF(NETDEV_CHANGE): eth0.1: link becomes ready
[   10.291586] jffs2: Too few erase blocks (1)
[   10.296338] mount_root: failed to mount -t jffs2 /dev/mtdblock5 /tmp/overlay: Invalid argument
[   10.305527] mount_root: overlay filesystem has not been fully initialized yet
[   10.313369] mount_root: switching to jffs2 overlay
[   10.318505] mount_root: switching to jffs2 failed - fallback to ramoverlay
[   10.349086] urandom-seed: Seed file not found (/etc/urandom.seed)
[   10.429461] eth0: link down
[   10.450250] procd: - early -
[   10.453318] procd: - watchdog -
[   11.019819] procd: - watchdog -

root@OpenWrt:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00060000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00790000 00010000 "firmware"
mtd3: 001c0000 00010000 "kernel"
mtd4: 005d0000 00010000 "rootfs"
mtd5: 00010000 00010000 "rootfs_data"
mtd6: 00790000 00010000 "ubnt-airos"
mtd7: 00020000 00010000 "bs"
mtd8: 00040000 00010000 "cfg"
mtd9: 00010000 00010000 "art"

Now so that I have more time to check, what is going on, I found out actually the partitions are not read only, just all the settings are gone after reboot, based on logs no surprise, if it's "ramonly", right ?
But why ?

[   10.291586] jffs2: Too few erase blocks (1)
[   10.296338] mount_root: failed to mount -t jffs2 /dev/mtdblock5 /tmp/overlay: Invalid argument
[   10.305527] mount_root: overlay filesystem has not been fully initialized yet
[   10.313369] mount_root: switching to jffs2 overlay
[   10.318505] mount_root: switching to jffs2 failed - fallback to ramoverlay
1 Like

It looks like the problem here is simply the rootfs is too large. There is only one block left for rootfs-data. The jffs cannot be created so it fails back to a RAM overlay. I think that 3 or 5 blocks are required.

1 Like

Thank you. And shouldn't that be somehow "safe guarded" when building it using build system (https://openwrt.org/docs/guide-developer/build-system/use-buildsystem) ? Eg. when building low mem systems, it is for sure checking the maximum allowed size.

I can easily check by removing some of the non-critical packages to see if that helps.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.