Sysupgrade logs: where are they?

jpsollie1 · April 19, 2022, 4:40am

I'm experimenting with the master branch to deploy ath11k module to my bananapi r64.
but, and I do not know why, when I sysupgrade a self-compiled image, it doesn't work:
I can see this at the kernel version: the installed kernel 10.109 is different from the one I compiled in the image (10.110).
Any clue where sysupgrade stores its logs?

hnyman · April 19, 2022, 4:46am

It doesn't store them.

jpsollie1 · April 19, 2022, 5:44am

Too bad. Any suggestion how I can find out what's wrong?
I also built the emmc image, but I do not have a clue which offset I need to pass to dd

slh · April 19, 2022, 5:51am

Serial console access.

jpsollie1 · April 19, 2022, 6:51am

is there an easy workaround to simply write the sdcard.img to a mtd device?

cat /proc/mtd:
dev:    size   erasesize  name
mtd0: 00080000 00020000 "bl2"
mtd1: 00200000 00020000 "fip"
mtd2: 07d80000 00020000 "ubi"

...so I guess that should be mtd2 (but I may be wrong, no idea how UBI works)

blkid:
/dev/ubi0_1: UUID="7018771b-fff8-4245-8ef1-8be6dd43cfef" TYPE="ubifs"
/dev/mmcblk0p128: PARTUUID="5452574f-2211-4433-5566-778899aabb80"
/dev/mmcblk0p65: TYPE="squashfs"
/dev/mmcblk0p5: UUID="47407162-1fb2-4697-bb63-8d87439288bb" TYPE="swap" PARTLABEL="Linux swap" PARTUUID="74a7fdc5-0777-4a27-890d-f88c4a8bcb49"
/dev/mmcblk0p3: PARTLABEL="recovery" PARTUUID="5452574f-2211-4433-5566-778899aabb03"
/dev/mmcblk0p1: PARTLABEL="fip" PARTUUID="5452574f-2211-4433-5566-778899aabb01"
/dev/mmcblk0p66: LABEL="rootfs_data" UUID="e5ee2dcb-68ec-4ce6-adcc-0155806b790a" BLOCK_SIZE="4096" TYPE="f2fs"
/dev/mmcblk0p6: UUID="51bb4f57-7c0f-4559-99ee-7236e207ac5d" BLOCK_SIZE="4096" TYPE="f2fs" PARTLABEL="private user data" PARTUUID="4ba54f75-ff7d-4da5-a3eb-f51718a5f9e9"
/dev/mmcblk0p4: PARTLABEL="production" PARTUUID="5452574f-2211-4433-5566-778899aabb04"
/dev/mmcblk0p2: PARTLABEL="ubootenv" PARTUUID="5452574f-2211-4433-5566-778899aabb02"
/dev/mtdblock2: UUID="3074057516" TYPE="ubi"
/dev/ubiblock0_0p1: TYPE="squashfs"

Which block device do I need to write to?

lleachii · April 19, 2022, 3:18pm

Did the Wiki instructions say do this?

This can seriously brick a device.

jpsollie1 · April 19, 2022, 3:43pm

Apparently, it DOES brick the device:
The only way it is bootable right now is via tftp recovery, no idea why it always goes that way, but fw_setenv doesn't seem to be helpful

lleachii · April 19, 2022, 3:44pm

This is usually how you install OpenWrt - boot using TFTP then flash the sysupgrade file in the web GUI.

In most devices, you don't need to edit the bootloader environment.

daniel · April 19, 2022, 4:36pm

As you are using BananaPi R64 which is MediaTek MT7622 you have the luxury (or obstacle, depends on the perspective) of a recovery/production dual-boot scheme which also checks pstore/ramoops.
If your custom build happens to crash, the board will automatically reboot into the recovery image instead, so you can examine the reason for the crash (instead of being stuck in a boot loop).
Most likely this is what you are experiencing.

First of all, you can see if you have booted into recovery more using

ubus call system board

Only if you see "rootfs_type": "squashfs" the production system has been started. If you see "rootfs_type": "initramfs" it's the recovery system running instead.

To see what happened, you can check the logs in pstore, ie.

cat /sys/fs/pstore/*

should give you an idea why the crash has happened. In order to make the bootloader go back to boot into the regular production image, you have to clear the logs in pstore, ie.

rm /sys/fs/pstore/*

So you will not necessarily need a serial console for debugging -- this is why pstore/ramoops can be considered a luxury (if you know that it exists).

Let me know if you need further assistance -- I got the board sitting here on my desk and can help you figure out what's wrong with your build and/or why sysupgrade doesn't work.

It also looks like (from the /proc/mtd and blkid output) you have previously attempted to write things to the SPI-NAND in a way which will most likely not work. See the OpenWrt wiki for this board to learn how installation of OpenWrt to either the SPI-NAND or eMMC works:

jpsollie1 · April 19, 2022, 5:13pm

Unfortunately, I guess I f***ed up: I followed the wiki, but /sys/fs/pstore is empty and it still keeps rebooting into recovery:
Every time I boot it, it simply waits until the remote device 1.254 is online before it gets alive.
Tried cold reset, warm reset, flash_reset, ... but nothing worked.
I'm not sure why it doesn't work in the first place
EDIT: this may be a clue:

[    0.008770] pstore: Registered ramoops as persistent store backend
[    0.008784] ramoops: using 0x10000@0x42ff0000, ecc: 0

a ram device does not survive a reboot, does it?

hnyman · April 19, 2022, 5:23pm

the pstore kernel crash pstore survives a warm reboot, but not a cold boot (power-off).
See background info at https://www.kernel.org/doc/html/latest/admin-guide/ramoops.html

jpsollie1 · April 19, 2022, 6:30pm

unfortunately, I couldnt find anything useful up there.
So, I'm summarizing:

The device boots the recovery image from tftp, and works fine.
lsblk from recovery image shows:

/dev/mmcblk0p128: PARTUUID="5452574f-2211-4433-5566-778899aabb80"
/dev/mmcblk0p65: TYPE="squashfs"
/dev/mmcblk0p5: PTTYPE="PMBR" PARTLABEL="install" PARTUUID="5452574f-2211-4433-5566-778899aabb05"
/dev/mmcblk0p3: PARTLABEL="ubootenv" PARTUUID="5452574f-2211-4433-5566-778899aabb03"
/dev/mmcblk0p1: PARTLABEL="bl2" PARTUUID="5452574f-2211-4433-5566-778899aabb01"
/dev/mmcblk0p6: PARTLABEL="production" PARTUUID="5452574f-2211-4433-5566-778899aabb06"
/dev/mmcblk0p4: PARTLABEL="recovery" PARTUUID="5452574f-2211-4433-5566-778899aabb04"
/dev/mmcblk0p2: PARTLABEL="fip" PARTUUID="5452574f-2211-4433-5566-778899aabb02"
/dev/mmcblk1p1: UUID="2152f05b-789f-4b25-89c2-a35d4de4ac36" BLOCK_SIZE="4096" TYPE="f2fs"
/dev/sda: UUID="20ca3c36-5282-469a-af87-1cdb584f65a1" BLOCK_SIZE="4096" TYPE="ext4"

cat /proc/mtd:

root@OpenWrt:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00080000 00020000 "bl2"
mtd1: 00200000 00020000 "fip"
mtd2: 07d80000 00020000 "ubi"

boot switch is set @ nand -> emmc > sdcard
contents of 1.254 tftp server:

openwrt.bin                                                      
openwrt-mediatek-mt7622-bananapi_bpi-r64-sdcard.img.gz
openwrt-mediatek-mt7622-bananapi_bpi-r64-emmc-bl31-uboot.fip     
openwrt-mediatek-mt7622-bananapi_bpi-r64-snand-bl31-uboot.fip
openwrt-mediatek-mt7622-bananapi_bpi-r64-emmc-preloader.bin      
openwrt-mediatek-mt7622-bananapi_bpi-r64-snand-preloader.bin
openwrt-mediatek-mt7622-bananapi_bpi-r64-initramfs-recovery.itb  
openwrt-mediatek-mt7622-bananapi_bpi-r64-squashfs-sysupgrade.itb
openwrt-mediatek-mt7622-bananapi_bpi-r64.manifest                
openwrt-toolchain-mediatek-mt7622_gcc-11.2.0_musl.Linux-x86_64.tar

I tried:

 gzip -cd ...sdcard.gz > /dev/mmcblk0

but it doesn't work,
after reboot, it simply picks up the recovery server again, it doesn't boot otherwise.

Any clues which file I should force where in order to make it boot?

lleachii · April 19, 2022, 6:50pm

~~Wait, you tried something that could brick your device?~~

~~Why do you keep trying to raw write when I thought we covered that wasn't OK.~~

~~Have you tried what was suggested to you???~~

~~Why won't you try this installation method - instead of doing things known to brick devices?~~

You know - the device page give you clear instructions.

(If you don't understand, feel free to ask.)

daniel · April 19, 2022, 7:00pm

That cannot work because you are writing the image intended for the SD card to the internal eMMC (which is a similar, but yet different thing having hardware boot partitions which the SoC wants to use)

My recommendation:

While being booted into the recovery system, insert the SD card and do gzip -cd ...sdcard.gz > /dev/mmcblk1
Put the boot switch SW1 in position 1 to boot SD card.
Now remove the device form power and re-connect it (warm reboot is not enough!), it will boot into production image on the SD card.
Follow closely the instructions to install to eMMC or SPI-NAND, depending on what you prefer (ie. use fw_setenv bootcmd ... and reboot, then remove the SD card once installation has completed)
4a. If you decided for SPI-NAND, set the boot switch SW1 to position 0.
Once again, remove the device from power and re-connect it.
Now the device should boot from SPI-NAND or eMMC without the SD card inserted into the production image which came from the SD card.
Use sysupgrade ...squashfs-sysupgrade.itb to flash you own build.
Be happy.

Do not wildly write things manually to any partitions of the eMMC or NAND flash, it will only make things worse.

jpsollie1 · April 19, 2022, 7:31pm

Because trying is fun ...
And yes, I'm bricking my device, but without try-and-error, I'll never be able to make a ath10k, ath11k, and mt76 work together to have an ap for all 3 frequencies.
Yes, I know it's annoying
I tried wiki before asking things, but if it doesn't work, I also want to know why.
And due to all help here, I'm learning a lot about embedded programming

jpsollie1 · April 20, 2022, 3:36am

and my mistake is clearly at position 2-3:
while I tried a separate SDcard as well, I thought setting fw_setenv bootcmd ... from recovery would be sufficient and a separate reboot would not be neecessary