Re: TRENDnet AC2600 (TEW-827DRU)

Regarding how OpenWRT currently does the UBI upgrade process:

package/system/procd/files/nand.sh is the file where bash funcs like nand_upgrade_tar() are located. This includes the logic to determine if the kernel is on it's own NAND partition or if it's supposed to be inside the UBI as a volume.

In our case, the kernel is in the UBI as a volume.

The OEM system says our Linux cmdline was: "console=ttyHSL1,115200n8 ubi.mtd=rootfs root=mtd:ubi_rootfs rootfstype=squashfs"

This presents a problem because these cmdline arguments are not in the uboot environment. They appear to be in uboot's "ipqboot", which I currently know nothing about, which means I can't change this.

OpenWRT assuming that the UBI NAND parition will be named "rootfs" is fine and that works for us. The kernel volume is named "kernel" and that's also fine. The name of the overlay/ubifs volume isn't relevant here because that's built into our system. However, OpenWRT seems to assume that our squashfs volume will be named "rootfs" and that cmdline needs it to be "ubi_rootfs". That may end up being a problem I have to fix. At least I know where to fix it now.

Re: TRENDnet AC2600 (TEW-827DRU)

Tonight I was looking for info on what format the factory image is going to need to be in. I looked at the OEM firmware file and noticed it had what looked like an ID tailer on the end.

Then I looked at the OEM filesystem and it's firmware updater web script.

Apparently it's Cameo based.

Our hardware ID for the TEW-827DRU is "AP148AR9880-RT-150127-00".

The GPL dump has a add_hwid.sh script which is literally:

echo -n "AP148AR9880-RT-150127-00" >> bin/$firmware

But it looks page aligned on the image, so I'm not sure that all we need. I'll look into it more in next few days.

Re: TRENDnet AC2600 (TEW-827DRU)

Remember how I said I could not find anything that was using the rootfs_1 and 0:APPSBL_1 NAND partitions?

It looks like the OEM upgrade process does write to them, which is strange.

I did an upgrade from the OEM web interface and caught some info from that. Then once I figured out what it was doing, I tftp'd over a copy of the OEM upgrade file and then ran "cameo_sysupgrade TEW827DRU_FW100B11.bin" on the console, which is what I suspect the web interface does (but directs stdout to /dev/null).

stdout says that it writes to these parts in the following order (probably with mtd based on the output):
SBL1
SBL2
SBL3
APPSBL_1
DDRCONFIG
SSD
TZ
RPM

Then it unmounts the current ubi0

It then writes to the BOOTCONFIG partition. That's interesting and we will come back to that again.

Next it attaches mtd14 as ubi0! That's rootfs_1.

Then... it restarts the system. WTF? It (apparently) didn't touch the current rootfs (mtd11) or or 0:APPSBL (uboot).

So, I don't understand how the mtd11 UBI is being written. The reboot process is pretty quick, so I don't see the time when mtd11(rootfs) and mtd8(0:APPSBL) gets overwritten.

That BOOTCONFIG partition? I suspect it's a pointer to something in the early boot loader.

This is in the OEM filesystem, and based on my previous dumps, this is all that's in that partition:

root@TEW-827DRU:/tmp#  strings bootconfig.bin
0:APPSBL
rootfs
kernel



I'm not sure that I'll care about any of this once LEDE/OpenWRT is running. It might become an issue again later when I try to use more of the NAND flash than is currently being used by the mtd11(rootfs) UBI.

I don't even understand why they would make a redundant scheme like this. It's fairly pointless given a good low-level recovery mechanism. Same goes for the Linksys EA8500, which has something similar. This all seems needlessly complicated.

Re: TRENDnet AC2600 (TEW-827DRU)

Source for the uboot "bootipq" command is at qsdk_gpl/qca/src/u-boot/arch/arm/cpu/armv7/ipq/cmd_bootipq.c in the OEM source dump. It's worth looking at.

Looks like some junk in there for kernel signing/DRM, the UBI assembly/setup commands, the BOOTCONFIG selection mechanism, and a lot of junk that doesn't apply to our hardware.

I will come back to this later when I think about replacing the OEM uboot.

Re: TRENDnet AC2600 (TEW-827DRU)

Regarding the OEM factory image format, I think I have that mostly figured out.

It's just a FIT image with a Cameo tailer on the end. That's it.

However neither the dumpimage command on my Linux distro and the LEDE dumpimage from /build_dir/host.../tools/ seems to work right to extract the files. The dumpimage on the OEM system works fine though. I'll look more into what the problem is there later. Our factory upgrade image will need to be extracted by the OEM system (and imxtract in uboot).

If we look on the OEM system, the /lib/upgrade/platform.sh file has all the useful functions that sysupgrade/cameo_upgrade uses on this system. It's worth reading. I need to study it further, but there is code regarding image signing verification (disabled on this system), the sysupgrade process, extraction from the sysupgrade image, and the "Fail safe upgrade" system which writes to the BOOTCONFIG NAND part.

Regarding the Cameo signature/Hardware_ID on the tail of the factory image; in LEDE there is an example in target/linux/ar71xx/image/legacy.mk in "define Image/Build/Cameo".

Re: TRENDnet AC2600 (TEW-827DRU)

I got dumpimage on my Linux desktop to work. I guess it could not automatically figure out the format, so I had to add -T flat_dt

https://patchwork.ozlabs.org/patch/429250/

The dumpimage command is a fucktarded mess of disfunctionality. The help output is wrong/missing/bad.

This works to dump the 19th file in the FIT image, which is the UBI.

dumpimage -i TEW827DRU_FW100B11.bin -T flat_dt -p 19 ubi.bin

Here is a bash script to exract everything:

# Extract all the files from a FIT image.
# WARNING: This won't work on ash shell because it doesn't support process substitution via "< <()"!
# File names will be taken from "Description:"
IMG_FILE=TEW827DRU_FW100B11.bin
XTRACT_DIR=${IMG_FILE}_extracted
mkdir $XTRACT_DIR
while read LINE ; do
    # Ignore all the types of lines we don't care about.
    (echo "$LINE" | egrep "Created: |Type: |Compression: |Data Size: |Architecture: |Load Address: |Hash algo: |Hash value: " &> /dev/null) && continue
    # Look for Image ID lines.
    if (echo "$LINE" | egrep "Image [0-9]+ (.+)" &> /dev/null) ; then
        # XIMG_ID=$(echo "$LINE" | awk '{print $3}' | tr -d '()')
        XIMG_ID=$(echo "$LINE" | awk '{print $2}')
        continue
    fi
    # Look for Description lines. That's our file name.
    if (echo "$LINE" | egrep "Description: " &> /dev/null) ; then
        XFILE_NAME=$(echo "$LINE" | awk '{print $2}')
        # Now we should have a match
        echo "FOUND: XIMG_ID = $XIMG_ID , XFILE_NAME = $XFILE_NAME"
        # Extract it.
        dumpimage -i $IMG_FILE -T flat_dt -p $XIMG_ID ./$XTRACT_DIR/$XFILE_NAME
    fi
done < <(dumpimage -l $IMG_FILE)
unset IMG_FILE XTRACT_DIR LINE XIMG_ID XFILE_NAME

32 (edited by jmomo 2016-07-30 05:50:56)

Re: TRENDnet AC2600 (TEW-827DRU)

Something is fucky about the way this thing boots.

I went and nand erased rootfs (mtd11 at 0x58a0000 0x4000000). That causes the system to stop booting, like I expected. uboot fails and it stops at the prompt:


(IPQ) # nand erase 0x58a0000 0x4000000

NAND erase: device 0 offset 0x58a0000, size 0x4000000
Erasing at 0x9880000 -- 100% complete.
OK
(IPQ) # 

U-Boot 2012.07 [Standard IPQ806X.LN,r40331] (Nov 17 2015 - 19:33:37)

smem ram ptable found: ver: 0 len: 5
DRAM:  491 MiB
NAND:  SF: Unsupported manufacturer 00
ipq_spi: SPI Flash not found (bus/cs/speed/mode) = (0/0/48000000/0)
256 MiB
MMC:   
PCI0 Link Intialized
PCI1 Link Intialized
In:    serial
Out:   serial
Err:   serial
cdp: get part failed for 0:HLOS
Net:   MAC1 addr:0:3:7f:xx:yy:zz
athrs17_reg_init: complete
athrs17_vlan_config ...done
S17c init  done
MAC2 addr:0:3:7f:xx:yy:zz
eth0, eth1
Hit any key to stop autoboot:  0 
MMC Device 0 not found
MMC Device 0 not found
invalid partition number 11 for device nand0 (nand0)
no such partition
Creating 1 MTD partitions on "nand0":
0x0000058a0000-0x0000098a0000 : "mtd=0"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: empty MTD device detected
UBI: create volume table (copy #1)
UBI: create volume table (copy #2)
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=0"
UBI: MTD device size:            64 MiB
UBI: number of good PEBs:        512
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     0
UBI: available PEBs:             503
UBI: total number of reserved PEBs: 9
UBI: number of PEBs reserved for bad PEB handling: 5
UBI: max/mean erase counter: 1/0
Volume kernel not found!
UBI: mtd1 is detached from ubi0

(IPQ) # 

So I did the uboot emergency web loader route and let it try to repair the system...

And it never touched rootfs. Only rootfs_1 was written to, per the console log:

Flashing ubi:                           ## Copying 'ubi-4247a56920048807ca170b332e81164b8067496f' subimage from FIT image at 42000000 ...
crc32+ 
NAND erase: device 0 offset 0x1340000, size 0x4000000
Erasing at 0x5320000 -- 100% complete.
OK

NAND write: device 0 offset 0x1340000, size 0x1860000
 25559040 bytes written: OK
[ done ]
resetting ...

Offset 0x1340000 is rootfs_1, not rootfs.

Additionally, this is consistent with the contents of the OEM FIT image script, which doesn't have any entry for rootfs/mtd11/0x58a0000. It only has lines which erase/load for rootfs_1 and 0:APPSBL_1.

But the system did recover and booted normally. Evidence indicates it booted from rootfs, not rootfs_1.

I'm not sure why this recovery is working when I've not seen any evidence that rootfs is being repaired.

Re: TRENDnet AC2600 (TEW-827DRU)

I was wrong. It DID boot from rootfs_1 after the recovery:

NAND write: device 0 offset 0xa00000, size 0x13800
 79872 bytes written: OK
[ done ]
Flashing ubi:                           ## Copying 'ubi-4247a56920048807ca170b332e81164b8067496f' subimage from FIT image at 42000000 ...
crc32+ 
NAND erase: device 0 offset 0x1340000, size 0x4000000
Erasing at 0x5320000 -- 100% complete.
OK

NAND write: device 0 offset 0x1340000, size 0x1860000
 25559040 bytes written: OK
[ done ]
resetting ...

Resetting with watch dog!


U-Boot 2012.07 [Standard IPQ806X.LN,r40331] (Nov 17 2015 - 19:33:37)

smem ram ptable found: ver: 0 len: 5
DRAM:  491 MiB
NAND:  SF: Unsupported manufacturer 00
ipq_spi: SPI Flash not found (bus/cs/speed/mode) = (0/0/48000000/0)
256 MiB
MMC:   
PCI0 Link Intialized
PCI1 Link Intialized
In:    serial
Out:   serial
Err:   serial
cdp: get part failed for 0:HLOS
Net:   MAC1 addr:0:3:7f:xx:yy:zz
athrs17_reg_init: complete
athrs17_vlan_config ...done
S17c init  done
MAC2 addr:0:3:7f:xx:yy:zz
eth0, eth1
Hit any key to stop autoboot:  0 
MMC Device 0 not found
MMC Device 0 not found
invalid partition number 11 for device nand0 (nand0)
no such partition
Creating 1 MTD partitions on "nand0":
0x000001340000-0x000005340000 : "mtd=0"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: volume 2 ("ubi_rootfs_data") re-sized from 1 to 310 LEBs
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=0"
UBI: MTD device size:            64 MiB
UBI: number of good PEBs:        512
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     3
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 512
UBI: number of PEBs reserved for bad PEB handling: 5
UBI: max/mean erase counter: 1/0
Read 0 bytes from volume kernel to 44000000
No size specified -> Using max size (1904640)
   Image Name:   Linux-3.4.103
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    1851792 Bytes = 1.8 MiB
   Load Address: 41508000
   Entry Point:  41508000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

device nand0 <nand0>, # parts = 1
 #: name                size            offset          mask_flags
 0: fs                  0x04000000      0x01340000      0

active partition: nand0,0 - (fs) 0x04000000 @ 0x01340000

defaults:
mtdids  : none
mtdparts: none
Using machid 0x1260 from environment

Starting kernel ...

Re: TRENDnet AC2600 (TEW-827DRU)

I tried using the uboot web recovery system again and it is still booting from rootfs_1/mtd14/0x1340000.

I don't think the web recovery system knows anything about this safe boot/BOOTCONFIG system. However, it is writing the bootconfig image, which is in the OEM image file.

It looks like the OEM OS reads that MTD part into this proc file:

root@TEW-827DRU:/# hexdump -C /proc/boot_info/getbinary
00000000  a0 a1 a3 a5 00 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
00000050  00 00 00 00                                       |....|
00000054

Notice byte 5 is 00.

I confirmed this via nand dump in uboot on the next boot.

Then I upgraded the image from the OEM primary web manager (not the uboot web recovery system).

On reboot I checked nand dump again

(IPQ) # nand dump 0x5340000
Page 05340000 dump:
        a0 a1 a3 a5 01 00 00 00  03 00 00 00 30 3a 41 50
        50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00
        01 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00
        00 00 00 00 00 00 00 00  01 00 00 00 6b 65 72 6e
        65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00
        00 00 00 00 ...

Byte 5 changed.

The whole thing is convoluted and pointless anyway. I just need to make sure it won't cause problems when we install OpenWRT from the OEM web interface.

Re: TRENDnet AC2600 (TEW-827DRU)

In the OEM upgrade image file, that flash.scr script (file 0 in the FIT image) appears to be a huge waste of space.

It's the same 390-line script repeated 14 times, except line 6 in each of the 14, which has a different $machid if;then.

Re: TRENDnet AC2600 (TEW-827DRU)

Today I'm looking at the "factory"/OEM image upgrade methods, so that I can make LEDE build a factory image that we can load.

For this router, I would like to appease both the uboot emergency HTTP upgrader and the OEM/main web upgrader tools.



First I'm looking at the uboot emergency HTTP image uploader, since that's low-level and our best bet for loading an image.

It should be noted that the uboot-HTTP method does not appear to look for the Cameo signature tailer. There is code in cgi.c and httpd.c to look for it, but it appears to be disabled/broken (grep for "hwid").

The method for doing the upgrade is very short and simple:

  • You HTTP POST the image file.

  • The image is written to RAM at 0x42000000

  • This is executed: setenv("bootcmd", "setenv imgaddr 0x42000000; source 0x42000000:script;reset");

  • This sets the $imgaddr environment var, executes "source 0x42000000:script", and then executes the "reset" command.

That's it.

OEM source files: qsdk_gpl/qca/src/u-boot/httpd/cgi.c and httpd.c

In regards to the "source 0x42000000:script" uboot command, I'm not familiar with the :script part. I guess it can read a FIT image and reference a specific file/image within it based on it's Image name (in parenthesis).

As a reminder, here's image 0 in the OEM upgrade file, which is a FIT/ITB format file:

-->dumpimage -l TEW827DRU_FW100B11.bin
FIT description: Flashing nand 800 20000
Created:         Tue Nov 17 04:38:56 2015
 Image 0 (script)
  Description:  flash.scr
  Created:      Tue Nov 17 04:38:56 2015
  Type:         Script
  Compression:  uncompressed
  Data Size:    207912 Bytes = 203.04 kB = 0.20 MB
  Hash algo:    crc32
  Hash value:   8509413b

I also suspect the uboot imxtract command can extract specific FIT Image files via name like so: "imxtract $imgaddr script".

In regards to LEDE, it doesn't look like the Makefile "fit" method supports arbitrarily including a script file, so that's going to take some work to create a new method, but I'll get to that later in the week.

37 (edited by jmomo 2016-07-31 02:08:36)

Re: TRENDnet AC2600 (TEW-827DRU)

Like Pepe observed earlier, "D-Link"s name is all over the source code for this router. It's in some config files. TRENDnet didn't do a good job of cleaning this code up before they shipped.  smile

38 (edited by jmomo 2016-07-31 03:15:48)

Re: TRENDnet AC2600 (TEW-827DRU)

In regards to the OEM TRENDnet's main HTTP UI image uploader/upgrade method:

It looks like the source of the Cameo signature/hardware_ID is in /etc/config/cameo on the local system as "option fw_hwid 'AP148AR9880-RT-150127-00'" under the "config system 'system'" section. I grepped through a dump of the filesystem from my booted system and don't see the signature string anywhere else.

Cameo's stuff is closed-source, but this is what I've figured out so far...

On the HTTP side, /www/cgi/ssi has all the magic. It's a stripped binary but there's plenty of strings and other stuff that gives me hints about what is going on.

It looks like /sbin/cameo_sysupgrade gets executed against the downloaded file. That probably checks the signature/hwid, though there's some evidence that the ssi file itself supports that.

cameo_sysupgrade is tiny. It just calls the regular old sysupgrade script.

sysupgrade get's all of it's useful functions from /lib/upgrade/platform.sh, which is where the important stuff is.

platform.sh functions indicate that the file should be a FIT image. The input file is validated via image_is_FIT().

image_demux() indicates "dumpimage -i" is used to extract. Note the lack of -T usage in this version of dumpimage.

One important feature is that platform_check_image() examines the FIT image to make sure it has an Image/File named "ubi.*". It checks this via the following command:

dumpimage -l ${img} | grep -q "^ Image.*(${sec}.*)"

Where sec=ubi

in the case of the OEM upgrade image file, Image 19 is "ubi-4247a56920048807ca170b332e81164b8067496f".

So our factory image will need to contain such a named Image.

This platform_check_image() func also allows and prohibits other named Images in the FIT. If the names are not right, they don't get loaded. Worth reading.

So when we get down to it, platform_do_upgrade() runs flash_section(), which then iterates through the list of possible images to flash. The script/flash.scr that I mentioned in the uboot-HTTP image loader? It's an ignored volume, so it does nothing for the OEM HTTP system upgrader.

Since the UBI image is the only one we really care about for now, we look at the do_flash_ubi() func.

do_flash_ubi() hit's things in /proc/boot_info for the Fail Safe Upgrader system. It's touching the BOOTCONFIG I think.

This is apparently where the system is deciding to switch our UBI MTD parts between rootfs and rootfs_1.

/proc/boot_info/rootfs/upgradepartition is called, which for my running system right now is "rootfs_1". So that's why the system switches between parts. However, this is stupidly broke, which I'll explain in a moment.

In do_flash_ubi(), "ubiformat /dev/${mtdpart} -y -f" gets called and that's what actually writes the new UBI image.

Finally it looks like platform_do_upgrade() is where the BOOTCONFIG related Safe Upgrade stuff happens to document that an upgrade is in progress/has-occurred. I still need to look into the uboot code to see what is reading that partition and how it evaluates the contents.


That Fail Safe Upgrade system is trash. It just does not work. Worse, I think it might cause upgrades to fail.

As an example, for my current running system, it did the following during boot:
  First the APPSBL uboot is loaded. I can tell this because the version of uboot in APPSBL_1 is different (compile date).
  Second, uboot selected rootfs_1 (mtd14) to boot the kernel from.
  Then kernel then mounted rootfs (mtd11) for the squash and ubifs!

Even when I broke the UBI, the system failed to switch over to the other.

I would be afraid to test overwriting APPBL/uboot. I don't know if the low-level multi-stage bootloader is smart enough to try the backup APPSBL_1.

It is also noteworthy that the regular upgrade process appears to not touch/upgrade APPSBL_1 at all. It doesn't get upgraded.

This must be why our uboot "bootcmd=bootipq". I bet bootipq is supposed to change the bootargs so that the kernel knows which UBI to load. Well, it's not working.

Whatever the intent of this Fail Safe system was, it's broken.



Anyway, what I learned about that OEM web system upgrader is that it needs a FIT image (compatible with that version of dumpimage) with a "ubi.*" named UBI Image inside and a Cameo signature/hwid on the end. It does not use the script like the uboot-HTTP upgrader does, but it's safe to include it (it will be ignored).

I'm not completely sure about the Cameo signature since I can't see how that's evaluated, but I figure it's like previous images we already know about.

39 (edited by jmomo 2016-07-31 03:23:55)

Re: TRENDnet AC2600 (TEW-827DRU)

FACTORY IMAGE REQUIREMENTS:
    FIT -T flat_dt format
    FIT Image named "script" (preferably in position 0) for the uboot-HTTP upgrader
    FIT Image named "ubi.*" for the OEM-HTTP upgrader
    UBI image
    Kernel, squash, and ubifs in UBI
    Cameo signature for the OEM-HTTP upgrader

40 (edited by jmomo 2016-07-31 06:54:18)

Re: TRENDnet AC2600 (TEW-827DRU)

That bootipq stuff is definitely going to be a problem, and it's a bug. Not sure if I should contact TRENDnet or not just because I'm not sure that they would care. They are going to find out if they issue a firmware update though, because this could break routers.

Every time you upgrade the firmware via the OEM web interface, the BOOTCONFIG partition gets updated.

I have not yet evaluated if the system is flipping between the primary and alternate u-boot images. I know that when I dumped the router's mtd partitions as soon as I got the router, they were different, but I've upgraded the firmware many times since then and it's possible they are the same now. That being said, even if we did corrupt the active u-boot partition, we don't have any way of altering BOOTCONFIG when u-boot is broken (that we know of yet).

bootipq is supposed to switch between an active and inactive UBI. It does that, but in the bootipq.c code, they failed to account for the fact that the bootcmd needs to be different for each. The result is thus:

PRIMARY:       uboot --> loads UBI rootfs/mtd11 --> kernel mounts UBI rootfs/mtd11  (correct)
ALTERNATE:   uboot --> loads UBI rootfs_1/mtd14 --> kernel mounts UBI rootfs/mtd11  (WRONG)

So if TRENDnet issues a firmware update, that rootfs isn't going to be the right one.

Re: TRENDnet AC2600 (TEW-827DRU)

The workaround seems to be that I need to compile my kernel to ignore it's boot args and use a custom bootarg/cmdline. I think OpenWRT already supports that via patch-cmdline.

Re: TRENDnet AC2600 (TEW-827DRU)

Dear TRENDnet, QCA, or whoever is responsible for this bullshit;

It was stupid of you to name mtd11, a UBI volume, "rootfs".

That is all.

Re: TRENDnet AC2600 (TEW-827DRU)

I successfully flashed an image from the OEM web upgrade tool, but it got stuck booting the kernel.

It looks like the system reverted it's bootconfig from the rootfs_1 partition back to rootfs. That's interesting. It detected that the kernel failed to boot and reverted.

Not entirely unexpected trouble. I still need to work around the issue of the UBI MTD partition being named "rootfs" and the issues with the "Fail Safe" upgrade system.

I will try flashing from the u-boot HTTP loader and see if my FIT script works.

Re: TRENDnet AC2600 (TEW-827DRU)

The u-boot-HTTP/emergency firmware updater also accepted my image and ran the script, but a typo prevented it from running. I am starting a new build with some changes and then will test it out.

At this point, I've got the factory image format right for both firmware uploader tools.

Re: TRENDnet AC2600 (TEW-827DRU)

I just noticed I previously had documented some mtdparts commands incorrectly, where I had swapped the address/location of APPSBL/APPSBL_1 and rootfs/rootfs_1. I have fixed these.

Re: TRENDnet AC2600 (TEW-827DRU)

I tried to manually boot by loading that kernel into memory:

    ubi part rootfs
    ubi read.rootfs 0x45000000 kernel 0x1F0000
    bootm 0x45000000

    --
    
    (IPQ) # ubi read.rootfs 0x45000000 kernel 0x1F0000
    Read 2031616 bytes from volume kernel to 45000000
    (IPQ) # bootm 0x45000000
       Image Name:   ARM LEDE Linux-4.4.15
       Image Type:   ARM Linux Kernel Image (uncompressed)
       Data Size:    2007254 Bytes = 1.9 MiB
       Load Address: 42208000
       Entry Point:  42208000
       Verifying Checksum ... OK
       Loading Kernel Image ... OK
    OK

    device nand0 <nand0>, # parts = 15
     #: name                size            offset          mask_flags
     0: 0:SBL1              0x00040000      0x00000000      0
     1: 0:MIBIB             0x00140000      0x00040000      0
     2: 0:SBL2              0x00140000      0x00180000      0
     3: 0:SBL3              0x00280000      0x002c0000      0
     4: 0:DDRCONFIG         0x00120000      0x00540000      0
     5: 0:SSD               0x00120000      0x00660000      0
     6: 0:TZ                0x00280000      0x00780000      0
     7: 0:RPM               0x00280000      0x00a00000      0
     8: 0:APPSBL_1          0x00500000      0x00c80000      0
     9: 0:APPSBLENV         0x00080000      0x01180000      0
    10: 0:ART               0x00140000      0x01200000      0
    11: rootfs_1            0x04000000      0x01340000      0
    12: 0:BOOTCONFIG        0x00060000      0x05340000      0
    13: 0:APPSBL            0x00500000      0x053a0000      0
    14: rootfs              0x04000000      0x058a0000      0

    active partition: nand0,0 - (0:SBL1) 0x00040000 @ 0x00000000

    defaults:
    mtdids  : none
    mtdparts: none
    Setting up atags for msm partition: 0:SBL1
    Setting up atags for msm partition: 0:MIBIB
    Setting up atags for msm partition: 0:SBL2
    Setting up atags for msm partition: 0:SBL3
    Setting up atags for msm partition: 0:DDRCONFIG
    Setting up atags for msm partition: 0:SSD
    Setting up atags for msm partition: 0:TZ
    Setting up atags for msm partition: 0:RPM
    Setting up atags for msm partition: 0:APPSBL_1
    Setting up atags for msm partition: 0:APPSBLENV
    Setting up atags for msm partition: 0:ART
    Setting up atags for msm partition: rootfs_1
    Setting up atags for msm partition: 0:BOOTCONFIG
    Setting up atags for msm partition: 0:APPSBL
    Setting up atags for msm partition: rootfs
    Using machid 0x1260 from environment

    Starting kernel ...

And after a few seconds, it just reboots. That's different from bootipq, which just sits there forever when trying to start the kernel.

No idea.

Re: TRENDnet AC2600 (TEW-827DRU)

Since I am getting nothing at all out, maybe the UART address is reconfigured or something else with the console port. Not sure. This isn't a problem I've ever had before. I don't think there is any problem with the actual format of the kernel.

There could be something in this device's uboot preventing boot, but I'm not currently aware of any such issue.

Re: TRENDnet AC2600 (TEW-827DRU)

I noticed this in the OEM boot log:

    [    0.540018] Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
    [    0.543361] msm_serial_hs.0: ttyHS0 at MMIO 0x12490000 (irq = 227) is a MSM HS UART
    [    0.545235] msm_serial_hs module loaded
    [    0.545423] msm_serial_hsl: detected port #1
    [    0.545548] msm_serial_hsl.1: ttyHSL1 at MMIO 0x16340000 (irq = 184) is a MSM
    [    0.545985] msm_serial_hsl: console setup on port #1
    [    1.328709] console [ttyHSL1] enabled
    [    1.331958] msm_serial_hsl: driver initialized

I think that's what the JP12 block on the board is. It's a second serial port. There is a nearby TI WT245. The main console port that I am using is off of a TI YE04 level shifter. I'm not sure why they chose that chip. I wonder why they put it there. We would still need something like a max3232 to bring it up to 5V for RS232... maybe this was on the reference board they used, or they left it there for possible use by other platforms.

References:
http://www.ti.com/lit/ds/symlink/sn74avc4t245.pdf
http://www.ti.com/lit/ds/symlink/txb0104.pdf

This is hypothetical. I'll have to test it out some time.

This might explain why the kernel is never putting out console text.

Or my DTSI might be all broken. I barely know what I'm doing.

Re: TRENDnet AC2600 (TEW-827DRU)

I have no clue why the system is hanging at boot.

U-Boot 2012.07 [Standard IPQ806X.LN,r40331] (Nov 17 2015 - 19:33:37)

smem ram ptable found: ver: 0 len: 5
DRAM:  491 MiB
NAND:  SF: Unsupported manufacturer 00
ipq_spi: SPI Flash not found (bus/cs/speed/mode) = (0/0/48000000/0)
256 MiB
MMC:   
PCI0 Link Intialized
PCI1 Link Intialized
In:    serial
Out:   serial
Err:   serial
cdp: get part failed for 0:HLOS
Net:   MAC1 addr:0:3:7f:xx:yy:zz
athrs17_reg_init: complete
athrs17_vlan_config ...done
S17c init  done
MAC2 addr:0:3:7f:xx:yy:zz
eth0, eth1
Hit any key to stop autoboot:  0 
MMC Device 0 not found
MMC Device 0 not found
Creating 1 MTD partitions on "nand0":
0x0000058a0000-0x0000098a0000 : "mtd=0"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=0"
UBI: MTD device size:            64 MiB
UBI: number of good PEBs:        512
UBI: number of bad PEBs:         0
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     3
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 512
UBI: number of PEBs reserved for bad PEB handling: 5
UBI: max/mean erase counter: 1/0
Read 0 bytes from volume kernel to 44000000
No size specified -> Using max size (2031616)
   Image Name:   ARM LEDE Linux-4.4.15
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    2008105 Bytes = 1.9 MiB
   Load Address: 42208000
   Entry Point:  42208000
   Verifying Checksum ... OK
   Loading Kernel Image ... OK
OK

device nand0 <nand0>, # parts = 1
 #: name                size            offset          mask_flags
 0: fs                  0x04000000      0x058a0000      0

active partition: nand0,0 - (fs) 0x04000000 @ 0x058a0000

defaults:
mtdids  : none
mtdparts: none
Using machid 0x1260 from environment

Starting kernel ...

I've tried building the kernel with LL debugging/printk stuff but I get nothing at all.

I don't think the problem is the serial port anymore. My OEM bootlog and similar device bootlogs all look the same. That serial port is on gspi4/0x16340000 just like the rest.

50 (edited by jmomo 2016-09-20 09:36:32)

Re: TRENDnet AC2600 (TEW-827DRU)

Here is what I know about the "Fail Safe" system and the BOOTCONFIG partition:

The partition is 392KB in size, but only the first 84 bytes is used by the Fail Safe system, so far as I can tell. The rest of the partition is usually blank.

The uboot-HTTP recovery loader system is not technically aware of this partition or it's data, but the OEM flash.scr script used to write the partition images overwrites bootconfig with a default image so that on next boot so that the _1 partitions are used. The flash.scr script only writes to those _1 partitions. It does not write to "rootfs" or "APPSBL". Our LEDE factory image also writes a default configuration to this partition for the same reasons.

The OEM/Cameo web firmware upgrader is Fail Safe aware, and it does not use the flash.scr script at all. The upgrade process uses  /proc/boot_info/ to mark the bootconfig partition bits in various ways. Depending on which the active partition set is, the OEM sysupgrade only writes to the alternate/backup partitions, and then the SBL/low-level bootloader boots from whichever u-boot(APPSBL) is active on next boot.

The bootipq bootcmd program in u-boot is where the failover decision is made. Bootipq removes the "upgraded" partition bits from bootconfig, but leaves the "upgrade-in-progress" bit alone. If bootipq then sees "upgrade-in-progress" set but the partition "upgraded" bits missing, it considers that an upgrade failure. It then flips the active partitions. I'm not certain if it also clears the "upgrade-in-progress" bit or not. I image it leaves it and potentially it would just flip back and forth forever if both were broken.

So, every time you upgrade by using the OEM/Cameo web firmware upgrade tool, it flips from the active to backup rootfs/uboot partitions.

Beware that I've found a few errors in my notes below. If you find something that looks wrong, point it out and I will correct it or clarify.

From uboot, we can dump the BOOTCONFIG part like so:

nand dump 0x5340000 0x60000


So, what' in those 84 bytes and what do they do?

byte 5 (0x04)        0x1 = Upgrade in progress bit
byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active
byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)
byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active
byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)

I've not yet had a chance to really follow the uboot code. I learned what those bits did by following the OEM system upgrade process and running a loop that dumped the mtd continuously during an upgrade. Then I tftp dumped the partition in u-boot on each boot, before bootipq and the OS touched it.

The OEM source files related to this are probably here. I have only briefly reviewed them:
  qsdk_gpl/qca/src/linux/arch/arm/mach-msm/bootconfig_partition.c
  qsdk_gpl/qca/src/u-boot/arch/arm/cpu/armv7/ipq/cmd_bootipq.c

--

In summary:

1.) The OEM upgrade system sets the "upgrade-in-progress" bit and partition "upgraded" bits in BOOTCONFIG after an upgrade.

2.)     The bootipq program in u-boot unsets the APPSBL "upgraded" bit, but leaves the "upgrade-in-progress" and "rootfs upgraded" bits alone. This is why a broken u-boot will brick the device: the low-level SBL boot loader has no means of switching to the previous u-boot partition as far as I can tell. It's dumb!

3.) Finally the new OS is supposed to clear the "upgrade-in-progress" and "rootfs upgraded" bits. If this fails to happen, the system falls back to the previous image the next time bootipq runs. The OEM system handles this in the /etc/init.d/commit_sysupgrade file

--


Understanding/documenting the upgrade process:

When the device is unboxed, it has a clean factory-fresh bootconfig. The default is that we are booting from APPSBL_1 and rootfs_1:

00000000  a0 a1 a3 a5 00 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
00000010  50 53 42 4c 00 00 00 00  00 00 00 00 01 00 00 00  |PSBL............|
00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
00000030  00 00 00 00 01 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000800  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00060000


Here is what it looks like after we install the OEM OS with the u-boot HTTP recovery loader. Notice the FIT header info in there! That's all garbage. Only the first 84 bytes matter:

00000000  a0 a1 a3 a5 00 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
00000050  00 00 00 00 00 00 00 03  00 00 00 09 00 00 00 11  |................|
00000060  66 69 72 6d 77 61 72 65  00 00 00 00 00 00 00 03  |firmware........|
00000070  00 00 00 04 00 00 00 16  61 72 6d 00 00 00 00 03  |........arm.....|
00000080  00 00 00 05 00 00 00 1b  6e 6f 6e 65 00 00 00 00  |........none....|
00000090  00 00 00 01 68 61 73 68  40 31 00 00 00 00 00 03  |....hash@1......|
000000a0  00 00 00 04 00 00 00 36  16 46 0a 74 00 00 00 03  |.......6.F.t....|
000000b0  00 00 00 06 00 00 00 27  63 72 63 33 32 00 00 00  |.......'crc32...|
000000c0  00 00 00 02 00 00 00 02  00 00 00 01 73 62 6c 31  |............sbl1|
000000d0  2d 37 34 61 66 36 36 63  32 38 31 37 66 34 38 64  |-74af66c2817f48d|
000000e0  62 65 62 35 62 37 64 62  39 34 30 36 30 32 65 33  |beb5b7db940602e3|
000000f0  66 38 36 38 35 65 36 32  38 00 00 00 00 00 00 03  |f8685e628.......|
00000100  00 00 00 0e 00 00 00 00  6e 61 6e 64 5f 73 62 6c  |........nand_sbl|
00000110  31 2e 6d 62 6e 00 00 00  00 00 00 03 00 01 00 10  |1.mbn...........|
00000120  00 00 00 0c d1 dc 4b 84  34 10 d7 73 5a 43 0b 7d  |......K.4..sZC.}|
00000130  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00060000


It is noteworthy that the OEM firmware upgrade image "bootconfig.bin" file is only 54 bytes in size. That's the cutoff for byte 53, which is used to indicate which UBI is active. Also, they don't clean memory before they write this, so the next 747 bytes is garbage from whatever the previously RAM-loaded image was, and the remaining 59200 bytes is basically never touched. That's why in the hexdump above we see ff from 0x800 onward:

00000000  a0 a1 a3 a5 00 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
00000050  00 00 00 00                                       |....|
00000054

--

Here is a run though some testing I did. This is a combination of my notes and dumping the output from /proc/boot_info on the OEM system:

Booted from rootfs: Pre-upgrade (no upgrade in progress)
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    0    no upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    0    no u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    0    no UBI upgrade in progress
    --
    00000000  a0 a1 a3 a5 00 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 01 00 00 00  |PSBL............|
    00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 01 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00                                       |....|
    00000054
    /proc/boot_info/rootfs/upgradepartition : rootfs_1
    /proc/boot_info/rootfs/upgraded : 0
    /proc/boot_info/rootfs/primaryboot : 1
    /proc/boot_info/APPSBL/upgradepartition : APPSBL_1
    /proc/boot_info/APPSBL/upgraded : 0
    /proc/boot_info/APPSBL/primaryboot : 1
    /proc/boot_info/numaltpart : 3
    /proc/boot_info/upgradeinprogress : 0
    /proc/boot_info/magic : a5a3a1a0

Booted from rootfs: First change seen, upgrade-in-progress, and appsbl-upgraded
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    1    yes upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    1    yes u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    0    no UBI upgrade in progress
    --
    00000000  a0 a1 a3 a5 01 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 01 00 00 00  |PSBL............|
    00000020  01 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 01 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00                                       |....|
    00000054
    /proc/boot_info/rootfs/upgradepartition : rootfs_1
    /proc/boot_info/rootfs/upgraded : 0
    /proc/boot_info/rootfs/primaryboot : 1
    /proc/boot_info/APPSBL/upgradepartition : APPSBL_1
    /proc/boot_info/APPSBL/upgraded : 1
    /proc/boot_info/APPSBL/primaryboot : 1
    /proc/boot_info/numaltpart : 3
    /proc/boot_info/upgradeinprogress : 1
    /proc/boot_info/magic : a5a3a1a0

Booted from rootfs: Now UBI got marked as upgraded
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    1    yes upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    1    yes u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    1    yes UBI upgrade in progress
    --
    00000000  a0 a1 a3 a5 01 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 01 00 00 00  |PSBL............|
    00000020  01 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 01 00 00 00  01 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00                                       |....|
    00000054
    /proc/boot_info/rootfs/upgradepartition : rootfs_1
    /proc/boot_info/rootfs/upgraded : 1
    /proc/boot_info/rootfs/primaryboot : 1
    /proc/boot_info/APPSBL/upgradepartition : APPSBL_1
    /proc/boot_info/APPSBL/upgraded : 1
    /proc/boot_info/APPSBL/primaryboot : 1
    /proc/boot_info/numaltpart : 3
    /proc/boot_info/upgradeinprogress : 1
    /proc/boot_info/magic : a5a3a1a0

Booted from rootfs: Next boot is on rootfs_1 (reboot happened)
    0x000001340000-0x000005340000 : "mtd=0"

    bootipq unsets byte 33 (0x20). However, it leaves byte 5 and 57 alone, which is the upgrade-in-progress and rootfs upgraded bytes, untouched.

    Unfortunately, if we were to corrupt our active u-boot partition, there would be no possibility of recovery as far as I can tell. The low-level SBL1-3 bootloader code would have to recognize a failure mode, and it apparently does not.

    The bootipq program in u-boot can recognize when it has attempted to boot an OS (linux), but that the OS failed to remove the "upgrade-in-progress" or "rootfs upgraded" bytes on boot like it should have. Should bootipq discover the condition where "upgrade-in-progress" or "rootfs upgraded" are set 1, it will assume an upgrade failure and then flip the "partition-active" bits. This will result in the system booting from the old UBI (and probably u-boot, not sure).

Booted from rootfs_1: Pre-upgrade (no upgrade in progress)
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    0    no upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL_1 u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    0    no u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs_1 UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    0    no UBI upgrade in progress
    --
    00000000  a0 a1 a3 a5 00 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
    00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 00 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00                                       |....|
    00000054
    /proc/boot_info/rootfs/upgradepartition : rootfs_1
    /proc/boot_info/rootfs/upgraded : 0
    /proc/boot_info/rootfs/primaryboot : 0
    /proc/boot_info/APPSBL/upgradepartition : APPSBL_1
    /proc/boot_info/APPSBL/upgraded : 0
    /proc/boot_info/APPSBL/primaryboot : 0
    /proc/boot_info/numaltpart : 3
    /proc/boot_info/upgradeinprogress : 0
    /proc/boot_info/magic : a5a3a1a0

Booted from rootfs_1: Upgrade now in progress, appsbl already marked upgraded
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    1    yes upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL_1 u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    1    yes u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs_1 UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    0    no UBI upgrade in progress
    --
    00000000  a0 a1 a3 a5 01 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
    00000020  01 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 00 00 00 00  00 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00                                       |....|
    00000054
    /proc/boot_info/rootfs/upgradepartition : rootfs_1
    /proc/boot_info/rootfs/upgraded : 0
    /proc/boot_info/rootfs/primaryboot : 0
    /proc/boot_info/APPSBL/upgradepartition : APPSBL_1
    /proc/boot_info/APPSBL/upgraded : 1
    /proc/boot_info/APPSBL/primaryboot : 0
    /proc/boot_info/numaltpart : 3
    /proc/boot_info/upgradeinprogress : 1
    /proc/boot_info/magic : a5a3a1a0

Booted from rootfs_1: Last message, both parts marked as upgraded
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    1    yes upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL_1 u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    1    yes u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs_1 UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    1    yes UBI upgrade in progress
    --
    00000000  a0 a1 a3 a5 01 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
    00000020  01 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 00 00 00 00  01 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00                                       |....|
    00000054
    /proc/boot_info/rootfs/upgradepartition : rootfs_1
    /proc/boot_info/rootfs/upgraded : 1
    /proc/boot_info/rootfs/primaryboot : 0
    /proc/boot_info/APPSBL/upgradepartition : APPSBL_1
    /proc/boot_info/APPSBL/upgraded : 1
    /proc/boot_info/APPSBL/primaryboot : 0
    /proc/boot_info/numaltpart : 3
    /proc/boot_info/upgradeinprogress : 1
    /proc/boot_info/magic : a5a3a1a0

Booted from rootfs_1: Next boot is on rootfs
    0x0000058a0000-0x0000098a0000 : "mtd=0"

Finally, here is a LEDE system right after an upgrade from the OEM web interface where we have not corrected the "upgrade-in-progress" bit.
    byte 5 (0x04)        0x1 = Upgrade in progress bit                                    =    1    yes upgrade in progress
    byte 29 (0x1c)        0x1 = APPSBL partition active, 0x0 = APPSBL_1 partition active    =    0    APPSBL_1 u-boot active
    byte 33 (0x20)        0x1 = APPSBL upgraded bit (SBL flips byte 29 on next boot)        =    0    no u-boot upgrade in progress
    byte 53 (0x34)        0x1 = rootfs partition active, 0x0 = rootfs_1 partition active    =    0    rootfs_1 UBI active
    byte 57 (0x38)        0x1 = rootfs upgraded bit (SBL flips byte 53 on next boot)        =    0    no UBI upgrade in progress
    --
    root@lede:/# hexdump -C /dev/mtd12
    00000000  a0 a1 a3 a5 01 00 00 00  03 00 00 00 30 3a 41 50  |............0:AP|
    00000010  50 53 42 4c 00 00 00 00  00 00 00 00 00 00 00 00  |PSBL............|
    00000020  00 00 00 00 72 6f 6f 74  66 73 00 00 00 00 00 00  |....rootfs......|
    00000030  00 00 00 00 00 00 00 00  01 00 00 00 6b 65 72 6e  |............kern|
    00000040  65 6c 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |el..............|
    00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
    *
    00000800  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
    *
    00060000

    This is basically a failed system at this point. If we reboot, the u-boot bootipq program is going to revert to the old partitions because it thinks the upgrade failed.