RAVPower WD-03, Add LZMA Loader

[arrmo]

A bit more info ... sysupgrade again, but block boot => initramfs, poking around . I do see, on mtd10ro,

root@OpenWrt:/mnt# head -n 2 /dev/mtd10ro | hexdump -C
00000000  de ad c0 de ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00005000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 85 19 01 e0  |... ............|
00005010  36 00 00 00 a4 e1 55 df  01 00 00 00 00 00 00 00  |6.....U.........|
00005020  02 00 00 00 00 00 00 00  0e 08 00 00 52 83 90 67  |............R..g|
00005030  bc 3d ff 31 73 79 73 75  70 67 72 61 64 65 2e 74  |.=.1sysupgrade.t|
00005040  67 7a ff ff 85 19 02 e0  bc 0f 00 00 68 82 79 d0  |gz..........h.y.|

sysupgrade.tgz => pretty sure the file is there! Not having any luck mounting this partition - but that's me, hunting to figure it out. That aside, I did see,

root@OpenWrt:/tmp# firstboot switch2jffs
This will erase all settings and remove any installed packages. Are you sure? [N/y]
y
/dev/mtdblock10 is not mounted
/dev/mtdblock10 will be erased on next mount

So somehow it is set to erase on reboot? We want it saved, right?

Thanks!

[xabolcs]

@arrmo

So somehow it is set to erase on reboot?

Agreed!

I still don't have any hint!
But poking around within initramfs is a good thing!

@arrmo

We want it saved, right?

I don't know how the sysupgrade works.

[arrmo]

@xabolcs

I don't know how the sysupgrade works.

LOL - me neither. Blind leading the blind ... . Trying something - failsafe, see what mount_root says there! Rather odd result, after clean sysupgrade, failsafe ...

mount_root: no usable overlay filesystem found, using tmpfs overlay

Really seems like rootfs_data is broken somehow. But I may be wrong!

[xabolcs]

In case you are stucked with the sysupgrade stuff ... :

• does this sysupgrade bug affect the HooToo PR#2465?
• that full flash erase bug was a one time bug, or it's reproducible
  • do you see someting in the HooToo u-boot source code?

[arrmo]

@xabolcs

• does this sysupgrade bug affect the HooToo PR#2465?

I have a bad feeling that it affects more than just this - seems to happen even without the HooToo changes (at least the mtd-concat / recent ones).

@xabolcs

that full flash erase bug was a one time bug, or it's reproducible

Really seems to be a one time thing, can't make it happen again.

[xabolcs]

Look!

[Solved] Ramips jffs2 sysupgrade to 18.06.1 losing config

I have a ramips mt7620a -based device that is losing it's sysupgrade.tgz config when upgraded from LEDE 17.01 series to 18.06.1. There seems to be some incompatibility when doing the 'zcat sysupgrade.bin | mtd write -j /tmp/sysupgrade.tgz - firmware', where it integrates the sysupgrade.tgz into the jffs2 filesystem. This is a custom device based on the IAR-142-G: https://oringnet.com/en-global/products/detail/P0000000512/91/?IAR-142-4G_US There is plenty of free flash space (writing to a 24MB …

And here is another:

Sysupgrade source

[arrmo]

@xabolcs

Look!

I don't think think this is it, as it's between versions. But I did see that the driver has changed (see the log above). So perhaps!

@xabolcs

And here is another:

Yes, and this is very interesting,

Overlay FS support (how to enable overlay for new device)

df -Th Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 2.0M 2.0M 0 100% /rom tmpfs tmpfs 123.6M 68.0K 123.6M 0% /tmp tmpfs tmpfs 123.6M 40.0K 123.6M 0% /tmp/root overlayfs:/tmp/root overlay 123.6M 40.0K 123.6M 0% / tmpfs tmpfs 512.0K 0 512.0K 0% /dev block info /dev/sdb1: UUID="4f31ca0e-1154-4f82-a032…

I say that because after a clean boot (not sysupgrade), at the start of rootfs => 0x1985, as expected!

00000000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 ff ff ff ff  |... ............|

But, in spite of what the link says (about out of date), I did see (in failsafe),

The JFFS2 mounting code expects a special marker sequence 0xdeadc0de at the beginning of the rootfs_data partition, if this byte sequence is found; the entire rootfs_data partition is erased and reformatted as JFFS2.

So it's saving the file, to the correct place, but telling it to erase!

[arrmo]

I think I see the issue. If I go to failsafe mode, and look at rootfs_data,

root@(none):/# hexdump -C /dev/mtd11 | head -8
00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00005000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 85 19 01 e0  |... ............|
00005010  36 00 00 00 a4 e1 55 df  01 00 00 00 00 00 00 00  |6.....U.........|
00005020  02 00 00 00 00 00 00 00  0e 08 00 00 52 83 90 67  |............R..g|
00005030  bc 3d ff 31 73 79 73 75  70 67 72 61 64 65 2e 74  |.=.1sysupgrade.t|
00005040  67 7a ff ff 85 19 02 e0  bc 0f 00 00 68 82 79 d0  |gz..........h.y.|

The needed header and files are there, but why the 0x5000 offset? mount_root fails, as below ... but that sort of makes sense,

root@(none):/# mount_root
[   57.691062] mount_root: no usable overlay filesystem found, using tmpfs overlay

[arrmo]

FIXED! I can't 100% explain it . But looking at the recipes, and the information here,

Overlay FS support (how to enable overlay for new device)

Hi All, The expected marker sequence 0x1985 (JFFS2) is not found in /dev/mtdblock3. So, Overlay is mounted in /tmp/root. Hence all configurations loaded in overlay are lost in next reboot. As suggested by @jow, I tried to add following line in Makefile. $(call prepare_generic_squashfs,$(KDIR)/root.$(1)) # pad to 4k, 8k, 16k, 64k, 128k, 256k and add jffs2 end-of-filesystem mark define prepare_generic_squashfs $(STAGING_DIR_HOST)/bin/padjffs2 $(1) 4 8 16 64 128 256 endef This invokes "padjff…

I started thinking about the BLOCKSIZE ... it wasn't there before. So I tried removing it ... and now, if I look at the rootfs_data partition,

root@travelRouter:/# hexdump -C /dev/mtd11 | head -8
00000000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 85 19 01 e0  |... ............|
00000010  36 00 00 00 a4 e1 55 df  01 00 00 00 00 00 00 00  |6.....U.........|
00000020  02 00 00 00 00 00 00 00  0e 08 00 00 52 83 90 67  |............R..g|
00000030  bc 3d ff 31 73 79 73 75  70 67 72 61 64 65 2e 74  |.=.1sysupgrade.t|
00000040  67 7a ff ff 85 19 02 e0  bc 0f 00 00 68 82 79 d0  |gz..........h.y.|
00000050  02 00 00 00 02 00 00 00  80 81 00 00 00 00 00 00  |................|
00000060  a2 1a 00 00 99 52 12 5f  99 52 12 5f 99 52 12 5f  |.....R._.R._.R._|
00000070  00 00 00 00 78 0f 00 00  78 0f 00 00 00 00 00 00  |....x...x.......|

It's aligned (to 0x0), like it should be! And ... sysupgrade doesn't trash my info, it is maintained! Also, in the log now,

[    7.213561] jffs2_scan_eraseblock(): End of filesystem marker found at 0x2000
[    7.228071] jffs2_build_filesystem(): unlocking the mtd device...
[    7.228149] done.
[    7.244316] jffs2_build_filesystem(): erasing all blocks after the end marker...
[   47.686327] done.
[   47.705140] jffs2: notice: (449) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   47.737432] mount_root: overlay filesystem has not been fully initialized yet
[   47.757073] mount_root: switching to jffs2 overlay
[   47.796909] overlayfs: upper fs does not support tmpfile.

As it should be! So all good - sigh. Time for a ... LOL.

When I push this, was there a name change needed also?

Thanks!

And PS, I'm not so sure this isn't a sysupgrade bug. Thinking about it ... it says to me that the padding used to determine the start of rootfs_data is not consistent between writing (storing backup data), and reading .. agreed?

[mpratt14]

@arrmo

root@OpenWrt:/tmp# firstboot switch2jffs
This will erase all settings and remove any installed packages. Are you sure? [N/y]
y
/dev/mtdblock10 is not mounted
/dev/mtdblock10 will be erased on next mount

So somehow it is set to erase on reboot? We want it saved, right?

This is because it asked if you want to erase jffs2 and you said yes lol

I did some digging on this...

switch2jffs doesn't exist anymore, not as a parameter or a hook or script or anything...
the mount_root program handles all those features now (through libfstools)

Here is the firstboot script from 10 years ago

openwrt/openwrt/blob/7af8ee04812f647ce74adbe04f0d5d18a39bbd29/package/base-files/files/sbin/firstboot

---

---

compared to today

openwrt/openwrt/blob/master/package/base-files/files/sbin/firstboot

Now it literally just calls jffs2reset which takes no parameters, however has two actions

• if /overlay is mounted, erases files
• if /overlay is not mounted, marks the partition for deletion on next mount

I am not good at reading C code but here it is
https://git.openwrt.org/?p=project/fstools.git;a=blob;f=jffs2reset.c

if you want to mount the jffs2 partition from initramfs boot you should do something like
mount /dev/mtd10 /overlay
or
mount_root

[mpratt14]

@arrmo

the padding used to determine the start of rootfs_data is not consistent between writing (storing backup data), and reading

There is a separate image command pad-rootfs for a reason...but never sure what that reason was...

@arrmo

sysupgrade doesn't trash my info, it is maintained!

do you see the line in kernel log that indicates that sysupgrade.tgz was unpackaged?
- config restore -

[xabolcs]

@arrmo

I started thinking about the BLOCKSIZE ... it wasn't there before. So I tried removing it ...

Awesome!
I asked about BLOCKSIZE at GiHub, because it looked as a leftover, but looks like my comment / review get lost.

@arrmo

And PS, I'm not so sure this isn't a sysupgrade bug. Thinking about it ... it says to me that the padding used to determine the start of rootfs_data is not consistent between writing (storing backup data), and reading .. agreed?

It is interesting, at least!
Dig yourself into the sysupgrade code, and try to find the culcprit code and it's history!

[arrmo]

@mpratt14

There is a separate image command pad-rootfs for a reason...but never sure what that reason was...

Agreed! And that's where it clued me in to BLOCKSIZE

@mpratt14

do you see the line in kernel log that indicates that sysupgrade.tgz was unpackaged?

Nope ... but I do see the files restored (checked them).

@xabolcs

Dig yourself into the sysupgrade code, and try to find the culcprit code and it's history!

I am! As a side interest of course ... . Seems like the write is after that padding, but read after is not (or at least, they are different).

OK, all working again, just the recipes to clean up on HooToo. Then .. disassemble, reflash RAVPower => combine the dts files! Agreed?

[xabolcs]

@arrmo

Then .. disassemble, reflash RAVPower => combine the dts files! Agreed?

If you have all the original functions of RavPower working: LEDs, buttons, SD card, etc ... then you could start combining the DTS'.

[arrmo]

@xabolcs

If you have all the original functions of RavPower working: LEDs, buttons, SD card, etc ... then you could start combining the DTS'.

Exactly! Let me disassemble, OEM flash - confirm my build, go from there. Thanks!

[xabolcs]

@arrmo

... OEM flash ...

Do you have it's 8 MByte backup.bin? For the MAC addresses, calibration datas and others.

[arrmo]

@xabolcs

Do you have it's 8 MByte backup.bin? For the MAC addresses, calibration datas and others.

You are kidding, right? It was next on my list, never got to it before the blow up. But will get it restored and working, NP. Just need to crack the case open again - dang it!

[arrmo]

@arrmo

@mpratt14

do you see the line in kernel log that indicates that sysupgrade.tgz was unpackaged?

OK, missed it before somehow, but the latest and greatest serial log shows,

[   47.843030] jffs2: notice: (449) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   47.875320] mount_root: overlay filesystem has not been fully initialized yet
[   47.894991] mount_root: switching to jffs2 overlay
[   47.934849] overlayfs: upper fs does not support tmpfile.
- config restore -

Life is good

[arrmo]

Well, except for git that is . Figured I'll rebased my ravpower branch to hootoo now, thought that made sense. Now I have a bunch of commits. Arrgh, git kills me!

[xabolcs]

$ git branch -v
* hootoo-tm05 f0bfa630c8 ramips: Add support for HooToo TM05
  master      0453c3866f vxlan: fix udp checksum control

$ git fetch arrmo
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (27/27), done.
remote: Total 31 (delta 26), reused 27 (delta 26), pack-reused 4
Unpacking objects: 100% (31/31), done.
From https://github.com/arrmo/openwrt
 + f0bfa630c8...8f23744452 hootoo-tm05   -> arrmo/hootoo-tm05  (forced update)
 * [new branch]            master        -> arrmo/master
 * [new branch]            openwrt-19.07 -> arrmo/openwrt-19.07
 * [new branch]            ravpower-wd03 -> arrmo/ravpower-wd03

$ git reset --hard arrmo/hootoo-tm05
HEAD is now at 8f23744452 ramips: Add support for HooToo TM05

$ git checkout ravpower-wd03
Previous HEAD position was 8f23744452 ramips: Add support for HooToo TM05
Switched to branch 'ravpower-wd03'
Your branch is up to date with 'arrmo/ravpower-wd03'.

$ git rebase hootoo-tm05
First, rewinding head to replay your work on top of it...
Applying: ramips: Add support for HooToo TM05
Using index info to reconstruct a base tree...
M       package/boot/uboot-envtools/files/ramips
M       target/linux/ramips/image/Makefile
M       target/linux/ramips/image/mt7620.mk
M       target/linux/ramips/mt7620/base-files/etc/board.d/02_network
M       target/linux/ramips/mt7620/config-4.14
M       target/linux/ramips/mt7620/config-5.4
Falling back to patching base and 3-way merge...
Auto-merging target/linux/ramips/image/mt7620.mk
CONFLICT (content): Merge conflict in target/linux/ramips/image/mt7620.mk
Auto-merging package/boot/uboot-envtools/files/ramips
error: Failed to merge in the changes.
Patch failed at 0001 ramips: Add support for HooToo TM05
hint: Use 'git am --show-current-patch' to see the failed patch

Resolve all conflicts manually, mark them as resolved with
"git add/rm <conflicted_files>", then run "git rebase --continue".
You can instead skip this commit: run "git rebase --skip".
To abort and get back to the state before "git rebase", run "git rebase --abort".

Here would happen the magic: you had to skip the old, superseded HooToo commit!

$ git rebase --skip
Applying: ramips: Add support for RAVPower WD03

$ git log
commit 45566406e0aebbbfec99bd492e12867f1ca45f89 (HEAD)
Author: Russell Morris <rmorris@rkmorris.us>
Date:   Tue Jul 14 21:23:53 2020 -0500

    ramips: Add support for RAVPower WD03

    Signed-off-by: Russell Morris <rmorris@rkmorris.us>

commit 8f237444520a795dba410c830ec42d315f39811e (arrmo/hootoo-tm05, hootoo-tm05)
Author: Russell Morris <rmorris@rkmorris.us>
Date:   Tue Dec 24 18:38:36 2019 -0600

    ramips: Add support for HooToo TM05

    The HooToo TM05 is a battery powered router, with an Ethernet and USB port.
    Vendor U-Boot limited to 1.5 MB kernel size, so use lzma loader (loader-okli).
    MAC assignment based on vendor firmware,
    2.4 GHz, *:b4 (factory 0x04)
    LAN/label, *:b4 (factory 0x28)
    WAN, *:b5 (factory 0x2e)

    Specifications:
    SOC:     MediaTek MT7620N
    .
    .
    .