RAVPower WD-03, Add LZMA Loader

[xabolcs]

@arrmo

Thoughts? Really wondering about that 0x7b0000 ... leaving some room at the top end?

The -03 build has this room too at it's fimware partition.

@arrmo

I tried shortening the ...

And what do you plan to achieve with that shortenings?

@arrmo

Stepping back (and thinking out loud ... careful ) => where is the info actually stored?

After doing sysupgrade just boot the initrams.bin through serial, and examine the flash!
I don't know how. Try to mount it, try to hexdump it! I really don't know.

[arrmo]

@xabolcs

And what do you plan to achieve with that shortenings?

Was thinking that we were overwriting the information stored by the backup portion of sysupgrade. But no joy.

Something amiss here ... why the heck are we losing the backup information?!?! Not sure where it's getting stored . Arrgh!

@xabolcs

After doing sysupgrade just boot the initrams.bin through serial

Will try that!

[arrmo]

@xabolcs

After doing sysupgrade just boot the initrams.bin through serial

OK, here is what I did ...

1. hexdump on my saved configuration (file), from a day or so ago. It makes sense, header matches to what is captured at https://en.wikipedia.org/wiki/Gzip (see the first 10 bytes),

00000000  1f 8b 08 00 00 00 00 00  02 03 ec 3d 09 94 1b e5  |...........=....|

2. executed sysupgrade, but on reboot ... stopped it, booted initramfs. I see that rootfs_data is mtd10, so then I went hunting for the same header,

root@OpenWrt:/tmp# hexdump -C /dev/mtd10ro | grep -A 2 "1f 8b 08 00 00 00 00 00"
00005080  c6 65 ef 54 9e 3e ab e1  1f 8b 08 00 00 00 00 00  |.e.T.>..........|
00005090  02 03 ec 3d 09 98 1b d5  79 eb 24 5c 4b 09 a5 21  |...=....y.$\K..!|

So it is there! And the probability of hitting the exact same 8 bytes seems very low to me . But it's not getting retreived somehow? Again, not quite sure how this process works, and if it should find it (and where it looks)

OK, tried one more thing ... then rebooted, let it carry on like normal, as if I hadn't interrupted it. OK, this is interesting ... but now, rootfs_data (mtd11, given mtd-concat) => the stored gz is not found! So it's getting trashed on boot somehow it seems? I even searched the rootfs (mtd10) ... nope.

[arrmo]

A bit more info ... sysupgrade again, but block boot => initramfs, poking around . I do see, on mtd10ro,

root@OpenWrt:/mnt# head -n 2 /dev/mtd10ro | hexdump -C
00000000  de ad c0 de ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
00000010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00005000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 85 19 01 e0  |... ............|
00005010  36 00 00 00 a4 e1 55 df  01 00 00 00 00 00 00 00  |6.....U.........|
00005020  02 00 00 00 00 00 00 00  0e 08 00 00 52 83 90 67  |............R..g|
00005030  bc 3d ff 31 73 79 73 75  70 67 72 61 64 65 2e 74  |.=.1sysupgrade.t|
00005040  67 7a ff ff 85 19 02 e0  bc 0f 00 00 68 82 79 d0  |gz..........h.y.|

sysupgrade.tgz => pretty sure the file is there! Not having any luck mounting this partition - but that's me, hunting to figure it out. That aside, I did see,

root@OpenWrt:/tmp# firstboot switch2jffs
This will erase all settings and remove any installed packages. Are you sure? [N/y]
y
/dev/mtdblock10 is not mounted
/dev/mtdblock10 will be erased on next mount

So somehow it is set to erase on reboot? We want it saved, right?

Thanks!

[xabolcs]

@arrmo

So somehow it is set to erase on reboot?

Agreed!

I still don't have any hint!
But poking around within initramfs is a good thing!

@arrmo

We want it saved, right?

I don't know how the sysupgrade works.

[arrmo]

@xabolcs

I don't know how the sysupgrade works.

LOL - me neither. Blind leading the blind ... . Trying something - failsafe, see what mount_root says there! Rather odd result, after clean sysupgrade, failsafe ...

mount_root: no usable overlay filesystem found, using tmpfs overlay

Really seems like rootfs_data is broken somehow. But I may be wrong!

[xabolcs]

In case you are stucked with the sysupgrade stuff ... :

• does this sysupgrade bug affect the HooToo PR#2465?
• that full flash erase bug was a one time bug, or it's reproducible
  • do you see someting in the HooToo u-boot source code?

[arrmo]

@xabolcs

• does this sysupgrade bug affect the HooToo PR#2465?

I have a bad feeling that it affects more than just this - seems to happen even without the HooToo changes (at least the mtd-concat / recent ones).

@xabolcs

that full flash erase bug was a one time bug, or it's reproducible

Really seems to be a one time thing, can't make it happen again.

[xabolcs]

Look!

[Solved] Ramips jffs2 sysupgrade to 18.06.1 losing config

I have a ramips mt7620a -based device that is losing it's sysupgrade.tgz config when upgraded from LEDE 17.01 series to 18.06.1. There seems to be some incompatibility when doing the 'zcat sysupgrade.bin | mtd write -j /tmp/sysupgrade.tgz - firmware', where it integrates the sysupgrade.tgz into the jffs2 filesystem. This is a custom device based on the IAR-142-G: https://oringnet.com/en-global/products/detail/P0000000512/91/?IAR-142-4G_US There is plenty of free flash space (writing to a 24MB …

And here is another:

Sysupgrade source

[arrmo]

@xabolcs

Look!

I don't think think this is it, as it's between versions. But I did see that the driver has changed (see the log above). So perhaps!

@xabolcs

And here is another:

Yes, and this is very interesting,

Overlay FS support (how to enable overlay for new device)

df -Th Filesystem Type Size Used Available Use% Mounted on /dev/root squashfs 2.0M 2.0M 0 100% /rom tmpfs tmpfs 123.6M 68.0K 123.6M 0% /tmp tmpfs tmpfs 123.6M 40.0K 123.6M 0% /tmp/root overlayfs:/tmp/root overlay 123.6M 40.0K 123.6M 0% / tmpfs tmpfs 512.0K 0 512.0K 0% /dev block info /dev/sdb1: UUID="4f31ca0e-1154-4f82-a032…

I say that because after a clean boot (not sysupgrade), at the start of rootfs => 0x1985, as expected!

00000000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 ff ff ff ff  |... ............|

But, in spite of what the link says (about out of date), I did see (in failsafe),

The JFFS2 mounting code expects a special marker sequence 0xdeadc0de at the beginning of the rootfs_data partition, if this byte sequence is found; the entire rootfs_data partition is erased and reformatted as JFFS2.

So it's saving the file, to the correct place, but telling it to erase!

[arrmo]

I think I see the issue. If I go to failsafe mode, and look at rootfs_data,

root@(none):/# hexdump -C /dev/mtd11 | head -8
00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................|
*
00005000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 85 19 01 e0  |... ............|
00005010  36 00 00 00 a4 e1 55 df  01 00 00 00 00 00 00 00  |6.....U.........|
00005020  02 00 00 00 00 00 00 00  0e 08 00 00 52 83 90 67  |............R..g|
00005030  bc 3d ff 31 73 79 73 75  70 67 72 61 64 65 2e 74  |.=.1sysupgrade.t|
00005040  67 7a ff ff 85 19 02 e0  bc 0f 00 00 68 82 79 d0  |gz..........h.y.|

The needed header and files are there, but why the 0x5000 offset? mount_root fails, as below ... but that sort of makes sense,

root@(none):/# mount_root
[   57.691062] mount_root: no usable overlay filesystem found, using tmpfs overlay

[arrmo]

FIXED! I can't 100% explain it . But looking at the recipes, and the information here,

Overlay FS support (how to enable overlay for new device)

Hi All, The expected marker sequence 0x1985 (JFFS2) is not found in /dev/mtdblock3. So, Overlay is mounted in /tmp/root. Hence all configurations loaded in overlay are lost in next reboot. As suggested by @jow, I tried to add following line in Makefile. $(call prepare_generic_squashfs,$(KDIR)/root.$(1)) # pad to 4k, 8k, 16k, 64k, 128k, 256k and add jffs2 end-of-filesystem mark define prepare_generic_squashfs $(STAGING_DIR_HOST)/bin/padjffs2 $(1) 4 8 16 64 128 256 endef This invokes "padjff…

I started thinking about the BLOCKSIZE ... it wasn't there before. So I tried removing it ... and now, if I look at the rootfs_data partition,

root@travelRouter:/# hexdump -C /dev/mtd11 | head -8
00000000  85 19 03 20 0c 00 00 00  b1 b0 1e e4 85 19 01 e0  |... ............|
00000010  36 00 00 00 a4 e1 55 df  01 00 00 00 00 00 00 00  |6.....U.........|
00000020  02 00 00 00 00 00 00 00  0e 08 00 00 52 83 90 67  |............R..g|
00000030  bc 3d ff 31 73 79 73 75  70 67 72 61 64 65 2e 74  |.=.1sysupgrade.t|
00000040  67 7a ff ff 85 19 02 e0  bc 0f 00 00 68 82 79 d0  |gz..........h.y.|
00000050  02 00 00 00 02 00 00 00  80 81 00 00 00 00 00 00  |................|
00000060  a2 1a 00 00 99 52 12 5f  99 52 12 5f 99 52 12 5f  |.....R._.R._.R._|
00000070  00 00 00 00 78 0f 00 00  78 0f 00 00 00 00 00 00  |....x...x.......|

It's aligned (to 0x0), like it should be! And ... sysupgrade doesn't trash my info, it is maintained! Also, in the log now,

[    7.213561] jffs2_scan_eraseblock(): End of filesystem marker found at 0x2000
[    7.228071] jffs2_build_filesystem(): unlocking the mtd device...
[    7.228149] done.
[    7.244316] jffs2_build_filesystem(): erasing all blocks after the end marker...
[   47.686327] done.
[   47.705140] jffs2: notice: (449) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
[   47.737432] mount_root: overlay filesystem has not been fully initialized yet
[   47.757073] mount_root: switching to jffs2 overlay
[   47.796909] overlayfs: upper fs does not support tmpfile.

As it should be! So all good - sigh. Time for a ... LOL.

When I push this, was there a name change needed also?

Thanks!

And PS, I'm not so sure this isn't a sysupgrade bug. Thinking about it ... it says to me that the padding used to determine the start of rootfs_data is not consistent between writing (storing backup data), and reading .. agreed?

[mpratt14]

@arrmo

root@OpenWrt:/tmp# firstboot switch2jffs
This will erase all settings and remove any installed packages. Are you sure? [N/y]
y
/dev/mtdblock10 is not mounted
/dev/mtdblock10 will be erased on next mount

So somehow it is set to erase on reboot? We want it saved, right?

This is because it asked if you want to erase jffs2 and you said yes lol

I did some digging on this...

switch2jffs doesn't exist anymore, not as a parameter or a hook or script or anything...
the mount_root program handles all those features now (through libfstools)

Here is the firstboot script from 10 years ago

openwrt/openwrt/blob/7af8ee04812f647ce74adbe04f0d5d18a39bbd29/package/base-files/files/sbin/firstboot

---

---

compared to today

openwrt/openwrt/blob/master/package/base-files/files/sbin/firstboot

Now it literally just calls jffs2reset which takes no parameters, however has two actions

• if /overlay is mounted, erases files
• if /overlay is not mounted, marks the partition for deletion on next mount

I am not good at reading C code but here it is
https://git.openwrt.org/?p=project/fstools.git;a=blob;f=jffs2reset.c

if you want to mount the jffs2 partition from initramfs boot you should do something like
mount /dev/mtd10 /overlay
or
mount_root

[mpratt14]

@arrmo

the padding used to determine the start of rootfs_data is not consistent between writing (storing backup data), and reading

There is a separate image command pad-rootfs for a reason...but never sure what that reason was...

@arrmo

sysupgrade doesn't trash my info, it is maintained!

do you see the line in kernel log that indicates that sysupgrade.tgz was unpackaged?
- config restore -

[xabolcs]

@arrmo

I started thinking about the BLOCKSIZE ... it wasn't there before. So I tried removing it ...

Awesome!
I asked about BLOCKSIZE at GiHub, because it looked as a leftover, but looks like my comment / review get lost.

@arrmo

And PS, I'm not so sure this isn't a sysupgrade bug. Thinking about it ... it says to me that the padding used to determine the start of rootfs_data is not consistent between writing (storing backup data), and reading .. agreed?

It is interesting, at least!
Dig yourself into the sysupgrade code, and try to find the culcprit code and it's history!

[arrmo]

@mpratt14

There is a separate image command pad-rootfs for a reason...but never sure what that reason was...

Agreed! And that's where it clued me in to BLOCKSIZE

@mpratt14

do you see the line in kernel log that indicates that sysupgrade.tgz was unpackaged?

Nope ... but I do see the files restored (checked them).

@xabolcs

Dig yourself into the sysupgrade code, and try to find the culcprit code and it's history!

I am! As a side interest of course ... . Seems like the write is after that padding, but read after is not (or at least, they are different).

OK, all working again, just the recipes to clean up on HooToo. Then .. disassemble, reflash RAVPower => combine the dts files! Agreed?

[xabolcs]

@arrmo

Then .. disassemble, reflash RAVPower => combine the dts files! Agreed?

If you have all the original functions of RavPower working: LEDs, buttons, SD card, etc ... then you could start combining the DTS'.

[arrmo]

@xabolcs

If you have all the original functions of RavPower working: LEDs, buttons, SD card, etc ... then you could start combining the DTS'.

Exactly! Let me disassemble, OEM flash - confirm my build, go from there. Thanks!

[xabolcs]

@arrmo

... OEM flash ...

Do you have it's 8 MByte backup.bin? For the MAC addresses, calibration datas and others.

[arrmo]

@xabolcs

Do you have it's 8 MByte backup.bin? For the MAC addresses, calibration datas and others.

You are kidding, right? It was next on my list, never got to it before the blow up. But will get it restored and working, NP. Just need to crack the case open again - dang it!