Flash memory architecture

When trying to install Samba on my Hnet HW-C108 travel router with OpenWRT 19.07.2 I unexpectedly ran out of space. It seems to me the device does not contain as much flash memory as claimed. I'm trying to understand the way the memory is used.

The device page claims the router comes with 16MB of flash memory. This is what I find in the kernel log at boot time:

[    0.611747] spi spi0.0: force spi mode3
[    0.625219] m25p80 spi0.0: s25fl064k (8192 Kbytes)
[    0.630159] 4 fixed-partitions partitions found on MTD device spi0.0
[    0.636667] Creating 4 MTD partitions on "spi0.0":
[    0.641583] 0x000000000000-0x000000030000 : "u-boot"
[    0.647561] 0x000000030000-0x000000040000 : "u-boot-env"
[    0.653878] 0x000000040000-0x000000050000 : "factory"
[    0.659822] 0x000000050000-0x000001000000 : "firmware"
[    0.665140] mtd: partition "firmware" extends beyond the end of device "spi0.0" -- size truncated to 0x7b0000
[    0.678996] 2 uimage-fw partitions found on MTD device firmware
[    0.685113] Creating 2 MTD partitions on "firmware":
[    0.690190] 0x000000000000-0x00000017df95 : "kernel"
[    0.696115] 0x00000017df95-0x0000007b0000 : "rootfs"
[    0.702024] mtd: device 5 (rootfs) set to be root filesystem
[    0.709353] 1 squashfs-split partitions found on MTD device rootfs
[    0.715749] 0x0000003e3000-0x0000007b0000 : "rootfs_data"

The sizes derived from the address ranges match the sizes of the MTDblock devices, with the peculiarity that the "firmware" partition seems to be sized for a 16MB device, but gets truncated according to a total 8MB device (note the kernel message):

   196608  mtdblock0   "u-boot"
    65536  mtdblock1   "u-boot-env"
    65536  mtdblock2   "factory"
  8060928  mtdblock3   "firmware"
  1564160  mtdblock4    |-> "kernel"
  6496256  mtdblock5    |-> "rootfs"
  3985408  mtdblock6         |->"rootfs_data"

The arrows denote a "comprises" relationship.

It seems there is only 4MB space rather than 12MB in the overlay filesystem ("rootfs_data"), which matches the output of df:

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 2560      2560         0 100% /rom
tmpfs                    30012       924     29088   3% /tmp
/dev/mtdblock6            3892      1800      2092  46% /overlay
overlayfs:/overlay        3892      1800      2092  46% /
tmpfs                      512         0       512   0% /dev
/dev/mmcblk0p1        62342144     60384  62281760   0% /storage

Is there a way I could have misinterpreted this? Should I correct the value on the device page?

The output of df shows I have plenty of space on my SD card. I'm thinking of utilising some of it to make room on the overlay filesystem. What's the smartest way of doing that? Right now it's an exfat filesystem. I have refrained from converting it to ext2/3/4, as the required kernel driver and required utilities are MASSIVE: together they get close to 1MB.

But there is only a single spi nor flash with 8mb detected.

I think there is a mistake in the wiki and in the initial device support commit message.

0x1000000 are also defined in the device tree if you sum up all partitions and this are exactly 8mb.

I beg to differ. In my world, 0x1000000 is 16MB.

You are absolutely right :shushing_face:

The firmware partition size in the current (trunk/master) device tree is correct!

I think you have to open up your device and determine the real flash chip type.

Are you saying this may be a device tree bug, and if the device indeed has 16MB flash memory, fixing it would give me another 8MB?

Now that would be a reason to open it :slight_smile:

PS.: In the meantime I checked the OpenWRT image that I flashed. It is indeed just 8MB in size.

No, the compatible "jedec,spi-nor" instructs the kernel to auto detect the spi nor flash according to the jedec standard.
As your flash chip is detected as "s25fl064k" instead of "en25q128" there are three reasons for your missing space.

  1. Your device has really this detected spi nor type
  2. The magic for your real flash chip does conflict with the one of "s25fl064k"
  3. The magic for "s25fl064k" are wrong

The image hasn´t to be the same size as the space on the flash chip!

I know. But it is one of many hints that the 16MB was some initial mistake that never got corrected. After all, 8MB is also more than the actual data inside.

This brings me to another question: The kernel provides device nodes to access the flash memory. There are (# for some decimal digits) /dev/mtd#, /dev/mtd#ro (both major number 90), and /dev/mtdblock# (major number 31), which for the same decimal digit (0 ... 6 in this device) refer to the same memory. The file 'device.txt' from the linux source describes very shortly the devices. The major number 90 ("Memory Technology Devices") is straightforward: mtd# is read/write, mtd#ro is read-only access. The description for major number 31 confuses me:

  31 block	ROM/flash memory card
		  0 = /dev/rom0		First ROM card (rw)
		      ...
		  7 = /dev/rom7		Eighth ROM card (rw)
		  8 = /dev/rrom0	First ROM card (ro)
		    ...
		 15 = /dev/rrom7	Eighth ROM card (ro)
		 16 = /dev/flash0	First flash memory card (rw)
		    ...
		 23 = /dev/flash7	Eighth flash memory card (rw)
		 24 = /dev/rflash0	First flash memory card (ro)
		    ...
		 31 = /dev/rflash7	Eighth flash memory card (ro)

		The read-write (rw) devices support back-caching
		written data in RAM, as well as writing to flash RAM
		devices.  The read-only devices (ro) support reading
		only.

Now, what is a ROM with read/write access?

I was looking for a tool that can backup the entire configuration, i.e. including software (installed packages). Basically what I want is a flash memory dump. Is there a special tool for backup/restore?

If not: Dumping via dd is easy, but the data may not be in a consistent state such that when restoring the router can boot again. For that I imagine I'd need to remount the overlay filesystem into ro mode. Is that sufficient for what I have in mind? Then, before restoring, I'd do the same (remount ro), then dd to the rw device, and reboot. Are there any obvious roadblocks to this approach?

Probably this is something for you:
https://openwrt.org/docs/guide-user/installation/generic.backup#generic_nor_backup

LuCI does also allow you to download tbe mtd partition contents.
If im right under System -> Backup / Restore

From my point of view yes.

Thanks, looks like exactly what I was looking for. I will study it carefully.

Aha, in fact I had not noticed it before. However, it says:

NOTE: THIS FEATURE IS FOR PROFESSIONALS!

I take that to mean that there are caveats involved that the user needs to be aware of, and the user should know exactly what LuCI is doing there in the background. Do you know of any docs related to this? An immediate question is, whether I get a pure image, or one that is tagged with info which block is backed up? I can check the size of course. But there is no specific option to restore an image, but only the sysupgrade option. How do I make a backup image sysupgrade-compatible (as the GUI calls it), such that sysupgrade knows which mtdblock to restore?

Maybe those questions are addressed in the document that you pointed me to. I will read that first.

So these backup scripts do not remount the overlay ro. Seems they rely on the fact that all normal system activity happens in RAM filesystems, and only the user would initiate write activities (via LuCI/uCI, or calling opkg). Still the filesystem is mounted rw, and most filesystems modify it anyway even if nothing writes to it, if for simply flagging it for non-proper unmounting after power failure.

Also the scripts assume nor flash. In an earlier message you called my device's flash nor-flash as well:

How can you be sure? Is that coded in the line

[ 0.625219] m25p80 spi0.0: s25fl064k (8192 Kbytes)

?