Wrt32x / snapshot sysupgrade - config not saved

Installing and Using OpenWrt
1

i've been using firmware selector to add modules (nlbwmon, luci-statistics, luci-attended-sysupgrade, luci-app-advancedd-reboot, auc, luci-app-sqm) to snapshot builds. somewhere after r22115, it looks like the default ssl changed form wolfssl to mbedtls (so the corresponding packages are now wpad-mbedtls px5g-mbedtls libuhttpd-mbedtls px5g-mbedtls) and since then my configs do not survive ssyupgrade - all settings are lost. other than the change from wolfssl to mbedtls , i'm not sure what else has changed. are others seeing this?

3

update -
i have confirmed that settings are reset with sysupgrade on both wrt32x and wrt3200acm snapshots. i cannot pinpoint the exact build that broke this, but i susupect it was with transtion from wolfssl to mbedtls libraries. ?
i loaded openwrt 23.02 on both devices and configs are saved across sysupgrade.with the same customization except wpad-wolfssl px5g-wolfssl
i suppose the mbedtls builds are stillbeing optimized

  • maybe devs note?
4

The SSL library transition should have no relation to sysupgrade. The settings backup is a simple tar.gz file in the router, and no encryption. (And no Https, SSL at all)

5

@hnyman - thank you, what about root password and dropbear? could htere be a chang ein shut diwn/startup sequence? i'm really at a loss on thiis. the only things i before saving was att luci, auc and advanced reboot to factory. the device always reboots with all setting lost, asking to set password again.

6

Another data point, for about the same time frame I have been seeing this on a mamba, but the same image flashed to a rango carries config fine. I exclude all mbedtls and wolfssl in favour of openssl in my build. I thought it might be tied to issue, but doubtful.

1 Like
7

@anomeome - i recall a post about this in mamaba now, - i think reported by someone else ?
to be clear - if i backup configuration prioir to sysupgrade, i can restore it (minus a few items like nlbwmon and collectd rrd data which must be more tediously restored). so it seems like something about the sequencing with the sysupgrade process. it will be about a week before i can get this on the serial interface to watch the upgrade process.
thank you...

8

Not related to wolfssl or mbedtls, but if you have openssl installed, there could be a sysupgrade problem for some devices with hardware crypto...

See mitigations in

9

interesting. my buildinfo showed libopenssl being installed. i'm not sure what called it.. i'm using the online firmware-selector.openwrt.org and will wathc with more recent snapshots.
thx

10

That's the resolution to the issue I linked above and as I said I am now doubtful that the two are related, at least did not help with mamba config.

11

One thing that you might check, is that the /tmp/syscfg is there.

mvebu dual-firmware routers use a unique method for storing the backup during sysupgrade: it is written in the permanently mounted /tmp/syscfg that is actually a rather large "syscfg" partition unused by OpenWrt, but which it used for OEM config etc. by the OEM firmware. It is used by OpenWrt for storing the config file during backup, as that partition is not duplicated and can be accessed from both firmwares.

If that does not get mounted (due to some DTS change or fstools changes or...), it could well be unnoticed by most users as the sysupgrade script probably does not really handle errors regarding that exceptional backup strategy.

The backup restore part (containing also error message about damaged partition):

backup storing:

12

Yep, empty on my mamba and extent on the rango. Not sure when or why it went walkabout.

Edit: on mamba not seeing message about damaged partition:

echo "ubifs syscfg partition is damaged, reformatting"

but I do not recall seeing these in the past:

[   11.563658] mtdblock: MTD device 'devinfo' is NAND, please consider using UBI block devices instead.
[   12.227981] mtdblock: MTD device 'devinfo' is NAND, please consider using UBI block devices instead.
13

That is normal.
It came with kernel 5.15, if I remember right.
Upstream Linux started to "helpfully" warn about non-UBI FS on NAND flash (also for wifi calibration etc. read-only partitions)

But you might be interested in

EDIT:
Regarding (the non-related) UBI NAND messages, I found the old message from bmork that he sent to kernel.org mailing list about those NAND warnings after the issue had risen here in the OpenWrt, but I can't find the original old local discussion.

EDIT2:
Surprisingly, the discussion was newer, and in the LuCI repo...

1 Like
14

On rango dmesg:

[    8.106096] ubi1: attached mtd9 (name "syscfg", size 86 MiB)
[    8.228609] UBIFS (ubi1:0): UBIFS: mounted UBI device 1, volume 0, name "syscfg"

mamba dmesg:

[    7.838078] ubi1: attached mtd8 (name "syscfg", size 38 MiB)

does not appear to try to mount, no message as to failure. But this seems wrong:

root@OpenWrt:/tmp/syscfg# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00100000 00020000 "u-boot"
mtd1: 00040000 00020000 "u_env"
mtd2: 00040000 00020000 "s_env"
mtd3: 00100000 00020000 "devinfo"
mtd4: 02800000 00020000 "kernel1"
mtd5: 02400000 00020000 "rootfs1"
mtd6: 02800000 00020000 "kernel2"
mtd7: 02400000 00020000 "ubi"
mtd8: 02600000 00020000 "syscfg"
mtd9: 00780000 00020000 "unused_area"
mtd10: 00008000 00008000 "spi0.0"

only recall there being 10 partitions.

Edit: guess not, old post, wiki is wrong as well.

15

to clarify: what is the mtd map supposed to be?
mine shows this on 22.02 (on which sysupgrade 'works' to preserve settings):

root:~# cat /proc/mtd             
dev:    size   erasesize  name             
mtd0: 00200000 00020000 "u-boot"           
mtd1: 00020000 00020000 "u_env"            
mtd2: 00040000 00020000 "s_env"            
mtd3: 005c0000 00020000 "unused_area"      
mtd4: 00040000 00020000 "devinfo"          
mtd5: 07b00000 00020000 "kernel1"          
mtd6: 07500000 00020000 "ubi"              
mtd7: 07b00000 00020000 "kernel2"          
mtd8: 07500000 00020000 "rootfs2"          
mtd9: 00100000 00020000 "BBT"              
root:~#
16

i'm also not sure whwere we are on fixing this issue that some are having,
my /tmp/syscfg shows as an empty directory with no alias.

if it is related to this:
"The restore is delayed until next boot on these"

then i'm wondering if the /tmp/syscfg mapping is missing in snapshots?

17

updates:
i was hopeful that the recent update to wpad-mbedtls would somehow fix this issue, but no.

i had 22.02 on both partitions, and used the web gui to sysupgrade to snapshot r22248.
on first sysupgrade (from 22.02): new image r22248 flashed suscessfully and confis were successfully transferred/retained.
on second sysupgrade (from r22248 to r22248 on other partition: config not save, all lost.
this is in dmesg:

 6.637619] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
[    6.734977] UBIFS (ubi0:1): default file-system created
[    6.740699] UBIFS (ubi0:1): Mounting in unauthenticated mode
[    6.752169] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 678
[    6.791488] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
[    6.799362] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[    6.809333] UBIFS (ubi0:1): FS size: 103485440 bytes (98 MiB, 815 LEBs), max 826 LEBs, journal size 5206016 bytes (4 MiB, 41 LEBs)
[    6.821130] UBIFS (ubi0:1): reserved for root: 4887873 bytes (4773 KiB)
[    6.827776] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID 536CA7EA-3FB9-4315-85BB-FBF8390D5015, small LPT model
[    6.839680] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
[    6.846747] block: unable to load configuration (fstab: Entry not found)
[    6.853514] block: attempting to load /tmp/ubifs_cfg/etc/config/fstab
[    6.860019] block: unable to load configuration (fstab: Entry not found)
[    6.866778] block: attempting to load /etc/config/fstab
[    6.874552] block: unable to load configuration (fstab: Entry not found)
[    6.881312] block: no usable configuration
[    6.887620] UBIFS (ubi0:1): un-mount UBI device 0
[    6.892363] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" stops
[    6.900540] UBIFS (ubi0:1): Mounting in unauthenticated mode
[    6.912178] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 681
[    6.950863] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
[    6.958754] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[    6.968719] UBIFS (ubi0:1): FS size: 103485440 bytes (98 MiB, 815 LEBs), max 826 LEBs, journal size 5206016 bytes (4 MiB, 41 LEBs)
[    6.980515] UBIFS (ubi0:1): reserved for root: 4887873 bytes (4773 KiB)
[    6.987160] UBIFS (ubi0:1): media format: w5/r0 (latest is w5/r0), UUID 536CA7EA-3FB9-4315-85BB-FBF8390D5015, small LPT model
[    7.010491] block: attempting to load /tmp/ubifs_cfg/upper/etc/config/fstab
[    7.017565] block: unable to load configuration (fstab: Entry not found)
[    7.024335] block: attempting to load /tmp/ubifs_cfg/etc/config/fstab
[    7.030845] block: unable to load configuration (fstab: Entry not found)
[    7.037668] block: attempting to load /etc/config/fstab
[    7.042966] block: unable to load configuration (fstab: Entry not found)
[    7.049729] block: no usable configuration
[    7.054586] mount_root: overlay filesystem has not been fully initialized yet
[    7.061885] mount_root: switching to ubifs overlay
[    7.115601] urandom-seed: Seed file not found (/etc/urandom.seed)
[    7.156082] procd: - early -
[    7.160390] procd: - watchdog -

i'm in circles on this.

18

Like I said earlier, the SSL lib has no role in the backup process inside the router. there is no SSL involved in the process itself.

Looks like the syscfg mount phase is missing.

I have a log from 2017 from WRT3200ACM, and there it is clearly after the fstab phase:

Sat Dec 30 18:52:30 2017 user.info kernel: [   13.677396] block: extroot: not configured
Sat Dec 30 18:52:30 2017 user.info kernel: [   13.717797] mount_root: overlay filesystem has not been fully initialized yet
Sat Dec 30 18:52:30 2017 user.info kernel: [   13.725531] mount_root: switching to ubifs overlay
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   13.833213] ubi1: attaching mtd9
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.080248] ubi1: scanning is finished
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.089293] ubi1: attached mtd9 (name "syscfg", size 86 MiB)
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.094979] ubi1: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.101898] ubi1: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.108716] ubi1: VID header offset: 2048 (aligned 2048), data offset: 4096
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.115707] ubi1: good PEBs: 680, bad PEBs: 8, corrupted PEBs: 0
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.121747] ubi1: user volume: 1, internal volumes: 1, max. volumes count: 128
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.129002] ubi1: max/mean erase counter: 6/3, WL threshold: 4096, image sequence number: 1423740224
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.138174] ubi1: available PEBs: 0, total reserved PEBs: 680, PEBs reserved for bad PEB handling: 32
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.147447] ubi1: background thread "ubi_bgt1d" started, PID 788
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.161789] UBIFS (ubi1:0): background thread "ubifs_bgt1_0" started, PID 792
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.202477] UBIFS (ubi1:0): UBIFS: mounted UBI device 1, volume 0, name "syscfg"
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.209930] UBIFS (ubi1:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.219898] UBIFS (ubi1:0): FS size: 80375808 bytes (76 MiB, 633 LEBs), journal size 4063232 bytes (3 MiB, 32 LEBs)
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.230389] UBIFS (ubi1:0): reserved for root: 3796347 bytes (3707 KiB)
Sat Dec 30 18:52:30 2017 kern.notice kernel: [   14.237041] UBIFS (ubi1:0): media format: w4/r0 (latest is w4/r0), UUID 17C96137-2D34-4C4F-9454-B65E98F535EF, small LPT model
Sat Dec 30 18:52:30 2017 user.warn kernel: [   14.251070] urandom-seed: Seed file not found (/etc/urandom.seed)
Sat Dec 30 18:52:30 2017 user.info kernel: [   14.353585] procd: - early -
Sat Dec 30 18:52:30 2017 user.info kernel: [   14.356510] procd: - watchdog -

It is.
I linked in my earlier message in this thread the exact preinit script /lib/preinit/81_linksys_syscfg that is supposed to mount the syscfg and restore config. And note that if the mount does not happen, then at the next sysupgrade there is no mounted path to which the config is saved. (It gets saved to the normal /tmp in RAM)

Curiously, @anomeome has the syscfg partition visible quite ok in /prod/mtd, while @ghoffman has some "BBT" that sounds quite strange, but there is no "syscfg". If there is no "syscfg", it naturally can't be mounted and settings be saved...

19

apparently other models (like WRT3200ACM) do have the syscfg in DTS (e.g. https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm/boot/dts/armada-385-linksys-rango.dts?h=v6.2.2 ), but WRT32x does not.

For wrt32x it is probably that restore script to should create the syscfg at the first time.
For some reason that does not happen to you.

Looking at the wrt32x, I am not actually sure, where the syscfg is supposed to be?

What does you boot log say about found mtd partitions?

This is from wrt3200acm:

[    1.073674] 11 ofpart partitions found on MTD device pxa3xx_nand-0
[    1.079878] Creating 11 MTD partitions on "pxa3xx_nand-0":
[    1.085393] 0x000000000000-0x000000200000 : "u-boot"
[    1.090656] 0x000000200000-0x000000220000 : "u_env"
[    1.095805] 0x000000220000-0x000000260000 : "s_env"
[    1.100944] 0x0000007e0000-0x000000820000 : "devinfo"
[    1.106247] 0x000000820000-0x000000a00000 : "sysdiag"
[    1.111556] 0x000000a00000-0x000005a00000 : "kernel1"
[    1.117024] 0x000001000000-0x000005a00000 : "ubi"
[    1.122153] 0x000005a00000-0x00000aa00000 : "kernel2"
[    1.127609] 0x000006000000-0x00000aa00000 : "rootfs2"
[    1.133065] 0x00000aa00000-0x000010000000 : "syscfg"
[    1.138454] 0x000000260000-0x000000820000 : "unused_area"
[    1.144381] libphy: Fixed MDIO Bus: probeda"
20

this is the dmesg from 22.03.02, which will retain config:

1.292257] nand: Macronix MX30LF2G18AC
[    1.296109] nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
[    1.304071] Bad block table found at page 131008, version 0x01
[    1.310554] Bad block table found at page 130944, version 0x01
[    1.316842] 10 fixed-partitions partitions found on MTD device pxa3xx_nand-0
[    1.323935] Creating 10 MTD partitions on "pxa3xx_nand-0":
[    1.329445] 0x000000000000-0x000000200000 : "u-boot"
[    1.334636] 0x000000200000-0x000000220000 : "u_env"
[    1.339665] 0x000000220000-0x000000260000 : "s_env"
[    1.344713] 0x000000260000-0x000000820000 : "unused_area"
[    1.350273] 0x0000007e0000-0x000000820000 : "devinfo"
[    1.355480] 0x000000900000-0x000008400000 : "kernel1"
[    1.360853] 0x000000f00000-0x000008400000 : "ubi"
[    1.365852] 0x000008400000-0x00000ff00000 : "kernel2"
[    1.371206] 0x000008a00000-0x00000ff00000 : "rootfs2"
[    1.376577] 0x00000ff00000-0x000010000000 : "BBT"

and the mtd map from my device when running oem firmware from a few years ago shows the same map, with no syscfg:

dev:    size   erasesize  name
mtd0: 00200000 00020000 "u-boot"
mtd1: 00020000 00020000 "u_env"
mtd2: 00040000 00020000 "s_env"
mtd3: 005c0000 00020000 "unused_area"
mtd4: 00040000 00020000 "devinfo"
mtd5: 07b00000 00020000 "kernel1"
mtd6: 07800000 00020000 "rootfs1"
mtd7: 07b00000 00020000 "kernel2"
mtd8: 07800000 00020000 "ubi"
mtd9: 00100000 00020000 "BBT"

so the oem mtd map is being used in openwrt. retention of this map was discusseed a few years ago during the developmen tof openwrt for the wrt32x.

so the restore script must be different in 22.02.03 than in the more recent snapshots?
i'm saving the sysupgrade script an dwill reflash the snapshot build (which i swore to myself i was done with until someone else figured this out!) to compare sysupgrade and dmesg again.and will post..
thnak you for your help and patience.
is there a way to issue a uboot 'resetenv' from openwrt?
thanks!

thnak you for being patient.

21

Flashing mamba with 22.02 via serial / tftp to both partitions does not yield a working setup. Since ghoffman can go back to a stable image and things work it is possible there are two different issues here.