Sysupgrade resets configs changed after flash to default

Nihilokrat · September 16, 2024, 1:33pm

Hi, I have been noticing strange behaviour with sysupgrades on my devices since switching to new Kernel versions.

The device in question is the Archer AX23, but I also tested it on an EAP615 Wall v1. This has happened since building from snapshots with Kernel 6.X, as opposed to 5.X.

Expected behaviour:

Modified configs wireless, network, system are kept as is
Custom config files A & B in /etc/config/ are kept as is

Actual behaviour:

Configs wireless, network and system are reset to default
Custom config file A is kept, B is gone

Additional info:

I build the images locally, with one additional package outside of OpenWrt
The contents of this package are kept
After flashing the image, I copy custom config files to the device and then run scripts that customise the aforementioned files

Observation:

Files copied after initial flash are lost when sysupgrading
Customizations made by scripts are gone

Any idea how to troubleshoot this? Sysupgrade -v gave the usual output, the config files for sysupgrade contain the same info as before.

The contents of the additional files, as well as the scripts have not been changed at all between snapshots.

hnyman · September 16, 2024, 1:56pm

If it is included in the firmware image, naturally it is kept.

I get lost with this statement about a second flash that somehow customises something inside the device...
Secondary flash?
What actually should happen here?

How do you think that your workflow goes?
What exactly should be included in the image in the second flash? (and how?)

Note that the kept settings are not stored into any image by the router. They are usually packages as a archive file that is temporarily stored at the end of the flashed image, but outside any file system, and found and restored by the early first boot (and then the archive gets overwritten by the overlayfs area).

To keep the settings, the expected process is to sysupgrade with "keep settings".

Nihilokrat · September 16, 2024, 2:13pm

Sorry for the confusion: "secondary flash" as in running scripts that edit the configs via uci, changing hostname and root password. All edits done by those scripts are lost, as well as all files copied to the device after flashing the image (e.g.: File B in /etc/config/)

I ran sysupgrade -i last time to make sure which files are kept. Didn't make any difference though. The output of sysupgrade -v did not change between images, too.

brada4 · September 16, 2024, 2:35pm

You can save your customizing script in /etc/uci-defaults and get away with single flashing.

Nihilokrat · September 16, 2024, 2:53pm

Not possible unfortunately, since the scripts are partially dependend on the files copied to the device after flashing, which themselves contain unique elements.

The whole process has been working just fine before the Kernel version was changed, too, so I am reluctant to pinpoint this on the way I tackle the customizations per se.

I can also rule out fault specific to one device, since I tested this on two AX23v1 and one EAP615. I can also rule out issues with one specific snapshot since I build 3 or 4 different images off of different snapshots, all sysupgrades exerting the same behaviour.

I check the Git for relevant changes to sysupgrade but didn't find any. There have been changes after the last working image I sysupgraded with (build on January 24th), but they seem not relevant.

hnyman · September 16, 2024, 3:31pm

Sounds almost like your overlayfs is on ramdisk, so that all settings are lost at reboot.

Might just be that your firmware size has grown so much that there is no free space on flash, so that overlay gets initialized to ramdisk.

What does system log says about "overlay"? and what does "df -h" command say?

Nihilokrat · September 16, 2024, 3:57pm

On the device that has had the issue after sysupgrade:

root@OpenWrt:~# dmesg | grep overlay
[   30.957891] mount_root: overlay filesystem has not been fully initialized yet
[   30.967371] mount_root: switching to jffs2 overlay
[   30.974543] overlayfs: upper fs does not support tmpfile.
root@OpenWrt:~# free
              total        used        free      shared  buff/cache   available
Mem:         119040       61368       27172         260       30500       17720
Swap:             0           0           0
root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.8M      5.8M         0 100% /rom
tmpfs                    58.1M    260.0K     57.9M   0% /tmp
/dev/mtdblock4            6.6M    364.0K      6.3M   5% /overlay
overlayfs:/overlay        6.6M    364.0K      6.3M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev
root@OpenWrt:~#

On a device with the last working version:

root@123456:~# free
              total        used        free      shared  buff/cache   available
Mem:         120204       59832       38376         380       21996       24608
Swap:             0           0           0
root@123456:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.5M      5.5M         0 100% /rom
tmpfs                    58.7M    380.0K     58.3M   1% /tmp
/dev/mtdblock4            7.3M    368.0K      6.9M   5% /overlay
overlayfs:/overlay        7.3M    368.0K      6.9M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev
root@123456:~# dmesg | grep overlay
[    7.099277] mount_root: switching to jffs2 overlay
[    7.112014] overlayfs: upper fs does not support tmpfile.

I'll try again in a bit with a stripped down version and will try with a build from stable 23.05.03 or .04, since they were introduced after my last working snapshot build from 24th of January, if the stripped version offers the same results.

Overall size of the sysupgrade files:
8520792 size of working image (January 24th)
8717400 size of one of the non working images (July 25th)

Full dmesg output

dmesg
[    0.000000] Linux version 6.6.51 (nihilokrat@build) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 13.3.0 r26990-06b37a5856) 13.3.0, GNU ld (GNU Binutils) 2.42) #0 SMP Mon Sep 16 08:33:28 2024
[    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
[    0.000000] printk: bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
[    0.000000] MIPS: machine is TP-Link Archer AX23 v1
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] VPE topology {2,2} total 4
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.000000] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] percpu: Embedded 12 pages/cpu s18560 r8192 d22400 u49152
[    0.000000] pcpu-alloc: s18560 r8192 d22400 u49152 alloc=12*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Kernel command line: console=ttyS0,115200 rootfstype=squashfs,jffs2
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Writing ErrCtl register=00056587
[    0.000000] Readback ErrCtl register=00056587
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32480
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 117812K/131072K available (7944K kernel code, 632K rwdata, 1704K rodata, 1228K init, 218K bss, 13260K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 256
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] clocksource: GIC: mask: 0xffffffffffffffff max_cycles: 0xcaf478abb4, max_idle_ns: 440795247997 ns
[    0.000004] sched_clock: 64 bits at 880MHz, resolution 1ns, wraps every 4398046511103ns
[    0.008057] Calibrating delay loop... 586.13 BogoMIPS (lpj=2930688)
[    0.066227] pid_max: default: 32768 minimum: 301
[    0.079717] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.086958] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.103604] RCU Tasks Trace: Setting shift to 2 and lim to 1 rcu_task_cb_adjust=1.
[    0.111792] rcu: Hierarchical SRCU implementation.
[    0.116495] rcu: 	Max phase no-delay instances is 1000.
[    0.122575] smp: Bringing up secondary CPUs ...
[    0.127947] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.127975] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.127991] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.128040] CPU1 revision is: 0001992f (MIPS 1004Kc)
[    0.182024] Synchronize counters for CPU 1: done.
[    0.214679] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.214703] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.214716] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.214745] CPU2 revision is: 0001992f (MIPS 1004Kc)
[    0.273647] Synchronize counters for CPU 2: done.
[    0.304348] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.304371] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.304384] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.304417] CPU3 revision is: 0001992f (MIPS 1004Kc)
[    0.358842] Synchronize counters for CPU 3: done.
[    0.388710] smp: Brought up 1 node, 4 CPUs
[    0.398985] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.408797] futex hash table entries: 1024 (order: 3, 32768 bytes, linear)
[    0.419618] pinctrl core: initialized pinctrl subsystem
[    0.427658] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.434322] thermal_sys: Registered thermal governor 'step_wise'
[    0.440472] FPU Affinity set after 11720 emulations
[    0.458764] clocksource: Switched to clocksource GIC
[    0.473698] NET: Registered PF_INET protocol family
[    0.478816] IP idents hash table entries: 2048 (order: 2, 16384 bytes, linear)
[    0.486783] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.495085] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.502762] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.510370] TCP bind hash table entries: 1024 (order: 2, 16384 bytes, linear)
[    0.517481] TCP: Hash tables configured (established 1024 bind 1024)
[    0.523944] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.530441] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
[    0.538341] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    0.544023] PCI: CLS 0 bytes, default 32
[    0.551653] workingset: timestamp_bits=14 max_order=15 bucket_order=1
[    0.559947] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.565691] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.582014] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    0.588093] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    0.594192] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    0.600396] mt7621-pci 1e140000.pcie: host bridge /pcie@1e140000 ranges:
[    0.607061] mt7621-pci 1e140000.pcie:   No bus range found for /pcie@1e140000, using [bus 00-ff]
[    0.615868] mt7621-pci 1e140000.pcie:      MEM 0x0060000000..0x006fffffff -> 0x0060000000
[    0.623985] mt7621-pci 1e140000.pcie:       IO 0x001e160000..0x001e16ffff -> 0x0000000000
[    0.988758] mt7621-pci 1e140000.pcie: pcie2 no card, disable it (RST & CLK)
[    0.995663] mt7621-pci 1e140000.pcie: PCIE0 enabled
[    1.000543] mt7621-pci 1e140000.pcie: PCIE1 enabled
[    1.005521] PCI coherence region base: 0x60000000, mask/settings: 0xf0000002
[    1.012744] mt7621-pci 1e140000.pcie: PCI host bridge to bus 0000:00
[    1.019056] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.024462] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
[    1.031315] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    1.037475] pci 0000:00:00.0: [0e8d:0801] type 01 class 0x060400
[    1.043421] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[    1.049648] pci 0000:00:00.0: reg 0x14: [mem 0x60600000-0x6060ffff]
[    1.055896] pci 0000:00:00.0: supports D1
[    1.059833] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.066215] pci 0000:00:01.0: [0e8d:0801] type 01 class 0x060400
[    1.072235] pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[    1.078426] pci 0000:00:01.0: reg 0x14: [mem 0x60610000-0x6061ffff]
[    1.084741] pci 0000:00:01.0: supports D1
[    1.088658] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
[    1.096727] pci 0000:01:00.0: [14c3:7916] type 00 class 0x000280
[    1.102767] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit pref]
[    1.109955] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00003fff 64bit pref]
[    1.117089] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit pref]
[    1.124401] pci 0000:01:00.0: supports D1 D2
[    1.128578] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    1.135209] pci 0000:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:00.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[    1.150284] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    1.155441] pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
[    1.161520] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    1.168215] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x602fffff pref]
[    1.175466] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.182328] pci 0000:02:00.0: [14c3:7915] type 00 class 0x000280
[    1.188295] pci 0000:02:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit pref]
[    1.195498] pci 0000:02:00.0: reg 0x18: [mem 0x00000000-0x00003fff 64bit pref]
[    1.202674] pci 0000:02:00.0: reg 0x20: [mem 0x00000000-0x00000fff 64bit pref]
[    1.209968] pci 0000:02:00.0: supports D1 D2
[    1.214146] pci 0000:02:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    1.220787] pci 0000:02:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0000:00:01.0 (capable of 4.000 Gb/s with 5.0 GT/s PCIe x1 link)
[    1.235938] pci 0000:00:01.0: PCI bridge to [bus 02-ff]
[    1.241152] pci 0000:00:01.0:   bridge window [io  0x0000-0x0fff]
[    1.247170] pci 0000:00:01.0:   bridge window [mem 0x60300000-0x603fffff]
[    1.253939] pci 0000:00:01.0:   bridge window [mem 0x60400000-0x605fffff pref]
[    1.261113] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.267693] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
[    1.274241] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
[    1.281155] pci 0000:00:01.0: BAR 0: no space for [mem size 0x80000000]
[    1.287689] pci 0000:00:01.0: BAR 0: failed to assign [mem size 0x80000000]
[    1.294655] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    1.301381] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x602fffff pref]
[    1.308521] pci 0000:00:01.0: BAR 8: assigned [mem 0x60300000-0x603fffff]
[    1.315290] pci 0000:00:01.0: BAR 9: assigned [mem 0x60400000-0x605fffff pref]
[    1.322463] pci 0000:00:00.0: BAR 1: assigned [mem 0x60600000-0x6060ffff]
[    1.329206] pci 0000:00:01.0: BAR 1: assigned [mem 0x60610000-0x6061ffff]
[    1.335912] pci 0000:00:00.0: BAR 7: assigned [io  0x0000-0x0fff]
[    1.341988] pci 0000:00:01.0: BAR 7: assigned [io  0x1000-0x1fff]
[    1.348019] pci 0000:01:00.0: BAR 0: assigned [mem 0x60100000-0x601fffff 64bit pref]
[    1.355755] pci 0000:01:00.0: BAR 2: assigned [mem 0x60200000-0x60203fff 64bit pref]
[    1.363444] pci 0000:01:00.0: BAR 4: assigned [mem 0x60204000-0x60204fff 64bit pref]
[    1.371139] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.376020] pci 0000:00:00.0:   bridge window [io  0x0000-0x0fff]
[    1.382099] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    1.388839] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x602fffff pref]
[    1.395986] pci 0000:02:00.0: BAR 0: assigned [mem 0x60400000-0x604fffff 64bit pref]
[    1.403712] pci 0000:02:00.0: BAR 2: assigned [mem 0x60500000-0x60503fff 64bit pref]
[    1.411421] pci 0000:02:00.0: BAR 4: assigned [mem 0x60504000-0x60504fff 64bit pref]
[    1.419109] pci 0000:00:01.0: PCI bridge to [bus 02]
[    1.423990] pci 0000:00:01.0:   bridge window [io  0x1000-0x1fff]
[    1.430069] pci 0000:00:01.0:   bridge window [mem 0x60300000-0x603fffff]
[    1.436777] pci 0000:00:01.0:   bridge window [mem 0x60400000-0x605fffff pref]
[    1.445683] Serial: 8250/16550 driver, 3 ports, IRQ sharing disabled
[    1.455028] printk: console [ttyS0] disabled
[    1.460091] 1e000c00.uartlite: ttyS0 at MMIO 0x1e000c00 (irq = 18, base_baud = 3125000) is a 16550A
[    1.469200] printk: console [ttyS0] enabled
[    1.477467] printk: bootconsole [early0] disabled
[    1.492050] spi-mt7621 1e000b00.spi: sys_freq: 220000000
[    1.498688] spi-nor spi0.0: en25qh128 (16384 Kbytes)
[    1.503891] 5 fixed-partitions partitions found on MTD device spi0.0
[    1.510312] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    1.517627] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    1.525315] Creating 5 MTD partitions on "spi0.0":
[    1.530171] 0x000000000000-0x000000040000 : "u-boot"
[    1.536986] 0x000000040000-0x000000fa0000 : "firmware"
[    1.543587] 2 uimage-fw partitions found on MTD device firmware
[    1.549586] Creating 2 MTD partitions on "firmware":
[    1.554537] 0x000000000000-0x000000316c3d : "kernel"
[    1.559507] mtd: partition "kernel" doesn't end on an erase/write block -- force read-only
[    1.568983] 0x000000316c3d-0x000000f60000 : "rootfs"
[    1.573979] mtd: partition "rootfs" doesn't start on an erase/write block boundary -- force read-only
[    1.584263] mtd: setting mtd3 (rootfs) as root device
[    1.589470] 1 squashfs-split partitions found on MTD device rootfs
[    1.595638] 0x000000880000-0x000000f60000 : "rootfs_data"
[    1.602400] 0x000000fa0000-0x000000fb0000 : "config"
[    1.608648] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    1.616432] 0x000000fb0000-0x000000ff0000 : "tplink"
[    1.622923] 0x000000ff0000-0x000001000000 : "radio"
[    1.629272] OF: Bad cell count for /palmbus@1e000000/spi@b00/flash@0/partitions
[    1.772068] mt7530-mdio mdio-bus:1f: MT7530 adapts as multi-chip module
[    1.786488] mtk_soc_eth 1e100000.ethernet eth0: mediatek frame engine at 0xbe100000, irq 19
[    1.798253] mtk_soc_eth 1e100000.ethernet eth1: mediatek frame engine at 0xbe100000, irq 19
[    1.809387] i2c_dev: i2c /dev entries driver
[    1.817375] NET: Registered PF_INET6 protocol family
[    1.825538] Segment Routing with IPv6
[    1.829378] In-situ OAM (IOAM) with IPv6
[    1.833405] NET: Registered PF_PACKET protocol family
[    1.838970] 8021q: 802.1Q VLAN Support v1.8
[    1.873308] mt7530-mdio mdio-bus:1f: MT7530 adapts as multi-chip module
[    1.907146] mt7530-mdio mdio-bus:1f: configuring for fixed/rgmii link mode
[    1.915573] mt7530-mdio mdio-bus:1f lan1 (uninitialized): PHY [mt7530-0:00] driver [MediaTek MT7530 PHY] (irq=21)
[    1.926377] mt7530-mdio mdio-bus:1f: Link is Up - 1Gbps/Full - flow control rx/tx
[    1.932096] mt7530-mdio mdio-bus:1f lan2 (uninitialized): PHY [mt7530-0:01] driver [MediaTek MT7530 PHY] (irq=22)
[    1.946940] mt7530-mdio mdio-bus:1f lan3 (uninitialized): PHY [mt7530-0:02] driver [MediaTek MT7530 PHY] (irq=23)
[    1.960173] mt7530-mdio mdio-bus:1f lan4 (uninitialized): PHY [mt7530-0:03] driver [MediaTek MT7530 PHY] (irq=24)
[    1.973037] mtk_soc_eth 1e100000.ethernet eth0: entered promiscuous mode
[    1.979941] DSA: tree 0 setup
[    1.983824] clk: Disabling unused clocks
[    1.994980] VFS: Mounted root (squashfs filesystem) readonly on device 31:3.
[    2.005733] Freeing unused kernel image (initmem) memory: 1228K
[    2.011741] This architecture does not have kernel memory protection.
[    2.018173] Run /sbin/init as init process
[    2.022270]   with arguments:
[    2.022278]     /sbin/init
[    2.022285]   with environment:
[    2.022290]     HOME=/
[    2.022296]     TERM=linux
[    2.565075] init: Console is alive
[    2.568991] init: - watchdog -
[    3.526656] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.590105] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    3.604649] init: - preinit -
[    4.430979] mtk_soc_eth 1e100000.ethernet wan: renamed from eth1
[    5.918764] random: crng init done
[    6.411849] mtk_soc_eth 1e100000.ethernet eth0: configuring for fixed/rgmii link mode
[    6.424628] mt7530-mdio mdio-bus:1f lan1: configuring for phy/gmii link mode
[    6.431812] mtk_soc_eth 1e100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[    8.875186] jffs2: notice: (540) jffs2_build_xattr_subsystem: complete building xattr subsystem, 30 of xdatum (13 unchecked, 4 orphan) and 37 of xref (1 dead, 3 orphan) found.
[    8.893283] mount_root: switching to jffs2 overlay
[    8.904697] overlayfs: upper fs does not support tmpfile.
[    8.928083] urandom-seed: Seeding with /etc/urandom.seed
[    9.052592] procd: - early -
[    9.055700] procd: - watchdog -
[    9.660465] procd: - watchdog -
[    9.664882] procd: - ubus -
[    9.831247] procd: - init -
[   10.555142] kmodloader: loading kernel modules from /etc/modules.d/*
[   11.070096] GACT probability on
[   11.075090] Mirror/redirect action on
[   11.090951] u32 classifier
[   11.093695]     input device check on
[   11.097341]     Actions configured
[   11.125514] compat: loading out-of-tree module taints kernel.
[   11.131664] Loading modules backported from Linux version v6.9.9-0-g28fdf4518483
[   11.139086] Backport generated by backports.git v6.1.97-1-29-gf1d24a3683b2
[   11.365777] pci 0000:00:00.0: enabling device (0006 -> 0007)
[   11.371508] mt7915e_hif 0000:01:00.0: enabling device (0000 -> 0002)
[   11.378410] pci 0000:00:01.0: enabling device (0006 -> 0007)
[   11.384160] mt7915e 0000:02:00.0: enabling device (0000 -> 0002)
[   11.693601] urngd: v1.0.2 started.
[   11.700472] mt7915e 0000:02:00.0: HW/SW Version: 0x8a108a10, Build Time: 20240429200716a
[   11.700472] 
[   12.060309] mt7915e 0000:02:00.0: WM Firmware Version: ____000000, Build Time: 20240429200752
[   12.094385] mt7915e 0000:02:00.0: WA Firmware Version: DEV_000000, Build Time: 20240429200812
[   12.213164] mt7915e 0000:02:00.0: registering led 'mt76-phy0'
[   12.324547] mt7915e 0000:02:00.0: registering led 'mt76-phy1'
[   17.664842] wireguard: WireGuard 1.0.0 loaded. See www.wireguard.com for information.
[   17.672753] wireguard: Copyright (C) 2015-2019 Jason A. Donenfeld <Jason@zx2c4.com>. All Rights Reserved.
[   17.685666] kmodloader: done loading kernel modules from /etc/modules.d/*
[   29.077649] mtk_soc_eth 1e100000.ethernet eth0: Link is Down
[   29.112951] mtk_soc_eth 1e100000.ethernet eth0: configuring for fixed/rgmii link mode
[   29.121245] mtk_soc_eth 1e100000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[   29.134123] mt7530-mdio mdio-bus:1f lan1: configuring for phy/gmii link mode
[   29.142980] br-lan: port 1(lan1) entered blocking state
[   29.148250] br-lan: port 1(lan1) entered disabled state
[   29.153704] mt7530-mdio mdio-bus:1f lan1: entered allmulticast mode
[   29.160066] mtk_soc_eth 1e100000.ethernet eth0: entered allmulticast mode
[   29.170086] mt7530-mdio mdio-bus:1f lan1: entered promiscuous mode
[   29.212573] mt7530-mdio mdio-bus:1f lan2: configuring for phy/gmii link mode
[   29.221615] br-lan: port 2(lan2) entered blocking state
[   29.226916] br-lan: port 2(lan2) entered disabled state
[   29.232419] mt7530-mdio mdio-bus:1f lan2: entered allmulticast mode
[   29.240814] mt7530-mdio mdio-bus:1f lan2: entered promiscuous mode
[   29.261515] mt7530-mdio mdio-bus:1f lan3: configuring for phy/gmii link mode
[   29.270439] br-lan: port 3(lan3) entered blocking state
[   29.275744] br-lan: port 3(lan3) entered disabled state
[   29.281365] mt7530-mdio mdio-bus:1f lan3: entered allmulticast mode
[   29.290007] mt7530-mdio mdio-bus:1f lan3: entered promiscuous mode
[   29.309872] mt7530-mdio mdio-bus:1f lan4: configuring for phy/gmii link mode
[   29.318618] br-lan: port 4(lan4) entered blocking state
[   29.324026] br-lan: port 4(lan4) entered disabled state
[   29.329546] mt7530-mdio mdio-bus:1f lan4: entered allmulticast mode
[   29.338209] mt7530-mdio mdio-bus:1f lan4: entered promiscuous mode
[   29.361804] mtk_soc_eth 1e100000.ethernet wan: PHY [mt7530-0:04] driver [MediaTek MT7530 PHY] (irq=POLL)
[   29.371430] mtk_soc_eth 1e100000.ethernet wan: configuring for phy/rgmii link mode
[   32.519163] mtk_soc_eth 1e100000.ethernet wan: Link is Up - 1Gbps/Full - flow control off
[   33.102891] br-lan: port 5(wlan0) entered blocking state
[   33.108321] br-lan: port 5(wlan0) entered disabled state
[   33.113999] mt7915e 0000:02:00.0 wlan0: entered allmulticast mode
[   33.120836] mt7915e 0000:02:00.0 wlan0: entered promiscuous mode
[   33.135937] mt7915e 0000:02:00.0 wlan0: left allmulticast mode
[   33.142073] mt7915e 0000:02:00.0 wlan0: left promiscuous mode
[   33.148131] br-lan: port 5(wlan0) entered disabled state
[   33.343828] br-lan: port 5(wlan0) entered blocking state
[   33.349379] br-lan: port 5(wlan0) entered disabled state
[   33.354869] mt7915e 0000:02:00.0 wlan0: entered allmulticast mode
[   33.361774] mt7915e 0000:02:00.0 wlan0: entered promiscuous mode
[   33.368360] br-lan: port 5(wlan0) entered blocking state
[   33.373884] br-lan: port 5(wlan0) entered forwarding state
[   33.529141] br-lan: port 5(wlan0) entered disabled state
[   33.656211] br-lan: port 5(wlan0) entered blocking state
[   33.661731] br-lan: port 5(wlan0) entered forwarding state
[   33.738565] br-lan: port 6(wlan1) entered blocking state
[   33.744085] br-lan: port 6(wlan1) entered disabled state
[   33.749706] mt7915e 0000:02:00.0 wlan1: entered allmulticast mode
[   33.756504] mt7915e 0000:02:00.0 wlan1: entered promiscuous mode
[   33.766106] mt7915e 0000:02:00.0 wlan1: left allmulticast mode
[   33.772187] mt7915e 0000:02:00.0 wlan1: left promiscuous mode
[   33.778246] br-lan: port 6(wlan1) entered disabled state
[   34.333243] br-lan: port 6(wlan1) entered blocking state
[   34.338680] br-lan: port 6(wlan1) entered disabled state
[   34.344285] mt7915e 0000:02:00.0 wlan1: entered allmulticast mode
[   34.351187] mt7915e 0000:02:00.0 wlan1: entered promiscuous mode
[   34.357810] br-lan: port 6(wlan1) entered blocking state
[   34.363321] br-lan: port 6(wlan1) entered forwarding state
[   34.608969] br-lan: port 6(wlan1) entered disabled state
[   34.965332] br-lan: port 6(wlan1) entered blocking state
[   34.970801] br-lan: port 6(wlan1) entered forwarding state

brada4 · September 16, 2024, 4:27pm

Well .6 MB less, what are space requirements for that customistaion you are doing? Probably temp files belong to /tmp.

Nihilokrat · September 16, 2024, 5:05pm

The additional files are 14.2kb and a couple get deleted after customization process has run through.

The size of my package, including the aforementioned scripts, but not the customizations, is 61440, according to grep -H Installed-Size: /usr/lib/opkg/info/*.control | sed 's,^.*/\([^/]\+\)\.control:Installed-Size: *\(.*\),\2\t\1,' | sort -n

df -h for the latest image I built:

root@12345:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.5M      5.5M         0 100% /rom
tmpfs                    58.1M    304.0K     57.8M   1% /tmp
/dev/mtdblock4            6.9M    388.0K      6.5M   6% /overlay
overlayfs:/overlay        6.9M    388.0K      6.5M   6% /
tmpfs                   512.0K         0    512.0K   0% /dev

I'll built an image with mbedtls instead of openssl later and try the sysupgrade process on a OEM-device tomorrow to check the behaviour. I think I did this in early summer though, to no avail. When doing so, I'll also create a file manually, outside of the copying process to check whether than one sticks.

Nihilokrat · September 19, 2024, 1:22pm

Update with a stripped down image, flashed onto a new device.

root@OpenWrt:~# free
              total        used        free      shared  buff/cache   available
Mem:         119040       61808       30768         328       26464       19260
Swap:             0           0           0
root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.8M      4.8M         0 100% /rom
tmpfs                    58.1M    232.0K     57.9M   0% /tmp
tmpfs                    58.1M     96.0K     58.0M   0% /tmp/root
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mtdblock4            7.6M    368.0K      7.3M   5% /overlay
overlayfs:/overlay        7.6M    368.0K      7.3M   5% /

Difference after uniqueness scripts:

root@12345:~# free
              total        used        free      shared  buff/cache   available
Mem:         119040       64200       30192         300       24648       17788
Swap:             0           0           0
root@12345:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.8M      4.8M         0 100% /rom
tmpfs                    58.1M    300.0K     57.8M   1% /tmp
/dev/mtdblock4            7.6M    392.0K      7.2M   5% /overlay
overlayfs:/overlay        7.6M    392.0K      7.2M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev

Sysupgrade resets again:

root@OpenWrt:~# free
              total        used        free      shared  buff/cache   available
Mem:         119104       60712       32348         236       26044       20676
Swap:             0           0           0
root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 4.8M      4.8M         0 100% /rom
tmpfs                    58.2M    236.0K     57.9M   0% /tmp
/dev/mtdblock4            7.8M    364.0K      7.4M   5% /overlay
overlayfs:/overlay        7.8M    364.0K      7.4M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev

Comparison between the output of two devices where sysupgrade worked below.

Old image

root@12345:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.5M      5.5M         0 100% /rom
tmpfs                    58.6M    116.0K     58.5M   0% /tmp
/dev/mtdblock4            7.1M    436.0K      6.7M   6% /overlay
overlayfs:/overlay        7.1M    436.0K      6.7M   6% /
tmpfs                   512.0K         0    512.0K   0% /dev
root@12345:~# free
              total        used        free      shared  buff/cache   available
Mem:         120108       57972       45504         116       16632       29188
Swap:             0           0           0
root@12345:~# dmesg
[    0.000000] Linux version 5.15.135 (nihilokrat@builder) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 12.3.0 r23491-816933bee6) 12.3.0, GNU ld (GNU Binutils) 2.40.0) #0 SMP Thu Oct 19 22:36:32 2023
root@12345:~# dmesg | grep overlay
[    7.101461] mount_root: switching to jffs2 overlay
[    7.115622] overlayfs: upper fs does not support tmpfile.

New image (applied via sysupgrade)

root@12345:~# dmesg
[    0.000000] Linux version 5.15.147 (nihilokrat@builder) (mipsel-openwrt-linux-musl-gcc (OpenWrt GCC 12.3.0 r24453-a39a49e323) 12.3.0, GNU ld (GNU Binutils) 2.40.0) #0 SMP Wed Jan 24 12:08:36 2024
root@12345:~# dmesg | grep overlay
[    7.115632] mount_root: switching to jffs2 overlay
[    7.127258] overlayfs: upper fs does not support tmpfile.
root@12345:~# free
              total        used        free      shared  buff/cache   available
Mem:         120204       64132       34324         392       21748       20428
Swap:             0           0           0
root@12345:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                 5.5M      5.5M         0 100% /rom
tmpfs                    58.7M    392.0K     58.3M   1% /tmp
/dev/mtdblock4            7.3M    392.0K      6.9M   5% /overlay
overlayfs:/overlay        7.3M    392.0K      6.9M   5% /
tmpfs                   512.0K         0    512.0K   0% /dev

The difference I see when comparing the dmesg of sysupgrades that worked with those that didn't, is the following:

[   35.141083] mount_root: overlay filesystem has not been fully initialized yet

Something is off here. Below is the filesize of the images:

Worked:
Imagesize of factory.bin is **8489818**
Imagesize of sysupgrade.bin for above factory.bin is **8520792**

Didn't work:
Imagesize of factory.bin is  **8097746**
Imagesize of sysupgrade.bin for above factory.bin is **7996504**