F2FS inode recovery

Hi,

Some of our embedded units enter a strange state on booting after a power-down:
We have noticed this on other units that have an overlay filessystem - and sometimes it can take a long time before the unit properly boots again.

[    9.448158] init: - preinit -
[   10.659347] nicvf 0000:05:00.1 eth0: Link is Up 1000 Mbps Full duplex
[   14.795447] mount_root: loading kmods from internal overlay
[   14.837603] kmodloader: loading kernel modules from //etc/modules-boot.d/*
[   14.847751] kmodloader: done loading kernel modules from //etc/modules-boot.d/*
[   15.137347] block: attempting to load /etc/config/fstab
[   15.147132] block: unable to load configuration (fstab: Entry not found)
[   15.153905] block: no usable configuration
[   15.380795] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.388293] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   15.396827] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.404279] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   15.412810] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.420260] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   15.428789] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.436240] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   15.444768] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.452227] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   15.460757] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.468209] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   15.476746] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.484213] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 1, err = 0
[   15.492744] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   15.500197] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0

Once complete it seems happy enough again:

[   16.164241] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   16.171702] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 1, err = 0
[   16.180240] F2FS-fs (loop0): recover_inode: ino = baf, name = syslog, inline = 1
[   16.187695] F2FS-fs (loop0): recover_data: ino = baf (i_size: recover) recovered = 0, err = 0
[   17.213819] F2FS-fs (loop0): checkpoint: version = 7f7acc28
[   17.219537] F2FS-fs (loop0): Mounted with checkpoint version = 7f7acc28
[   17.351770] block: attempting to load /etc/config/fstab
[   17.357116] block: unable to load configuration (fstab: Entry not found)
[   17.363886] block: no usable configuration
[   17.369894] mount_root: switching to f2fs overlay
[   17.410907] overlayfs: "xino" feature enabled using 32 upper inode bits.
[   17.508656] urandom-seed: Seeding with /etc/urandom.seed
[   17.813323] procd: - early -
[   17.816305] procd: - watchdog -
[   18.381135] procd: - watchdog -
[   18.384626] procd: - ubus -
[   18.454204] procd: - init -
[   18.819612] urngd: v1.0.2 started.
[   18.919262] kmodloader: loading kernel modules from /etc/modules.d/*
# df -h
Filesystem                Size       Used Available Use% Mounted on
/dev/root                   48.0M     48.0M         0 100% /rom
tmpfs                   495.3M     14.9M    480.3M   3% /tmp
/dev/loop0              918.0M    167.9M    750.1M  18% /overlay
overlayfs:/overlay      918.0M    167.9M    750.1M  18% /
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/mmcblk1p4           28.7G     44.0M     27.1G   0% /media
#

Does anyone know how we can prevent this behaviour?
If we shrunk the overlay partition would it at least reduce the time it takes?
I know the above example only took 2 seconds but sometimes it can take 30 minutes.
Not sure what "name = syslog" is about either because there's not one on OpenWRT.
Thanks in advance for any advice.
Stuart

Is this after installing an update that changes the kernel-version?

Nope. No changes, just an image that 99% of the time boots up fine after a power cycle. We have seen the issue on different units so it’s likely not a h/w problem.

What version of the kernel?

User profile -

Devices in use with OpenWrt: Gateworks Octeontx

If this is the device, it looks like it's using a custom version of OpenWrt...not official firmware.

Gateworks Octeontx has been discussed here before -

Yes - we are building Getworks Newport Octeontx.
From the build log the kernel version is:

/target-aarch64_generic_musl/linux-octeontx/linux-5.4.45/.config.override
KERNELRELEASE=5.4.45