OpenWrt 19.07.5 UBIFS file system read-only after power cut

Hi everyone,

Please help me out, my OpenWrt's file system became read-only after a single power outage. Currently router works fine (with all my settings as on the moment outage happened) but I can't save and apply any changes to settings via LuCi anymore.

I'm running vanilla OpenWrt 19.07.5 (openwrt-19.07.5-ramips-mt7621-xiaomi_mir3p-squashfs-factory.bin) on Xiaomi Mi Router Pro (R3P, ESMT NAND, no bad blocks) without any additional packages. I cannot reinstall OpenWrt because my laptop doesn't have Ethernet port anymore. And I'm looking for a proper solution to be able to deal with this in case it happens again.

Here's troubleshooting data:

/$ df -Th
Filesystem           Type            Size      Used Available Use% Mounted on
/dev/root            squashfs        3.0M      3.0M         0 100% /rom
tmpfs                tmpfs         250.6M     88.0K    250.5M   0% /tmp
/dev/ubi0_1          ubifs         210.1M      6.1M    199.3M   3% /overlay
overlayfs:/overlay   overlay       210.1M      6.1M    199.3M   3% /
tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
/$ mount
/dev/root			on /rom					type squashfs	(ro,relatime)
proc				on /proc				type proc		(rw,nosuid,nodev,noexec,noatime)
sysfs				on /sys					type sysfs		(rw,nosuid,nodev,noexec,noatime)
tmpfs				on /tmp					type tmpfs		(rw,nosuid,nodev,noatime)
/dev/ubi0_1			on /overlay				type ubifs		(ro,noatime,ubi=0,vol=1)
overlayfs:/overlay	on /					type overlay	(rw,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
tmpfs				on /dev					type tmpfs		(rw,nosuid,relatime,size=512k,mode=755)
devpts				on /dev/pts				type devpts		(rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs				on /sys/kernel/debug	type debugfs	(rw,noatime)

As far as I can tell there's plenty of free space available in /overlay. I suspect the problem is in /dev/ubi0_1 on /overlay type ubifs (ro,noatime,ubi=0,vol=1).

I'm able to successfully mount /dev/ubi0_1 as read-write using this command
mount -o rw,remount /dev/ubi0_1
and change/edit files (like /etc/config/wireless contents) via SCP/SSH since /dev/ubi0_1 becomes read-write in mount report. However after reboot the edited files are back to their pre-edit state!

If I click save and apply any changes via LuCi after successful execution mount -o rw,remount /dev/ubi0_1, the file system becomes read-only and no changes are saved! Using SSH I can see /dev/ubi0_1 on /overlay type ubifs (rw,noatime,ubi=0,vol=1) becomes read-only /dev/ubi0_1 on /overlay type ubifs (ro,noatime,ubi=0,vol=1) after reboot.

As far as I could figure, file system becomes read-only of OS marked it as not clean (dirty, unsafe, corrupt) after power cut. So user has run file system integrity check and fix error to mark file system as safe to operate (clean). This is typically done via fsck. I've created a VM (with KDE neon guest) and tried running fsck /dev/ubi0_1 via SSH from Konsole. But no luck, the command is not found (well, probably because it is not present in my OpenWrt installation).

Maybe there's another way to mark file system as safe somewhere manually without fsck (likely f2fsck for my router)?

Or I may be completely wrong and this has to do with UBIFS suddenly read-only issue (but my /proc/mounts file is emply so no clues there) as dmesg says UBIFS (ubi0:1): recovery needed & UBIFS (ubi0:1): recovery completed during boot and after that (in the very end of dmesg):

[   36.510035] UBIFS error (ubi0:1 pid 1328): 0x801b3f18: failed to read inode 1865, error -2
[   36.510093] UBIFS error (ubi0:1 pid 1328): 0x801afe90: dead directory entry 'iw', error -2
[   36.510108] UBIFS warning (ubi0:1 pid 1328): 0x801b6ef0: switched to read-only mode, error -2

To make things worse, I believe my router was running without power cuts previously, so my issue may be related to the thread UBIFS fails to mount on second boot (without solution). And as far as I googled, there's no fsck for UBIFS, a different tricky system integrity check method should be utilized, here's result:

/$ find / -type f -print -exec cat {} + > /dev/null
find: /overlay/upper/usr/sbin/iw: No such file or directory
cat: can't open '/proc/sys/net/ipv4/route/flush': Permission denied
cat: read error: I/O error
cat: can't open '/proc/sys/net/ipv6/route/flush': Permission denied
cat: can't open '/proc/sys/vm/compact_memory': Permission denied
cat: can't open '/proc/sys/vm/drop_caches': Permission denied

or

/$ find . -type f -exec pv -N {} {} \; > /dev/null
find: ./overlay/upper/usr/sbin/iw: No such file or directory
find: ./proc/12323: No such file or directory

iw file is not there. And I've tried running mount -o rw,remount /dev/ubi0_1 and then manually copying /usr/sbin/iw to /overlay/upper/usr/sbin/iw but I get error No such file or directory, same with 12323 directory:
/$ cd /proc /proc$ mkdir 12323 mkdir: can't create directory '12323': No such file or directory

I have tried rebooting and cutting power multiple times, the UBIFS errors are same every time.

Please help me out, Sergey Brin called me and told I've already broken Google while searching for solution.

Kind regards,

bananakid

P.S. I'm providing dmesg data, but it's probably irrelevant:

/$ dmesg
[    0.000000] Linux version 4.14.209 (builder@buildhost) (gcc version 7.5.0 (OpenWrt GCC 7.5.0 r11257-5090152ae3)) #0 SMP Sun Dec 6 07:31:03 2020
[    0.000000] SoC Type: MediaTek MT7621 ver:1 eco:3
[    0.000000] bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 0001992f (MIPS 1004Kc)
[    0.000000] MIPS: machine is Xiaomi Mi Router 3 Pro
[    0.000000] Determined physical RAM map:
[    0.000000]  memory: 1c000000 @ 00000000 (usable)
[    0.000000]  memory: 04000000 @ 20000000 (usable)
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] VPE topology {2,2} total 4
[    0.000000] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.000000] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x000000000fffffff]
[    0.000000]   HighMem  [mem 0x0000000010000000-0x0000000023ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000001bffffff]
[    0.000000]   node   0: [mem 0x0000000020000000-0x0000000023ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000023ffffff]
[    0.000000] On node 0 totalpages: 131072
[    0.000000] free_area_init_node: node 0, pgdat 805747c0, node_mem_map 81003000
[    0.000000]   Normal zone: 512 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 65536 pages, LIFO batch:15
[    0.000000]   HighMem zone: 65536 pages, LIFO batch:15
[    0.000000] random: get_random_bytes called from 0x80577744 with crng_init=0
[    0.000000] percpu: Embedded 14 pages/cpu s26224 r8192 d22928 u57344
[    0.000000] pcpu-alloc: s26224 r8192 d22928 u57344 alloc=14*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 130560
[    0.000000] Kernel command line: console=ttyS0,115200n8 rootfstype=squashfs,jffs2
[    0.000000] PID hash table entries: 1024 (order: 0, 4096 bytes)
[    0.000000] Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
[    0.000000] Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
[    0.000000] Writing ErrCtl register=0004c810
[    0.000000] Readback ErrCtl register=0004c810
[    0.000000] Memory: 512044K/524288K available (4754K kernel code, 241K rwdata, 588K rodata, 1252K init, 255K bss, 12244K reserved, 0K cma-reserved, 262144K highmem)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS: 256
[    0.000000] CPU Clock: 880MHz
[    0.000000] clocksource: GIC: mask: 0xffffffffffffffff max_cycles: 0xcaf478abb4, max_idle_ns: 440795247997 ns
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 4343773742 ns
[    0.000010] sched_clock: 32 bits at 440MHz, resolution 2ns, wraps every 4880645118ns
[    0.007829] Calibrating delay loop... 586.13 BogoMIPS (lpj=2930688)
[    0.073978] pid_max: default: 32768 minimum: 301
[    0.078750] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.085259] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
[    0.094088] Hierarchical SRCU implementation.
[    0.099256] smp: Bringing up secondary CPUs ...
[    0.104667] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.104676] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.104688] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.104826] CPU1 revision is: 0001992f (MIPS 1004Kc)
[    0.164009] Synchronize counters for CPU 1: done.
[    0.198594] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.198604] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.198611] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.198680] CPU2 revision is: 0001992f (MIPS 1004Kc)
[    0.255203] Synchronize counters for CPU 2: done.
[    0.285867] Primary instruction cache 32kB, VIPT, 4-way, linesize 32 bytes.
[    0.285875] Primary data cache 32kB, 4-way, PIPT, no aliases, linesize 32 bytes
[    0.285882] MIPS secondary cache 256kB, 8-way, linesize 32 bytes.
[    0.285959] CPU3 revision is: 0001992f (MIPS 1004Kc)
[    0.340381] Synchronize counters for CPU 3: done.
[    0.370238] smp: Brought up 1 node, 4 CPUs
[    0.378330] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.388125] futex hash table entries: 1024 (order: 3, 32768 bytes)
[    0.394402] pinctrl core: initialized pinctrl subsystem
[    0.401081] NET: Registered protocol family 16
[    0.411096] FPU Affinity set after 11720 emulations
[    0.411797] pull PCIe RST: RALINK_RSTCTRL = 4000000
[    0.717085] release PCIe RST: RALINK_RSTCTRL = 7000000
[    0.722134] ***** Xtal 40MHz *****
[    0.725489] release PCIe RST: RALINK_RSTCTRL = 7000000
[    0.730590] Port 0 N_FTS = 1b105000
[    0.734057] Port 1 N_FTS = 1b105000
[    0.737496] Port 2 N_FTS = 1b102800
[    1.892727] PCIE2 no card, disable it(RST&CLK)
[    1.897096]  -> 21007f2
[    1.899491] PCIE0 enabled
[    1.902086] PCIE1 enabled
[    1.904686] PCI host bridge /pcie@1e140000 ranges:
[    1.909459]  MEM 0x0000000060000000..0x000000006fffffff
[    1.914623]   IO 0x000000001e160000..0x000000001e16ffff
[    1.919819] PCI coherence region base: 0xbfbf8000, mask/settings: 0x60000000
[    1.935813] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    1.941677] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    1.947433] mt7621_gpio 1e000600.gpio: registering 32 gpios
[    1.955185] PCI host bridge to bus 0000:00
[    1.959222] pci_bus 0000:00: root bus resource [mem 0x60000000-0x6fffffff]
[    1.966053] pci_bus 0000:00: root bus resource [io  0xffffffff]
[    1.971906] pci_bus 0000:00: root bus resource [??? 0x00000000 flags 0x0]
[    1.978665] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    1.986555] pci 0000:00:00.0: [0e8d:0801] type 01 class 0x060400
[    1.986596] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[    1.986608] pci 0000:00:00.0: reg 0x14: [mem 0x60200000-0x6020ffff]
[    1.986673] pci 0000:00:00.0: supports D1
[    1.986682] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.986921] pci 0000:00:01.0: [0e8d:0801] type 01 class 0x060400
[    1.986951] pci 0000:00:01.0: reg 0x10: [mem 0x00000000-0x7fffffff]
[    1.986965] pci 0000:00:01.0: reg 0x14: [mem 0x60210000-0x6021ffff]
[    1.987014] pci 0000:00:01.0: supports D1
[    1.987022] pci 0000:00:01.0: PME# supported from D0 D1 D3hot
[    1.987414] pci 0000:01:00.0: [14c3:7615] type 00 class 0x000280
[    1.987461] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[    1.987758] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.987924] pci 0000:02:00.0: [14c3:7615] type 00 class 0x000280
[    1.987974] pci 0000:02:00.0: reg 0x10: [mem 0x00000000-0x000fffff 64bit]
[    1.988268] pci_bus 0000:02: busn_res: [bus 02-ff] end is updated to 02
[    1.988287] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 02
[    1.988356] pci 0000:00:00.0: BAR 0: no space for [mem size 0x80000000]
[    1.994894] pci 0000:00:00.0: BAR 0: failed to assign [mem size 0x80000000]
[    2.001781] pci 0000:00:01.0: BAR 0: no space for [mem size 0x80000000]
[    2.008356] pci 0000:00:01.0: BAR 0: failed to assign [mem size 0x80000000]
[    2.015265] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    2.022015] pci 0000:00:01.0: BAR 8: assigned [mem 0x60100000-0x601fffff]
[    2.028740] pci 0000:00:00.0: BAR 1: assigned [mem 0x60200000-0x6020ffff]
[    2.035506] pci 0000:00:01.0: BAR 1: assigned [mem 0x60210000-0x6021ffff]
[    2.042223] pci 0000:01:00.0: BAR 0: assigned [mem 0x60000000-0x600fffff 64bit]
[    2.049497] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.054409] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    2.061162] pci 0000:02:00.0: BAR 0: assigned [mem 0x60100000-0x601fffff 64bit]
[    2.068405] pci 0000:00:01.0: PCI bridge to [bus 02]
[    2.073332] pci 0000:00:01.0:   bridge window [mem 0x60100000-0x601fffff]
[    2.081393] clocksource: Switched to clocksource GIC
[    2.087938] NET: Registered protocol family 2
[    2.093054] TCP established hash table entries: 2048 (order: 1, 8192 bytes)
[    2.099941] TCP bind hash table entries: 2048 (order: 2, 16384 bytes)
[    2.106363] TCP: Hash tables configured (established 2048 bind 2048)
[    2.112782] UDP hash table entries: 256 (order: 1, 8192 bytes)
[    2.118546] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
[    2.125002] NET: Registered protocol family 1
[    2.129322] PCI: CLS 0 bytes, default 32
[    2.361334] 4 CPUs re-calibrate udelay(lpj = 2924544)
[    2.367735] Crashlog allocated RAM at address 0x3f00000
[    2.373148] workingset: timestamp_bits=30 max_order=17 bucket_order=0
[    2.386788] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    2.392591] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    2.405030] random: fast init done
[    2.409288] bounce: pool size: 64 pages
[    2.413122] io scheduler noop registered
[    2.416947] io scheduler deadline registered (default)
[    2.423214] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    2.433147] console [ttyS0] disabled
[    2.436695] 1e000c00.uartlite: ttyS0 at MMIO 0x1e000c00 (irq = 19, base_baud = 3125000) is a 16550A
[    2.445746] console [ttyS0] enabled
[    2.452628] bootconsole [early0] disabled
[    2.462940] MediaTek Nand driver init, version v2.1 Fix AHB virt2phys error
[    2.470012] Enable NFI Clock
[    2.472956] # MTK NAND # : Use HW ECC
[    2.476625] Device found in MTK table, ID: c8da, EXT_ID: 909544
[    2.482538] Support this Device in MTK table! c8da 

[    2.487486] [NAND]select ecc bit:4, sparesize :64 spare_per_sector=16
[    2.493961] nand: device found, Manufacturer ID: 0xc8, Chip ID: 0xda
[    2.500281] nand: ESMT NAND 256MiB 3,3V 8-bit
[    2.504638] nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
[    2.512205] Scanning device for bad blocks
[    2.800327] 10 fixed-partitions partitions found on MTD device MT7621-NAND
[    2.807188] Creating 10 MTD partitions on "MT7621-NAND":
[    2.812505] 0x000000000000-0x000000040000 : "Bootloader"
[    2.818904] 0x000000040000-0x000000080000 : "Config"
[    2.824915] 0x000000080000-0x0000000c0000 : "Bdata"
[    2.830782] 0x0000000c0000-0x000000100000 : "Factory"
[    2.836924] 0x000000100000-0x000000140000 : "crash"
[    2.842839] 0x000000140000-0x0000001c0000 : "crash_syslog"
[    2.849346] 0x0000001c0000-0x000000200000 : "reserved0"
[    2.855655] 0x000000200000-0x000000600000 : "kernel_stock"
[    2.862226] 0x000000600000-0x000000a00000 : "kernel"
[    2.868246] 0x000000a00000-0x00000ff80000 : "ubi"
[    2.875885] [mtk_nand] probe successfully!
[    2.880692] Signature matched and data read!
[    2.884976] load_fact_bbt success 2047
[    2.889491] libphy: Fixed MDIO Bus: probed
[    2.963944] libphy: mdio: probed
[    4.367577] mtk_soc_eth 1e100000.ethernet: loaded mt7530 driver
[    4.374302] mtk_soc_eth 1e100000.ethernet eth0: mediatek frame engine at 0xbe100000, irq 21
[    4.385372] NET: Registered protocol family 10
[    4.391567] Segment Routing with IPv6
[    4.395312] NET: Registered protocol family 17
[    4.399802] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    4.412730] 8021q: 802.1Q VLAN Support v1.8
[    4.419943] UBI: auto-attach mtd9
[    4.423369] ubi0: attaching mtd9
[    4.647411] mtk_soc_eth 1e100000.ethernet eth0: port 4 link up
[    6.695327] ubi0: scanning is finished
[    6.716614] ubi0: attached mtd9 (name "ubi", size 245 MiB)
[    6.722149] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
[    6.728990] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
[    6.735761] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
[    6.742706] ubi0: good PEBs: 1964, bad PEBs: 0, corrupted PEBs: 0
[    6.748767] ubi0: user volume: 2, internal volumes: 1, max. volumes count: 128
[    6.755971] ubi0: max/mean erase counter: 2/1, WL threshold: 4096, image sequence number: 62933034
[    6.764904] ubi0: available PEBs: 0, total reserved PEBs: 1964, PEBs reserved for bad PEB handling: 40
[    6.774210] ubi0: background thread "ubi_bgt0d" started, PID 396
[    6.776222] block ubiblock0_0: created from ubi0:0(rootfs)
[    6.785701] ubiblock: device ubiblock0_0 (rootfs) set to be root filesystem
[    6.792659] hctosys: unable to open rtc device (rtc0)
[    6.797937] usb_vbus: disabling
[    6.807712] VFS: Mounted root (squashfs filesystem) readonly on device 254:0.
[    6.819182] Freeing unused kernel memory: 1252K
[    6.823765] This architecture does not have kernel memory protection.
[    7.387266] init: Console is alive
[    7.390917] init: - watchdog -
[    8.044744] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    8.109862] usbcore: registered new interface driver usbfs
[    8.115485] usbcore: registered new interface driver hub
[    8.120962] usbcore: registered new device driver usb
[    8.133576] xhci-mtk 1e1c0000.xhci: 1e1c0000.xhci supply vusb33 not found, using dummy regulator
[    8.142565] xhci-mtk 1e1c0000.xhci: xHCI Host Controller
[    8.147885] xhci-mtk 1e1c0000.xhci: new USB bus registered, assigned bus number 1
[    8.161537] xhci-mtk 1e1c0000.xhci: hcc params 0x01401198 hci version 0x96 quirks 0x0000000000210010
[    8.170708] xhci-mtk 1e1c0000.xhci: irq 20, io mem 0x1e1c0000
[    8.177465] hub 1-0:1.0: USB hub found
[    8.181288] hub 1-0:1.0: 2 ports detected
[    8.185975] xhci-mtk 1e1c0000.xhci: xHCI Host Controller
[    8.191289] xhci-mtk 1e1c0000.xhci: new USB bus registered, assigned bus number 2
[    8.198790] xhci-mtk 1e1c0000.xhci: Host supports USB 3.0  SuperSpeed
[    8.205397] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    8.214370] hub 2-0:1.0: USB hub found
[    8.218190] hub 2-0:1.0: 1 port detected
[    8.227547] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    8.251706] init: - preinit -
[    8.954492] mtk_soc_eth 1e100000.ethernet: PPE started
[    9.028146] random: procd: uninitialized urandom read (4 bytes read)
[   12.124471] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 502
[   12.197552] UBIFS (ubi0:1): recovery needed
[   12.446211] UBIFS (ubi0:1): recovery completed
[   12.450770] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
[   12.458578] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[   12.468461] UBIFS (ubi0:1): FS size: 238968832 bytes (227 MiB, 1882 LEBs), journal size 11935744 bytes (11 MiB, 94 LEBs)
[   12.479288] UBIFS (ubi0:1): reserved for root: 4952683 bytes (4836 KiB)
[   12.485887] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 0525424A-2206-4675-AC79-4E140757101E, small LPT model
[   12.510773] mount_root: switching to ubifs overlay
[   12.542654] urandom-seed: Seeding with /etc/urandom.seed
[   12.844519] mtk_soc_eth 1e100000.ethernet: 0x100 = 0x6060000c, 0x10c = 0x80818
[   12.913960] procd: - early -
[   12.918770] procd: - watchdog -
[   13.591546] procd: - watchdog -
[   13.595016] procd: - ubus -
[   13.635056] random: ubusd: uninitialized urandom read (4 bytes read)
[   13.657018] random: ubusd: uninitialized urandom read (4 bytes read)
[   13.665700] random: ubusd: uninitialized urandom read (4 bytes read)
[   13.672946] procd: - init -
[   14.548499] urngd: v1.0.2 started.
[   14.684961] random: crng init done
[   14.688378] random: 6 urandom warning(s) missed due to ratelimiting
[   14.724042] kmodloader: loading kernel modules from /etc/modules.d/*
[   14.743600] ip6_tables: (C) 2000-2006 Netfilter Core Team
[   14.769326] Loading modules backported from Linux version v4.19.137-0-gc076c79e03c6
[   14.777021] Backport generated by backports.git v4.19.137-1-0-g60c3a249
[   14.789154] ip_tables: (C) 2000-2006 Netfilter Core Team
[   14.821460] nf_conntrack version 0.5.0 (8192 buckets, 32768 max)
[   14.872540] xt_time: kernel timezone is -0000
[   15.162539] bus=0x1, slot = 0x0, irq=0x0
[   15.190382] mt7615e 0000:01:00.0: HW/SW Version: 0x8a108a10, Build Time: 20180518100604a
[   15.190382] 
[   15.363587] mt7615e 0000:01:00.0: N9 Firmware Version: _reserved_, Build Time: 20190103180756
[   15.422913] mt7615e 0000:01:00.0: CR4 Firmware Version: _reserved_, Build Time: 20181207140436
[   23.071971] ieee80211 phy0: Selected rate control algorithm 'minstrel_ht'
[   23.075307] bus=0x2, slot = 0x1, irq=0x0
[   23.101901] mt7615e 0000:02:00.0: HW/SW Version: 0x8a108a10, Build Time: 20180518100604a
[   23.101901] 
[   23.126704] mt7615e 0000:02:00.0: N9 Firmware Version: _reserved_, Build Time: 20190103180756
[   23.144009] mt7615e 0000:02:00.0: CR4 Firmware Version: _reserved_, Build Time: 20181207140436
[   30.813451] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   30.827656] PPP generic driver version 2.4.2
[   30.833718] NET: Registered protocol family 24
[   30.842625] kmodloader: done loading kernel modules from /etc/modules.d/*
[   36.184335] mtk_soc_eth 1e100000.ethernet: PPE started
[   36.189409] device eth0 entered promiscuous mode
[   36.190123] br-lan: port 1(eth0.1) entered blocking state
[   36.190139] br-lan: port 1(eth0.1) entered disabled state
[   36.190701] device eth0.1 entered promiscuous mode
[   36.193658] br-lan: port 1(eth0.1) entered blocking state
[   36.193684] br-lan: port 1(eth0.1) entered forwarding state
[   36.193962] IPv6: ADDRCONF(NETDEV_UP): br-lan: link is not ready
[   36.214951] br-wan: port 1(eth0.2) entered blocking state
[   36.214971] br-wan: port 1(eth0.2) entered disabled state
[   36.215718] device eth0.2 entered promiscuous mode
[   36.220712] br-wan: port 1(eth0.2) entered blocking state
[   36.220735] br-wan: port 1(eth0.2) entered forwarding state
[   36.510035] UBIFS error (ubi0:1 pid 1328): 0x801b3f18: failed to read inode 1865, error -2
[   36.510093] UBIFS error (ubi0:1 pid 1328): 0x801afe90: dead directory entry 'iw', error -2
[   36.510108] UBIFS warning (ubi0:1 pid 1328): 0x801b6ef0: switched to read-only mode, error -2
[   36.510129] CPU: 0 PID: 1328 Comm: sh Not tainted 4.14.209 #0
[   36.510138] Stack : 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   36.510186]         00000000 00000000 00000000 00000000 00000000 00000001 8e249a58 1cc28231
[   36.510233]         8e249af0 00000000 00000000 000056e0 00000038 8049da98 00000001 00000000
[   36.510280]         00000000 80550000 0000011d 806e0000 8e249a38 80000000 00000000 8f9f1770
[   36.510324]         8fd5a000 8ee21a00 00000384 00000000 00000001 80550000 08042098 806b0000
[   36.510366]         ...
[   36.510379] Call Trace:
[   36.510399] [<8049da98>] 0x8049da98
[   36.510453] [<8000c1a0>] 0x8000c1a0
[   36.510463] [<8000c1a8>] 0x8000c1a8
[   36.510471] [<804868d4>] 0x804868d4
[   36.510483] [<801afe90>] 0x801afe90
[   36.510499] [<801b6ef0>] 0x801b6ef0
[   36.510529] [<801afe9c>] 0x801afe9c
[   36.510578] [<80125fd8>] 0x80125fd8
[   36.510604] [<80126188>] 0x80126188
[   36.510634] [<801e0dac>] 0x801e0dac
[   36.510663] [<801e10ec>] 0x801e10ec
[   36.510676] [<80135764>] 0x80135764
[   36.510707] [<801e1720>] 0x801e1720
[   36.510724] [<80135e8c>] 0x80135e8c
[   36.510739] [<80053e60>] 0x80053e60
[   36.510771] [<80125fd8>] 0x80125fd8
[   36.510789] [<80126fe8>] 0x80126fe8
[   36.510801] [<80127b14>] 0x80127b14
[   36.510822] [<80127f10>] 0x80127f10
[   36.510845] [<80026f14>] 0x80026f14
[   36.510863] [<80129b04>] 0x80129b04
[   36.510891] [<800f3e74>] 0x800f3e74
[   36.510936] [<801297c8>] 0x801297c8
[   36.510953] [<8006d418>] 0x8006d418
[   36.510963] [<8011db0c>] 0x8011db0c
[   36.510972] [<80018a78>] 0x80018a78
[   36.510981] [<800324f4>] 0x800324f4
[   36.510992] [<8011e234>] 0x8011e234
[   36.510999] [<80030544>] 0x80030544
[   36.511009] [<800325c0>] 0x800325c0
[   36.511043] [<80113bc4>] 0x80113bc4
[   36.511057] [<8013c7dc>] 0x8013c7dc
[   36.511069] [<80015578>] 0x80015578
[   36.511087] 
[   37.242203] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   39.461593] IPv6: ADDRCONF(NETDEV_UP): wlan1: link is not ready
[   40.285023] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   40.288245] br-lan: port 2(wlan0) entered blocking state
[   40.288258] br-lan: port 2(wlan0) entered disabled state
[   40.288645] device wlan0 entered promiscuous mode
[   40.303329] br-lan: port 3(wlan1) entered blocking state
[   40.303348] br-lan: port 3(wlan1) entered disabled state
[   40.303762] device wlan1 entered promiscuous mode
[   41.898355] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   41.898711] br-lan: port 2(wlan0) entered blocking state
[   41.898727] br-lan: port 2(wlan0) entered forwarding state
[   43.404815] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   43.405144] br-lan: port 3(wlan1) entered blocking state
[   43.405158] br-lan: port 3(wlan1) entered forwarding state

Maybe reflash the image, keeping settings. Otherwise the wifi will be disabled and OP's laptop doesn't have ethernet.

3 Likes

Yes I would start with a re-install keeping settings. That will reformat the flash and likely solve the problem.

Any old router even with stock firmware can be configured as an AP, which could then be plugged into the LAN of a default install to obtain access with a wifi-only PC.

1 Like

Save yourself a lot of agony and buy a UE300 tp link gigabit USB NIC. They work great and cost $13

1 Like

I've got a few more of the same routers ordered for remote places so a stable solution or reliable fix is in priority (I've borrowed USB Ethernet adapter for my installation). I've found a thread at Linux developer mailing list talking about the UBIFS power cut file system corruption issue in Linux Kernel 4.14.96. However I have no idea if this fix made it to 4.14.209 (what comes with OpenWrt 19.07.5) and if it's possible to inject it somewhere. And I cannot figure out how to restore missing iw file manually.

With default config, OpenWrt writes nothing to flash after it has booted (and written urandom_seed). Thus, a power break could only be harmful during a few seconds during the boot process.

You also seemed to have 6 MB data in /overlay, which suggests to me that you are writing something to flash during normal operations.

In that case, you might consider using an USB stick for storage and keep the core flash unmodified. Then you would only need to care about the external storage but OS itself should be ok even after a power break.

2 Likes

I've managed to restore OpenWrt to functional state after UBIFS failed only by re-flashing the factory firmware via SSH. All other methods to factory reset firmware didn't work, including attempt to flash sysupgrade via LuCi.
Failed attempt #1:

Using username "root".
root@10.0.1.1's password:


BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.5, r11257-5090152ae3
 -----------------------------------------------------
root@Mi-Router-Pro:~# umount /overlay
root@Mi-Router-Pro:~# firstboot
This will erase all settings and remove any installed packages. Are you sure? [N        /y]
y
/dev/ubi0_1 is not mounted
/dev/ubi0_1 will be erased on next mount
writing /dev/ubi0_1 failed: Bad file descriptor
root@Mi-Router-Pro:~# reboot
root@Mi-Router-Pro:~#

Failed attempt #2:

Using username "root".
root@10.0.1.1's password:


BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.5, r11257-5090152ae3
 -----------------------------------------------------
root@Mi-Router-Pro:~# firstboot
This will erase all settings and remove any installed packages. Are you sure? [N/y]
y
/dev/ubi0_1 is mounted as /overlay, only erasing files
root@Mi-Router-Pro:~# reboot
root@Mi-Router-Pro:~#

Successful attempt (firstly openwrt-19.07.5-ramips-mt7621-xiaomi_mir3p-squashfs-factory.bin was copied to /tmp):

Using username "root".
root@10.0.1.1's password:


BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.5, r11257-5090152ae3
 -----------------------------------------------------
root@Mi-Router-Pro:~# cd /tmp
root@Mi-Router-Pro:/tmp# dd if=factory.bin bs=1M count=4 | mtd write - kernel
Unlocking kernel ...

Writing from <stdin> to kernel ...  [e]4+0 records in
4+0 records out
root@Mi-Router-Pro:/tmp# dd if=factory.bin bs=1M skip=4 | mtd write - ubi
Unlocking ubi ...

Writing from <stdin> to ubi ...  [e]4+0 records in
4+0 records out
root@Mi-Router-Pro:/tmp# reboot
root@Mi-Router-Pro:/tmp#

I also believe I found the source of UBIFS failure: my wall socket's contacts were not tight enough inside and the plug's contact to electricity was probably a little unreliable. I figured it only after I disassembled the damn wall socket, you couldn't tell otherwise. Hopefully fixing that will prevent same situation from happening again in the future.

Maybe someone finds this information useful!

P.S. @hnyman, 6.1–6.2 MB in /overlay looks like to be default space usage of OpenWrt 19.07.5 for Mi Router Pro. That's the usage after factory image installation.

1 Like

If your problem is solved, please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.