Updating base-files remounts root as readonly and update fails

It's one oddness over another. At least I'm learning with them.

I'm using OpenWrt 21.02-SNAPSHOT r15986-cc51d97200 and was gonna upgrade packages, but it was failing:

# opkg upgrade base-files
Upgrading base-files on root from 1406-r15986-cc51d97200 to 1406-r16015-beab1bf163...
Downloading https://downloads.openwrt.org/releases/21.02-SNAPSHOT/targets/x86/64/packages/base-files_1406-r16015-beab1bf163_x86_64.ipk
Command failed: Not found
Command failed: Not found
umount: devpts busy - remounted read-only
umount: tmpfs busy - remounted read-only
umount: can't remount tmpfs read-only
umount: proc busy - remounted read-only
Collected errors:
 * copy_file: unable to open `/etc/group-opkg.backup': Read-only file system.
 * file_copy: Failed to copy file /etc/group to /etc/group-opkg.backup.
 * backup_make_backup: Failed to copy /etc/group to /etc/group-opkg.backup
 * pkg_write_filelist: Failed to open //usr/lib/opkg/info/base-files.list: Read-only file system.

IDK why, root is being mounted as readonly:

# mount
/dev/root on / type ext4 (ro,noatime)
proc on /proc type proc (ro,noatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (ro,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (ro,relatime,mode=600,ptmxmode=000)

boot partition also isn't being mounted, I'm not sure but it was before.

Well, I remounted it with mount -o remount,rw / and was able to install block-mount.

> # mount
> /dev/root on / type ext4 (rw,noatime)
> proc on /proc type proc (ro,noatime)
> tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
> tmpfs on /dev type tmpfs (ro,relatime,size=512k,mode=755)
> devpts on /dev/pts type devpts (ro,relatime,mode=600,ptmxmode=000)

But just after verifying it's rw, I try to upgrade a package and it's back ro. It's as if the opkg upgrade itself is remounting root as ro.

Indeed, it's mounting fine on boot

# mount
/dev/root on / type ext4 (rw,noatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/nvme0n1p1 on /boot type ext4 (rw,noatime)
/dev/nvme0n1p1 on /boot type ext4 (rw,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)

It's when upgrading base-files that causes it to remount as readonly.

This is a symptom of a corrupted file system.
Force the file system check and verify SMART from a Linux live session:

sudo fsck.ext4 -f /dev/root_part_name
sudo smartctl -a /dev/disk_name

If the issue persists, it can be related to the following:

4 Likes

smartctl and fsck passed.

I doubt the SSD is corrupted. As I said on 2nd post, it boots fine including the boot partition. It's when upgrading base-files that remounts it to readonly.

There are tons of kmods available to upgrade. I need to upgrade kernel so I can install mwan3...

1 Like

one should never upgrade packages ....

base-files triggering ro on the fs have been reported in the past.

2 Likes

Yeah I had read the article saying that upgrading packages is discouraged and we should prefer sysupgrading.

But mwan3 was being rejected to be installed requiring a newer kernel.

I know it's not the recommended path, but I just installed all other dependences and forced opkg to install mwan3 with --force-depends.

I hope 21.02.0 is released soon so I don't need to use snapshot anymore :-x

tnx for all the help :smiley:

You can't install kmod upgrades. there is strict version checksumming, which prevents kmod upgrades in snapshots. The kmods need to be from the same build run as the kernel itself (which you can only upgrade with a sysupgrade)

You may somewhat safely try to upgrade such add-on packages that you have installed into the router after flashing the router.

But trying to upgrade core system packages like base-files, ubus, procd etc. can lead into major trouble (as you have now noticed).

2 Likes

it's a catch 22.

mwan3 wants additional modules, additional modules require newer snapshot ... you get it.

grab newest snapshot, make a local copy of all modules, or install 19.07.7 stable, to avoid all the headache.

new snapshots invalidate the previous modules every 24h or so.

1 Like

... and I can't use 19.07.7 because it doesn't have NVMe driver :smiley:

I think it's unavoidable as long as 21.02.0 isn't released.

In fact, I'll need to wait for 22 or maybe 23 version so I have kernel 5.9+ which is when my onboard Realtek chip becomes supported.

I still have my EdgeRouter, so if needed I'll just plug it back and reinstall the whole thing. I just don't wanna redo all the work on every issue I find, I'd rather discover the cause and learn from it.

ah, another catch 22 :slight_smile:

you could enable it yourself, in the menuconfig, and create an image.
might even be enough with the kernel binary, since the driver have to be there, to be able to boot.
if you clone the 19.07.7 release sources from github, you should be able to use the
online repository for all packages.

or boot from an USB flash drive or SD card, until next stable is out.

2 Likes

I had read about making my compile and did it once. But I think it's more reliable to use buildbot's 21.02 snapshot than compiling it myself. menuconfig has too many configs for me and I didn't even find the NVMe driver on it.

Well looks like the kernel dependency is indeed breaking mwan3 :confused:

# mwan3 status
Interface status:
 interface want is error (15) and tracking is paused
 interface wan_mrd is error (15) and tracking is paused
 interface want6 is error (15) and tracking is paused
 interface wan_mrd6 is error (15) and tracking is paused

# mwan3 restart
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.
ipset v7.6: Cannot open session to kernel.

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.