[Solved] Configuration changes do not persist after reboot

Installing and Using OpenWrt
1

Hello everyone,

I'm facing a strange problem with my x86 based router running OpenWrt 21.02.0 r16279-5cc0535800.

I'm not sure exactly when it started, but after a recent power cut, I noticed that my most recent configuration changes did not come back after power was restored.

My system was first installed by using image builder, from which I setup all services I wanted to be installed by default.

I'm quite sure everything was working as expected for several months, including configuration changes surviving reboots. I'm sure because over the months I had to make a few manual adjustments to static leases (from Luci) and AdGuardHome filters (from it's own configuration page) and those changes never got lost after reboots. I also updated the running version of AdGuardHome from it's own page at least 2 or 3 times up to the point I was running AdGuardHome v0.108.0-b.3, and everything just worked...

I cannot put my finger on what changed, but I also noticed that there is no use saving copies of configuration files with the last version, as those copies simply disappear after reboots, which is something I used to do extensively in the past with no problems and that after reboots, AdGuardHome tells me that there is a new version available, a version which I have already installed before, which means this update also vanishes somehow...

Can any of the more experienced users here point me in the right direction to troubleshoot this issue?

TIA
Cesar

2

What is the output of mount; df -h ?

3

Thanks for your help @trendy

Here is the output:

root@FW6B:~# mount; df -h ?
/dev/root on / type ext4 (rw,noatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/sda1 on /boot type ext4 (rw,noatime)
/dev/sda1 on /boot type ext4 (rw,noatime)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
Filesystem                Size      Used Available Use% Mounted on
df: ?: can't find mount point

can't find mount point message on the last line seem bad to me. I'm not a Linux user, but I guess this indicate something went wrong...

EDIT: Found out that I inadvertently appended a ? at the end of the commands above. Here is the correct output of df -h:


Filesystem                Size      Used Available Use% Mounted on
/dev/root               117.7G      1.2G    116.5G   1% /
tmpfs                     1.9G      1.3M      1.9G   0% /tmp
/dev/sda1                15.7M      4.8M     10.7M  31% /boot
/dev/sda1                15.7M      4.8M     10.7M  31% /boot
tmpfs                   512.0K         0    512.0K   0% /dev
root@FW6B:~#
4

I don't see why there are 2 entries for /boot and why / is mounted on the /dev/root and not on a block device.
What is the output of uci export fstab ? <-careful with this, not part of the command

5 
root@FW6B:~# uci export fstab
uci: Entry not found

Do you think this could happen because I did something wrong while creating the image in image builder? Sorry I'm a novice in this as well...

6

@trendy ,

Here are the steps I followed to create the image with Image Builder. I originally wrote this to remind myself of what to do, because I'm not a linux user on a daily basis and I have a tendency of forgetting things I don't use often. I also wrote this in my native language, so there could be some errors in the text below, caused by google translator. I read it briefly before posting here, and it looked ok to me, but you never know with machine translations:

On an Ubuntu virtual machine containing the OpenWrt ImageBuilder image generation folder for x86/64:

a) download the most updated version of OpenWrt ImageBuilder on the Ubuntu Desktop and unzip it with the command tar -J -x -f <filename.tar>, which will create an image generation folder, named <name of the file>.

Example: tar -J -x -f openwrt-imagebuilder-21.02.0-x86-64.Linux-x86_64.tar.xz
To create the folder: ~/Desktop/openwrt-imagebuilder-21.02.0-x86-64.Linux-x86_64

b) In the image generation folder, created in the previous item, create a folder ./files

c) Inside the folder created above, create a files/etc subfolder, a files/opt subfolder and a files/root subfolder.

d) Inside the files/etc folder created above, create a files/etc/uci-defaults subfolder.

e) Copy the files from the files folder on the Mac to the files folder created above in item b).

f) Then copy the contents of the etc folder to /files/etc and the contents of the etc-uci-defaults folder to files/etc/uci-defaults. Then copy the contents of the opt folder to files/opt. Finally, copy the contents of the root post to the files/root folder.

g) Execute the image generation command, from the unzipped folder in item a):
make image PROFILE=generic PACKAGES="luci grep procps-ng-pkill coreutils-split nano dnsmasq-full ddns-scripts ddns-scripts-noip luci-app-ddns rng-tools openssh-sftp-server curl wget ca-bundle ca- certificates cfdisk kmod-sched-mqprio tcpdump luci-app-wireguard sqm-scripts-extra luci-app-sqm bash vpn-policy-routing luci-app-vpn-policy-routing mwan3 luci-app-mwan3 -dnsmasq -odhcp6c -odhcpd -ipv6only luci-proto-ipv6" FILES=./files

NOTE: You can pass the following additional parameters to increase the size of default partitions: CONFIG_TARGET_KERNEL_PARTSIZE=128 CONFIG_TARGET_ROOTFS_PARTSIZE=512

h) copy generated files to an auxiliary USB drive which will be used as a temporary drive to transfer the image to the x86/64 router.

i) Boot the x86/64 machine where OpenWrt will be installed using a USB flash drive containing the Ubuntu Live CD as the boot drive.

j) After Ubuntu Live starts, copy the compressed image from the temporary USB drive to the Ubuntu Desktop.

k) Unzip this image on the Desktop, using the gunzip command.

l) Copy the uncompressed image on the Desktop to the internal drive, using the following command:

   dd if=<unzipped image> of=/dev/<sdx>.

   Where /dev/sdx corresponds to the internal drive of the x86/64 machine (usually /dev/sda), from which OpenWrt will be launched in the future.

m) Boot the router and select the internal drive as the boot drive. Since OpenWrt is now present on the internal disk, it will start automatically.

n) After the newly installed OpenWrt starts correctly, run the ./stl.sh script (this is to add static leases do dnsmasq's dhcp file).

o) switch to the /opt/AdGuardHome folder and execute the command ./AdGuardHome -s install

p) restart the x86/64 router and verify that WAN, WANB and WG0 connections exist in Luci/Network/Interfaces. Correct if necessary.

q) check mwan3 service functioning in Luci/Network/Load Balancing. Remove ipv6 interfaces and policies dependent on those interfaces. Activate wanb. Correct if necessary.

r) check the operation of the Wireguard VPN and the vpn-policy-routing service in Luci / VPN / VPN Policy Routing. Correct if necessary.

s) check the operation of opkg update.


NOTE) It may be necessary to manually restart VPN=Policy-Routing with each reboot.
7

I am not so sure if those two are connected to the problem. I'll let someone else to chip in and help you identify the problem.

1 Like
8

I too have two /boot entries listed in mount, so it shouldn't be the cause of the error.
/dev/root on / is also there.

2 Likes
9 
logread | sed -n -e "/- preinit -/,/- init -/p"
10

Hi @anon50098793 ,

Thanks for chiming in!


(root@FW6B:/etc/config# logread | sed-n-e "/preinit-/,/init-/p"
root@FW6B:/etc/config#

Nothing comes out of the command...

11

copy paste the command above exactly as it is.

1 Like
12

Still empty results.

I had not notice before that the copy and paste operation from the terminal screen somehow changed the spacing in my previous posts. Below you can find a screen grab, which did not suffer from the same issue:
Screen Shot 2022-02-20 at 19.17.43

13

needs to be done after a reboot...

14

Thanks, @anon50098793

Here is the output of the command:

Screen Shot 2022-02-21 at 07.44.58
Screen Shot 2022-02-21 at 07.44.58994×440 17 KB

1 Like
15

i've got no idea whats going on... i mean... you can boot your ubuntu live again and do an fsck... but openwrt will usually spit you into a 'ro' rootfs if the filesystem is that bad (which does not seem to be the case here based on diagnostics above)

your symptoms remind me of sdcards with the read-only switch or a laptop I recently had to 'fix' that had the bootdrive backed by some weird 'optane' drive.... as the primary drive was failing... i'd re-install and the old data would still be there (i think your actually supposed to do something special with the optane when you re-install but bios was pretty locked and unclear on what to do)...

in other words... I see nothing obvious about what could be going wrong other than knowing a little bit more about the drive hardware specifically...

16

Thanks. Since it's about to land, I think I will wait 21.02.2 to come out to rebuild the image from scratch. I hope this fixes this strange problem. The disk is a mSATA SSD. If believe the BIOS/UEFI is standard American Megatrends or something. I will post back here when I take the system out to deploy the newer image.

1 Like
17

you could try installing hwinfo ... it prints alot of stuff but usually should show you some good things about the hardware...

1 Like
18

I also thought that something was read-only, but the changes I make using the interface do get written to disk, as I verified the files in /etc/config. The problem occurs with reboots, something is rolling back the changes and making the modified files go back to the previous state... Never saw such behavior. Investigating it, I also verified that my 99_OpenWrtConfig.sh script was not present in /etc/uci/defaults which would explain settings reverting to the state after first boot, but no, the file was not there, meaning it was correctly deleted after first boot.

19

yeah... when you say adguard goes back to an old version... nothing really explains that... at least unless you have some automated boot backup script thats backing up the whole adguard binary to /boot/sysupgrade.tgz which would mean it's restored every boot...

there is no overlay or extroot so simple there...

which pretty much just leaves us with a disk level oddity...

20

Yes, I also thought that some newer version of AdGuardHome (been updating it regularly from it's interface) may be causing this issue, because the last update to 108b3 did not "stick". So I guess something in one of it's previous version may have done something to the disk which I did not notice until the power cut happened.

1 Like