[Solved] Sysupgrade to recent snapshot hangs on boot with "Waiting for root device /dev/ubiblock0_0"

syto203 · June 14, 2019, 2:11pm

I just tried to do sysupgrade from today’s file. Is there a way to revert to older firmare or stock without serial access?.
Also ssh is getting connection refused.

jeff · June 14, 2019, 2:24pm

@syto203 Would you describe the problem you’re having, if any. Sometimes what seems to be “the same problem” turns out to be something different.

I should be able to provide steps for that, I just need to know the state of your EA8300.

syto203 · June 14, 2019, 2:42pm

I triggered failsafe and reverted, tried reconnecting and it failed but it seems the problem was something local since renewing my connections fixed it.

My current build is r10199-04b45d3a31.
does current sysupgrade have issues or was it just a fluke.

Now I’m looking for a way to revert to stock, do I just flash the stock firmware, asking since you said the device already juggles between both firmwares.

jeff · June 14, 2019, 2:51pm

I don't yet know why there is an issue. I haven't been able to replicate it locally.

OEM firmware cannot be directly flashed by sysupgrade and, if there is some problem with sysupgrade, that wouldn't be a reliable option.

Right now, my priority is figuring out why there was a sysupgrade failure. Determining how to robustly guide a non-expert in flashing OEM firmware from running OpenWrt is lower priority.

Edit: How many times have you flashed OpenWrt? If only once, you should still have OEM firmware as the "other" firmware. You should be able to switch back to that with luci-app-advanced-reboot, or by examining the current value returned by fw_printenv boot_part and switching to the other firmware with fw_setenv boot_part N where N is 2 if it is presently 1, or is 1 if presently 2.

syto203 · June 14, 2019, 2:54pm

Ok good luck.

Edit: thanks for the extra info.
Advanced Reboot reports that "linksys ea8300 is unknown or isn't a dual partition device".
fw_printenv reports boot_part= 2.

jeff · June 14, 2019, 3:11pm

@eas -- If you have time, would you be able to run the script from Script: Mount "Alternate" NAND Firmware (Linksys +?) -- I called it mount_alt.sh on my machine.

sh -vx ./mount_alt.sh

will echo the line, as well as the interpolations. I'm looking for errors to understand what might be wrong. From my EA8300, it looked like

root@test:~# sh -vx ./mount_alt.sh 
#!/bin/sh

if [ "$( cat /sys/devices/virtual/ubi/ubi0/mtd_num )" = "11" ] ; then
	altpart=13
else
	altpart=11
fi
+ cat /sys/devices/virtual/ubi/ubi0/mtd_num
+ '[' 11 '=' 11 ]
+ altpart=13

mkdir -p /alt/rom
+ mkdir -p /alt/rom
mkdir -p /alt/overlay
+ mkdir -p /alt/overlay
mkdir -p /alt/firmware
+ mkdir -p /alt/firmware

ubiattach -m ${altpart}
+ ubiattach -m 13
UBI device number 1, total 680 LEBs (86343680 bytes, 82.3 MiB), available 0 LEBs (0 bytes), LEB size 126976 bytes (124.0 KiB)

ubiblock --create /dev/ubi1_0
+ ubiblock --create /dev/ubi1_0
mount -t squashfs -o ro /dev/ubiblock1_0 /alt/rom
+ mount -t squashfs -o ro /dev/ubiblock1_0 /alt/rom

mount -t ubifs /dev/ubi1_1 /alt/overlay
+ mount -t ubifs /dev/ubi1_1 /alt/overlay

mount -t overlay overlay -o noatime,lowerdir=/alt/rom,upperdir=/alt/overlay/upper,workdir=/alt/overlay/work /alt/firmware
+ mount -t overlay overlay -o 'noatime,lowerdir=/alt/rom,upperdir=/alt/overlay/upper,workdir=/alt/overlay/work' /alt/firmware
root@test:~#

eas · June 15, 2019, 4:58am

Yes, my router boots into the OpenWRT snapshot I installed around the beginning of June (OpenWrt SNAPSHOT r10121-06e63aa). The WDS client mode is flakey, as described elsewhere, and I want to confirm that some kernel patches applied to snapshot at the beginning of the week will fix the problem.

I do have serial access to the console, which is how I'm getting the logs I've posted. I haven't tried intervening with UBoot, yet, but the option is there.

As far as I know, the OEM firmware is at least partially gone, since after sysupgrade, it looks like a boots a newer kernel build (log shows June 12) before choking on trying to mount the non-existant new root fs.

Reiterating, I don't actually know what happened the first time it failed since I hadn't opened the case and connected a serial cable.

Where should I/we go from here?

I'm willing to try to gather more info to try and help figure out what went wrong in the first place, in case there is a problem that could trip up others in the future (I guess, at the very least, sysupgrade isn't handling exceptions properly).

If we've exhausted that avenue, I'd appreciate help getting back to a system that can be upgraded normally.

Thanks for working through this with me.

Update: missed this:

Yeah, I'll do it tomorrow (Saturday). I had to pack the router up today to make room for repairing the 2.4GHz WiFi ( and GPS and Bluetooth) on my wife's iPhone. Very tedious getting in deep enough to replace the suspect parts. It would have been a real bummer if it didn't fix the issue, but signs are good. I'll pull the EA8300 out tomorrow.

jeff · June 15, 2019, 5:09am

OK, I think I literally just replicated it here, or at least parts of it.

Yep

[    2.448095] ubi0: background thread "ubi_bgt0d" started, PID 85�[    2.464144] Waiting for root device /dev/ubiblock0_0...

Problem seems to be flashing over OEM -- at least I can replicate it here now.

Now that I can replicate it, please feel free to "fix up" your other partition.

I'd flash an (OpenWrt) factory.img using U-Boot and TFTP as the easiest way.

Which was last slated to boot (and will be next time)

boot_part=2

TFTP parameters; IP addresses can be changed just by setenv -- no save required, and will revert on boot if not saved (I only change them for "right now" and don't save)

image=dallas.img
ipaddr=192.168.1.1
netmask=255.255.255.0
serverip=192.168.1.254

then

run flashimg

or

run flashimg2

depending on the one you want to flash.

If that's not enough detail, please let me know.

Edit:

You could, for example, if running off firmware 1:

Flash firmware 2 with OpenWrt
Boot existing, configured firmware 1
Sysupgrade to firmware 2, retaining your current settings

jeff · June 15, 2019, 6:51pm

I've found the problem and it can be resolved prior to me deciding the best way to "fix" the upstream source.

What happens is that the OEM firmware has already taken over the root file system partition in a way that sysupgrade, right now, can't take it back. It can be resolved "by hand" using the command line (over SSH) relatively easily.

Step 1) Determine which partition you've booted into

root@test:~# cat /sys/devices/virtual/ubi/ubi0/mtd_num 
11

If this is 11, then you will be working with 13 in the later steps, if it is 13, then use 11

Step 2) Connect the "other" firmware to a UBI device

root@test:~# ubiattach -m 13
UBI device number 1, total 680 LEBs (86343680 bytes, 82.3 MiB), available 0 LEBs (0 bytes), LEB size 126976 bytes (124.0 KiB)

If this does not say UBI device number 1 here, stop now and ask for clarification.

Step 3) Confirm that there is an OEM volume there that needs to be removed

root@test:~# ubinfo -d 1 -a
ubi1
Volumes count:                           1
Logical eraseblock size:                 126976 bytes, 124.0 KiB
Total amount of logical eraseblocks:     680 (86343680 bytes, 82.3 MiB)
Amount of available logical eraseblocks: 0 (0 bytes)
Maximum count of volumes                 128
Count of bad physical eraseblocks:       0
Count of reserved physical eraseblocks:  40
Current maximum erase counter value:     1
Minimum input/output unit size:          2048 bytes
Character device major/minor:            246:0
Present volumes:                         0

Volume ID:   0 (on ubi1)
Type:        dynamic
Alignment:   1
Size:        636 LEBs (80756736 bytes, 77.0 MiB)
State:       OK
Name:        ubifs
Character device major/minor: 246:1

If is says Name: ubifs then proceed. If not, recheck your steps, especially which partition you've booted into, and don't proceed if you aren't seeing Name: ubifs

Step 4) Remove the OEM ubifs volume

root@test:~# ubirmvol /dev/ubi1 -N ubifs

You should be able to use sysupgrade now.

Patch submitted http://patchwork.ozlabs.org/patch/1116472/

Downloading https://gist.github.com/jeffsf/cd516fb84d0a46f1c3f9d49f2abea7ad and replacing /lib/upgrade/linksys.sh with it should give you the effect of that patch on your local, running install.

eas · June 16, 2019, 6:58am

FWIW, the output when run on my ea8300:

Test Script Output

#!/bin/sh

if [ "$( cat /sys/devices/virtual/ubi/ubi0/mtd_num )" = "11" ] ; then
	altpart=13
else
	altpart=11
fi
+ cat /sys/devices/virtual/ubi/ubi0/mtd_num
+ '[' 11 '=' 11 ]
+ altpart=13

mkdir -p /alt/rom
+ mkdir -p /alt/rom
mkdir -p /alt/overlay
+ mkdir -p /alt/overlay
mkdir -p /alt/firmware
+ mkdir -p /alt/firmware

ubiattach -m ${altpart}
+ ubiattach -m 13
UBI device number 1, total 680 LEBs (86343680 bytes, 82.3 MiB), available 0 LEBs (0 bytes), LEB size 126976 bytes (124.0 KiB)

ubiblock --create /dev/ubi1_0
+ ubiblock --create /dev/ubi1_0
mount -t squashfs -o ro /dev/ubiblock1_0 /alt/rom
+ mount -t squashfs -o ro /dev/ubiblock1_0 /alt/rom
mount: mounting /dev/ubiblock1_0 on /alt/rom failed: Invalid argument

mount -t ubifs /dev/ubi1_1 /alt/overlay
+ mount -t ubifs /dev/ubi1_1 /alt/overlay
mount: mounting /dev/ubi1_1 on /alt/overlay failed: Invalid argument

mount -t overlay overlay -o noatime,lowerdir=/alt/rom,upperdir=/alt/overlay/upper,workdir=/alt/overlay/work /alt/firmware
+ mount -t overlay overlay -o 'noatime,lowerdir=/alt/rom,upperdir=/alt/overlay/upper,workdir=/alt/overlay/work' /alt/firmware
mount: mounting overlay on /alt/firmware failed: No such file or directory

root@ea8300:~# cat /sys/devices/virtual/ubi/ubi0/mtd_num
11
root@ea8300:~# ubiattach -m 13
ubiattach: error!: cannot attach mtd13
           error 17 (File exists)
root@ea8300:~#

~~Stopped, awaiting clarification~~
Update: Now I see that it was left attatched by your diagnostic script, and that the output looks right, so, I'll try sysupgrade tomorrow when I'm in front of the device and can capture the console output.

eas · June 17, 2019, 1:21am

And sysupgrade succeeded! Thanks for getting OpenWRT working on the EA8300 in the first place, and thanks for helping me get over this hump!

tmomas · June 17, 2019, 6:07am

@eas If your problem is solved, please consider marking this topic as [Solved]. (Click the pencil behind the topic...)

You can also mark the reply that solved your problem:
grafik

syto203 · June 17, 2019, 9:57am

so does this mean that the OEM firmware is already corrupt when i tried to sysupgrade and there is no need for it to remain OR is this the only way to be able to use sysupgrade?

however, if the OEM firmware is intact and i want it to remain an option without having to manually flash it, couldn't i just revert to it using fw_setenv and re-flash factory openwrt to get the latest fixes?

jeff · June 17, 2019, 11:56am

If you have run into this already, the other firmware is an OpenWrt kernel and an OEM root file system, which isn’t viable. Uploading a single file to the running OpenWrt is perhaps the easiest.

Flashing OpenWrt over itself, while possible, isn't something I’d generally recommend to a non-technical user. If they really wanted to potentially revert to OEM in the future, postpone that until you’ve got running firmware on both partitions and an actual need.

syto203 · June 17, 2019, 3:13pm

just to make sure I’ve understood this right.

I’ve lost the ability to boot into OEM firmware since sysupgrade tried to replace it and half succeeded by only replacing the kernel and not the root partition.
the only way to get back to stock would be via serial.
if I’m booted in OEM should’nt flashing over the “other” would effectively overwrite it even if it was openwrt.

note: I have a fully functioning openwrt firmware.

jeff · June 17, 2019, 3:34pm

Yes, it is likely that the "other" firmware version is unbootable at this time
You can revert to stock on either firmware version from the OpenWrt command line, no serial required
Yes, both OpenWrt and OEM firmware flash the other firmware version when "upgrading"

Reverting to OEM is accomplished by determining which version to overwrite, erasing the NAND, writing the proper firmware image, and, if needed, changing boot_part. Directions are not provided at this time as I have found that "old" directions tend to be followed blindly in the future, resulting in bricked devices.

syto203 · June 17, 2019, 3:58pm

so i can go ahead apply the patch and sysupgrade and if the need ever comes to return to stock it's possible thru command line.

some directions even if old would be better than since they can put you on the right track at least, but i do understand your point of view of taking it case by case.

thank you for your time.

@jeff linksys.sh isn't in /lib/firmware it is however in /lib/upgrade/ and in /rom/lib/upgrade/ so which one should i replace?

jeff · June 17, 2019, 5:35pm

My error -- /lib/upgrade/linksys.sh

The /rom/ is the ROM itself, which is "interesting" to see what was originally flashed to the device, or recover from inadvertent or unwanted changes, but there's no point writing to it. Changes there, while on the overlay (not on the flashed ROM), aren't reflected where the OS is looking for files.

eas · June 17, 2019, 6:52pm

Thanks, @tmomas . I'd looked for a way to mark the topic as solved and couldn't find one. I did mark the appropriate reply.

If I understand your suggestion regarding the topic/thread, I should edit the title text as I've now done?

system · June 27, 2019, 6:52pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.