Attended system upgrade broken

Oops, should have added, "No, owut is only on main SNAPSHOT, not on any of the release snapshots..."

https://openwrt.org/about/history#branch_logic

Thank You,
That's what i thought and tried. But i think i'm hitting the same issue rkboni is receiving with the incorrect version error with that command so when i read the above thread, it sounded like owut was only usable at the moment for that process because you can ignore a version check. I can easily be misunderstanding what i'm reading though. I don't fully understand the issue, but it sounds like so many changes are going on that maybe the build server and caching is out of sync causing auc command to fail?

The LuCI Attended Sysupgrade app does not specify a value for version_code, so maybe try that instead?

Here's a summary of the problem: you run one of auc, owut or Firmware Selector, all of which ask ASU server "What is the current version_code?" The request includes that version in its build request to the ASU server. Occasionally, ASU server's metadata (containing the version_code) and its imagebuilder are out-of-sync. This will cause the build request to error out without doing a build with the incorrect version... error. With auc, owut (until the next release) and Firmware Selector, you don't have a choice in this matter, these three clients send version_code and if it fails, you're done. LuCI Attended Sysupgrade client does not specify (or check) the version code, so sometimes it works when the others do not.

The root cause appears to be that there is a small time window between when the ASU server updates its metadata and when the corresponding imagebuilder is deployed, so if you do a build request inside that window, you (usually) get the "requested version" being newer than "build version". If the order is reversed (client requested something older than expected), that means the ASU server somehow missed an update and is working with old metadata, which is an ASU server bug of some sort.

In either case, ASU server could actually do a build and produce a working firmware (well, that's not always true, but almost, the build might fail due to missing packages or whatever). Hence my decision to change owut to allow ASU server to cough up whatever, download it and tell you "Hey! Something is wrong here!", but then let you decide if you want to proceed.

1 Like

Here's a summary of the problem: you run one of auc , owut or Firmware Selector, all of which ask ASU server "What is the current version_code ?" The request includes that version in its build request to the ASU server.

It would also be great if we could override that version code and specify our own so that when ASU says "nope, sorry moved on to vXXXX" (or "nope, sorry, still on vYYYY"), the user could override that with either vXXXX or vYYYY depending on the situation. But obviously that touches a lot more than just owut :person_shrugging:

1 Like

I appreciate all the explanation in this thread and your help. It's making more sense as i read it. I tried the Luci Attended Sysupgrade, but if advanced mode is disabled, it says i'm up to date and there are no options. If i turn on advanced mode, it says "new firmware upgrade available" but shows the current version in the drop down, but with a long list of packages. So there is no new image to pick, but it still lets me request an image. I haven't tried requesting a new image yet in advanced mode?

I cut the screenshot so i didn't have a huge screenshot, but i can post the whole image if it helps.

image

I think (haven't checked) that LuCI will try to do a new build of the same OpenWrt version, but with new package versions if you do that. On the other hand, I strongly suspect it might just hit the ASU server's cache, and see that all the work has already been done and return the previous build (at least for the week until the cache expires)...

I was thinking that same thing last night as I lay in bed having nightmares about version codes and mismatched build requests. I ended up reverting yesterday's change, which simply removed the version code. I replaced it with this commit that 1) selects the version code from the upstream servers instead of the sysupgrade server (hopefully more often up-to-date, leading to a better outcome); 2) adds a new --rev-code xxx option so you can specify the one that was expected in the image builder (which again hopefully will fix those that slip through); and finally 3) allow the special version code --rev-code none, which removes the version code from the build request altogether, as a last ditch attempt to get something working if you are truly desperate.

1 Like

Just thought i'd post. Luci upgrade also ended up with a checksum fail with advanced settings enabled. It attempted to build the image, it built the image, then on download the checksum failed.

On another note, When i was researching the use of these tools, i noticed that asu.aparcar.org has an invalid cert. It looks like a local generated cert for Kubernetes deployed back-end is in use.

image
image

I used your owut once already and really liked it!

How do I keep your tool itself up-to-date, and should it work better than ASU going forward?

$ opkg update && opkg install owut

^^ Will that update your script if already installed?

and then

$ owut upgrade

The wiki is very informative but doesn't contain specific instructions for updating your script itself.

Wow, haven't seen that one before. If it was a line error, maybe try again and see if it downloads properly?

Yes, that's Paul's test site, so I wouldn't trust it too much, it probably has unstable/experimental code running the server most of the time.

Exactly, just like any other package that's safe* to upgrade, that command line will upgrade if there's a newer version. Or, once installed, it upgrades automatically whenever you do a full upgrade... I'd suggest "don't bother", unless there is some new feature in owut that you are specifically looking for (the new version_code handling probably qualifies here, as it's sort of the topic of the thread - but it hasn't been built into the packages yet).

At this point, owut is still relatively immature so I've been doing releases weekly, but expect that to slow down as things get fixed and it's more stable.

* - "safe" in the sense that its underlying libraries are pretty stable so opkg won't accidentally brick your router by installing some incompatible binary. Doing blind opkg upgrade on some packages can actually (albeit quite rarely) brick things, so it's best not to upgrade without a good reason (and it's always safe to just do a full firmware upgrade).

1 Like

Not sure what caused the previous error earlier today, buy after trying again this evening, the firmware downloaded and flashed via luci. It's still the same version though. So i thought i would try to upgrade to snapshot with auc again, but i still get the incorrect version. At this point it seems i might take a break and visit again refreshed, or grab the firmware from other methods.

How does this look? https://openwrt.org/docs/guide-user/installation/sysupgrade.owut#installation_and_upgrading

2 Likes

Wonderful!

I'd even draw more attention to that final sentence about not needing to upgrade; I hadn't even realized that owut was a package just like luci, etc.

Is there a way to get 23.05-SNAPSHOT builds other than via auc / Attended SysUpgrade (and building them yourself)? I was hoping that one of https://firmware-selector.openwrt.org/ or https://downloads.openwrt.org/ would have the branch snapshots, but that doesn't appear to be the case.... auc still gives me version misatches, and Attended Sysupgrade only offers the same version as is on the router (23.05.4).

https://downloads.openwrt.org/releases/23.05-SNAPSHOT/targets/

EDIT:
AUC gives me version misatches too (ZTE MF286D):

Error: Received incorrect version r24023-07cb7cb885 (requested r24021-6edde2b502)
Bad message (74)

3 Likes

Well, hell, thanks! Didn't think to try that since it wasn't linked from the releases/ page :man_facepalming:

I wanted to post that today I was able to update to snapshot. I don't know if this was version specific, but for the Netgear WNDR3800 I was able to upgrade, And I did not get the version mismatch I was getting earlier. I assume this means other people that were getting the mismatch that it might be fixed as well?
Should have mentioned I was using AUC to go from 23.05 stable to 23.05 snapshot.

1 Like

Another upgrade using auc. But i thought of something. When using "Attended Sysupgrade" via Luci, the option to preserve settings is presented. With auc, i did not see this option. Then i looked at the awsome documentation provided by efahl on the usage of owut and i didn't' see a preserve settings option either. My last 2 upgrades using auc, it seems to preserve settings by default (like the luci variant also). Should there be a command to not preserve settings for either auc or owut to match gui functionality?

Hmm, maybe. Since auc has been removed from the packages, I'm pretty sure it won't get one.

owut could, but there's an easy workaround, just do owut download, if that works, then sysupgrade -n /tmp/firmware.bin. I'll add that to the docs (which reminds me I haven't added the --rev-code to the wiki yet).

2 Likes