Where did you find the r26805 version code, on downloads.openwrt.org or sysupgrade.openwrt.org? If sysupgrade gets behind downloads, then what you're seeing is the result.
But, Paul's recent switch to webhooks in the ASU server update.py seems to have pretty much cured that problem. (Which is why I asked about your specific target and device, so I can monitor its builds and see if it's having issues.)
I believe @ahuman was trying to get a custom build of the latest snapshot on https://firmware-selector.openwrt.org without "customizing" the "Script to run on first boot" field. Because the customization hasn't changed, he got back the same build he requested a week ago from cache. It happens to me all the time, so I just change the commented customization script, e.g. from # 1 to # 2. It is indeed annoying.
He probably saw the current available build version (26805) by looking at "About this build - Version" on that same firmware-selector page (which is sometimes lagging a couple of hours behind the latest version found on https://downloads.openwrt.org/snapshots/targets/ ).
@KrypteX is correct on all counts.
This is the device (it just has different names in EU and US):
Hmm, seems to be up to date right now.
$ owut check --device ipq40xx/generic:asus_rt-ac42u:squashfs
Board-name asus_rt-ac42u
Target ipq40xx/generic
Root-FS-type squashfs
Sys-type sysupgrade
Package-arch arm_cortex-a7_neon-vfpv4
Version-to SNAPSHOT r26809-7c9644a7b5 (kernel 6.6.35)
Build-FS-type squashfs
Build-at 2024-06-27T15:59:31Z
I'll add it to my watch list and see if that platform lags behind.
Here's a thought - disregard if I'm way off...
Is that the "regular" snapshot build with the "standard" set of included packages and no custom script? That might not be applicable for the issue of getting a cached copy of a per request custom build.
Since people report getting the same (cached) custom build from days prior when their repeated request has the same set of customizations, I wonder if the imagebuilder code that checks for the possibility of just fetching a cached copy is not actually taking into consideration if there is a new (snapshot) image version available or something to that effect.
Let me know if rewording/clarification is needed.
Hi Spence, all that I'm watching is the difference between the version_code values as published on the ASU server and on the downloads site, as this was historically a major issue. The downloads site should be the "source of truth" as it's all populated by the build bots at the end of a specific target build, then the ASU server picks up the changes and publishes them there.
The when some client (auc, owut, LuCI Attended Sysupgrade, Firmware Selector) submits a build request, they can pickup (or not!) the version code from either site and submit it (or not!) in the build request. If it is submitted, it becomes part of the hash key that becomes the signature for the build cache (basically hash = sum(packages requested) + version_code + other stuff).
If the signature doesn't get a cache hit, then ASU build server creates a new docker imagebuilder by downloading target/board/imagbuilder.tar.gz and tries to use that for the build. One of the first steps that the ASU server takes is to make sure that if the client explicitly specified version_code, that it matches what the imagebuilder is actually building. All that happens here: https://github.com/openwrt/asu/blob/main/asu/build.py#L72
If the version code was not specified, ASU server just carries on and gives you whatever it has on hand. This is what LuCI ASU does - no version_code, which is why sometimes it "works" when FS croaks on "incorrect version blah blah".
But, getting back to the question at hand after my wall-of-text preamble: The package versions play no role in any of this. They are not part of the hashing and thus the signature is identical regardless of whether they've been rebuilt. That may be the root cause of @ahuman's issue, I need to look deeper into this (if it's not deep enough already).
Anyhow, at the bottom is what I've been monitoring. I have an hourly cron job that grabs both the ASU server and download sites' metadata, and compares the version codes. If they differ, then that's logged. Back in Olden Times, like 2-3 weeks ago, the delays between updates were often many hours, sometimes days. Since Paul's "webhook commit" rewrite of ASU server's update code, this has dwindled to sometimes the builds I'm watching appearing once, on rare occasion twice in a row. This means that the updates are happening within the 60-minute window of my cron job, and probably a lot quicker than that for successful builds.
Example from my log just now "as=" is ASU server's data; "bs=" build server, then date and d=target-version should be obvious. Note here that the ASU server got ahead of the build server. Not sure how this is happening, but it is fairly common now, rather than the old behavior which was reverse and with big lags.
Well, I said a lot of stuff, I hope some of it makes sense and/or clarifies things.
2024-06-28T20:00:04+00:00 bs=r26809-7c9644a7b5 as=r26810-7c32295b00 d=x86/64-snapshots
2024-06-28T20:00:07+00:00 bs=r26809-7c9644a7b5 as=r26810-7c32295b00 d=ipq40xx/generic-snapshots
2024-06-29T00:00:02+00:00 bs=r26810-7c32295b00 as=r26811-007437c225 d=mediatek/mt7622-snapshots
LuCI Attended Sysupgrade
Itâs not clear how it works at all, one might say it doesnât work at all. As in his own settings...
Search on opening
Search for new sysupgrades on opening the tab
So it is in the ânon-SNAPSHOTâ version. For example, when accessing the plugin, it does not start automatically and does not understand that since the release of the official openWrt release, a bunch of packages have been updated in this release.
He simply checked that the version on the router corresponded to the version of the official build and wrote that everything was fine, everything was up to date, but this was far from the case.
If gasoline was poured into the car after purchase, this does not mean that it is always available and does not need to be filled until you buy a new car.))
It doesn't build again...
Generate local signing keys...
Generate local certificate...
Package list missing or not up-to-date, generating it.
Building package index...
Downloading https://downloads.openwrt.org/releases/23.05.3/targets/mediatek/filogic/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_core
Downloading https://downloads.openwrt.org/releases/23.05.3/targets/mediatek/filogic/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/base/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_base
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/base/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/luci/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_luci
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/luci/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_packages
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/routing/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_routing
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/telephony/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_telephony
Downloading https://downloads.openwrt.org/releases/23.05.3/packages/aarch64_cortex-a53/telephony/Packages.sig
Signature check passed.
Downloading file:packages/Packages
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/imagebuilder
Downloading file:packages/Packages.sig
Signature check passed.
Collected errors:
It's because it was in the middle of building the new packages. Wait a couple of hours and it'll be fine again.
There's a build issue with RAX120v2 where even the default packages will fail building.
The relevant last few lines which show a recipe error are the following:
Downloading file:packages/Packages.sig
Signature check passed.
Pseudo file "dev" exists in source filesystem "/builder/build_dir/target-aarch64_cortex-a53_musl/root-qualcommax/dev".
Ignoring, exclude it (-e/-ef) to override.
make[3]: *** No rule to make target 'netgear_rax120v2-initramfs-images', needed by '/builder/build_dir/target-aarch64_cortex-a53_musl/linux-qualcommax_ipq807x/tmp/openwrt-db6d11b5f2d3-qualcommax-ipq807x-netgear_rax120v2-squashfs-sysupgrade.bin'. Stop.
make[2]: *** [Makefile:247: build_image] Error 2
make[1]: *** [Makefile:153: _call_image] Error 2
make: *** [Makefile:310: image] Error 2-
The firmware selector is acting up again. Requesting a custom build (with the script trick) returns version 26870, even though after installing, it proves to be a newer version, e.g. 26899.
@efahl I've noticed @aparcar rewrote ASU a few days ago in https://github.com/openwrt/asu/commit/c10687bd5ac5aef0163af44a9942e76f053fa9d6 - does it have to do anything with this issue?
You may try a workaround described here: The OpenWrt Firmware Selector - #788 by nextgen-networks
I've already said that I tried the custom build (with the script trick), this is I think the 2nd time you mention this, as if I still didn't get it or something. The kind of caching you are thinking about (or probably assuming I'm running into) is not the cause here, since I always use a fresh, unique, never-before-used comment in the script area, e.g. # 82364872.
The real problem is that now if I ask for a fresh build, it comes back with the stuck-on 26870 version (supposedly) and also if I edit the file, at the bottom it contains the 26870 string, but after actually installing on the device, it's a newer version in fact: 26899. I have never seen this strange behavior ever before.
Anyway, it's getting ridiculous.
That's almost always due to the ASU server being out of sync with the build server, but usually ends with the dreaded "Received incorrect version ..." error before the build even starts, so apparently not the problem here.
The FS site was changed 3 weeks ago to use the build server's files for the version_code, which you'll see below are the ones that fall behind now: https://github.com/openwrt/firmware-selector-openwrt-org/commit/f38d80f4788dce869d5069b5666b1f10e4653b73#diff-345666fa6b7af5b970aa948a041a397de4b93c18271e08faa97ac3c90e632194R669
I don't know if Paul has actually deployed the new ASU framework from three days ago yet. I have not seen any difference in my monitors. (Oh, and I still don't understand what that new code is doing and/or doing differently, haven't dug through it or tried to run it yet.)
Here's the last four days of logs for the 8 different variants I'm watching: Every hour, look at build server and ASU server version_code for each of the 8 configs, log a line if they are different. Nothing jumps out at me, except that last line where there was no file on the ASU server, but that went away as soon as I tried to see what was up...
2024-07-08T00:00 bs=r20316-7b8ccbd37c as=r20323-86e290e1b6 d=x86/64-releases/22.03-SNAPSHOT
2024-07-08T01:00 bs=r20316-7b8ccbd37c as=r20323-86e290e1b6 d=x86/64-releases/22.03-SNAPSHOT
2024-07-08T11:00 bs=r23938-2a099d36a7 as=r23946-2a25de25fa d=x86/64-releases/23.05-SNAPSHOT
2024-07-08T22:00 bs=r26883-75cd4ef48d as=r26899-2bae9d04af d=mediatek/mt7622-snapshots
2024-07-09T00:00 bs=r26883-75cd4ef48d as=r26899-2bae9d04af d=tegra/generic-snapshots
2024-07-09T02:00 bs=r26883-75cd4ef48d as=r26899-2bae9d04af d=ipq40xx/generic-snapshots
2024-07-09T04:00 bs=r26885-2ded54972e as=r26899-2bae9d04af d=x86/64-snapshots
2024-07-09T05:00 bs=r26885-2ded54972e as=r26899-2bae9d04af d=x86/64-snapshots
2024-07-09T11:00 bs=r20323-86e290e1b6 as=r20336-47c917313d d=x86/64-releases/22.03-SNAPSHOT
2024-07-09T22:00 bs=r23946-2a25de25fa as=r23987-d53f1caeb0 d=x86/64-releases/23.05-SNAPSHOT
2024-07-10T09:00 bs=r26899-2bae9d04af as=r26912-3c95641366 d=mediatek/mt7622-snapshots
2024-07-10T10:00 bs=r26899-2bae9d04af as=r26912-3c95641366 d=mediatek/mt7622-snapshots
2024-07-10T13:00 bs=r26899-2bae9d04af as=r26915-7a96d36188 d=ipq40xx/generic-snapshots
2024-07-10T14:00 bs=r26899-2bae9d04af as=r26917-d92c42f469 d=x86/64-snapshots
2024-07-11T01:00 bs=r20336-47c917313d as=r20338-456fd63e8f d=x86/64-releases/22.03-SNAPSHOT
2024-07-11T09:00 bs=r23987-d53f1caeb0 as=r24003-0cdbbd8868 d=x86/64-releases/23.05-SNAPSHOT
2024-07-11T14:00 bs=r24003-0cdbbd8868 as= d=ipq40xx/generic-releases/23.05-SNAPSHOT
I need to dig into how the imagebuilders get uploaded to downloads from the build bots, and see when ASU gets informed of a new build vs when the metadata on downloads gets updated.
AHA. It is neither the ASU server nor Firmware Selector this time, it's the CDN mirrors are out of sync. I believe mirror-03.infra is the primary host in Germany to which the buildbots upload, and downloads is whatever fastly host is nearest you, so for me in SoCal, probably something in Los Angeles.
$ curl -s https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/profiles.json | jsonfilter -e '$.version_code'
r26912-3c95641366
$ curl -s https://mirror-03.infra.openwrt.org/snapshots/targets/mediatek/mt7622/profiles.json | jsonfilter -e '$.version_code'
r26928-238aa35c49
And now that I see this, I'm pretty sure it's responsible for a lot of bad behavior from various tools.
The FS uses downloads.openwrt.org to grab the "current" version_code, but the ASU build server uses the imagebuilder that comes directly from its local server, namely mirror-03.infra. So, ASU and its local mirror are up-to-date, while all of us who use CDN mirrors via downloads are behind the times and we see wackiness when we tell FS -> ASU to build us the "current" version...
@KrypteX What's your gross geographic location? Outside Europe I'm guessing?
@efahl I'm not sure if that's the cause, let me explain why. I will reiterate in more details what is going on.
So, I go to https://firmware-selector.openwrt.org and select WNDRMAC v2 and SNAPSHOT and it will say the latest build is Version SNAPSHOT (r26899-2bae9d04af) - this 26899 is also what I'm seeing for at least 2 days under https://downloads.openwrt.org/snapshots/targets/ath79/generic/. OK, that's all nice and dandy.
Then I customize the script with a never-used-before comment, say #900, but when the build is finished, it jumps back to version 26870, see screenshot below:
I download the sysupgrade at the bottom of the screen, I check inside the file, at the end it clearly says version 26870 (not 26899), then I flash it on router. And surprise surprise, guess what LuCI reports? 26899! This has been going on for approximately 3 days. How do you explain that? Certainly not by having 1-2 hour delays between different build/asu/whatever servers.
Edit: Perhaps it's only a hiccup for the ath79/generic target. I've just checked a device under ath79/tiny and it builds a proper custom image 26928, same version as the one in downloads.openwrt.org
I'll wait for 26928 to finish building for ath79/generic as well, it's almost always the last one to complete the latest builds.
Edit 2: 26928 didn't finish and nothing else after it. The latest build is still 26899 for 4 days now, all other targets have newer versions built, at least once daily. So there's definitely something wrong going on with ath79/generic builds.
It also happened to MT7621 routers, last? week. e.g. DIR-882 rev.A1
Adding Adblock also caused it to fail build entirely.
Regarding ath79, it's an issue with that specific target, if you look at https://buildbot.openwrt.org/images/#/builders/219 you can see that all builds since at least a week by this point has been failing. It's been pointed out at https://github.com/openwrt/openwrt/pull/12959#issuecomment-2230897392.
Hopefully it's been fixed by https://github.com/openwrt/openwrt/commit/c85348d9abf4b89d37a7fa134cf6424bdf16787f but we got to wait for the builders to catch up to that commit, a day or two maybe?
Error: Impossible package selection
STDERR
Generate local signing keys...
Generate local certificate...
Package list missing or not up-to-date, generating it.
Building package index...
Downloading https://downloads.openwrt.org/releases/23.05.4/targets/mediatek/filogic/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_core
Downloading https://downloads.openwrt.org/releases/23.05.4/targets/mediatek/filogic/packages/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/base/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_base
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/base/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/luci/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_luci
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/luci/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_packages
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/packages/Packages.sig
Signature check failed.
Remove wrong Signature file.
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/routing/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_routing
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/routing/Packages.sig
Signature check passed.
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/telephony/Packages.gz
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/openwrt_telephony
Downloading https://downloads.openwrt.org/releases/23.05.4/packages/aarch64_cortex-a53/telephony/Packages.sig
Signature check passed.
Downloading file:packages/Packages
Updated list of available packages in /builder/build_dir/target-aarch64_cortex-a53_musl/root-mediatek/../../../../builder/dl/imagebuilder
Downloading file:packages/Packages.sig
Signature check passed.
Collected errors:
