Is there trouble building main snapshots?

[badulesia]

Hi.
Snapshots are built on a roughly daily average. For a few weeks, I have noticed some long delays between two consecutive builds, sometimes up to 5 days.
Actually mt7621 subtarget hasn't been updated for 5 days (also some other subtargets), and x86/64 for 8 days. This is rather unusual. Is there any trouble in the building process, an important bug to be solved in the code, or anything else explaining this?

[hnyman]

See my recent message about the problem...

• https://lists.openwrt.org/pipermail/openwrt-devel/2023-November/041769.html
  "Unbalanced prioritization in the images buildbot? (main branch deprioritized too much)

The release branches 22.03 and 23.05 are currently apparently prioritized over main/master, and as there has been (nonessential) new commits in them, the combined images buildbot is spending time in building the release branch head snapshot images.

• https://buildbot.staging.openwrt.org/images/#/grid
• https://buildbot.staging.openwrt.org/images/#/

[badulesia]

Crystal clear. Who said "as always" ?
Thank you.

[efahl]

@badulesia

"as always"

I've been waiting for a clean x86/64 snapshot to complete for like two weeks... Guess I'll have to finish up my new workstation and start building locally (which I've meant to do for many, many months anyhow).

[badulesia]

Houra, x86/64 has been updated today, and many other targets.

[efahl]

But the ASU server still seems behind. I just tried my usual update (x86/64, combined, squashfs):

$ auc -y
auc/0.3.2-1
Server:    https://sysupgrade.openwrt.org
Running:   SNAPSHOT r24194-e1aaa1defd on x86/64 (generic)
Available: SNAPSHOT r24315-ae500e62e2
Requesting package lists...

... a billion log lines deleted ...

+ rm -f /builder/build_dir/target-x86_64_musl/linux-x86_64/tmp/openwrt-0fd2832ef682-x86-64-generic-squashfs-combined-efi.img.gz.kernel
/builder/staging_dir/host/bin/grub-bios-setup: warning: Your BIOS Boot Partition is under 1 MiB, please increase its size..
cp: cannot stat '/builder/staging_dir/target-x86_64_musl/root-x86/boot/.': No such file or directory
...
Cannot open '/builder/tmp/.packageinfo': No such file or directory
make[3]: *** [Makefile:157: install] Error 1
make[2]: *** [Makefile:208: build_image] Error 2
make[1]: *** [Makefile:146: _call_image] Error 2
make: *** [Makefile:262: image] Error 2
Error: Error while building firmware. See stdout/stderr
Bad message (74)

and if I look in https://sysupgrade.openwrt.org/json/v1/snapshots/targets/x86/64/generic.json, I see a build date 11 days old.

{
   "arch_packages":"x86_64",
   "build_at":"2023-11-03T22:06:33.000000Z",

[hnyman]

@efahl

the ASU server still seems behind.

If I remember the earlier discussions right, ASU server updates its build database 2-3 times per day. It is not real-time.

At what hours is firmware selector updated?

I have checked some images timestamps and their availabilities (this morning and this afternoon) in the firmware selector: they were updated at 14:00 UTC. So the answer is: the firmware selector list of images is updated at 22:00, 06:00 and 14:00 UTC. Plus about 2 minutes in order for the script to be performed. Oh it's quite simple. When I visit a relative, I update his router with a custom build. If I read the image timestamp, and know when it will be available in the firmware selector, I…

Ps.

@efahl

Guess I'll have to finish up my new workstation and start building locally

Building from source by yourself is always advisable You always get a coherent image.

[efahl]

@hnyman

If I remember the earlier discussions right, ASU server updates its build database 2-3 times per day. It is not real-time.

Yup, but the ASU server file's date stamps are 5 days old (the ones in the x86 snapshot dir) and the contents point to a build 11 days old, hence my suspicion that something is amiss... (The last time I was able to auc-update a snapshot was Oct 24, I've been testing it daily since then, waiting for the blockage to clear, as it always does sooner or later.)

I've been trying to get a local instance of the ASU server running in an Ubuntu VM, but haven't had much luck there. Podman is giving me grief with Error: plugin type="bridge" failed (add): failed to find plugin "bridge" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin] and I just haven't dug deep enough to find what's up with that.

[efahl]

@efahl

The last time I was able to auc-update a snapshot was Oct 24

The stars (or maybe the CPU cycles) have aligned and my auc updates are working fine again. I just upgraded an x86 VM from 23.05.2 to snapshot without a hiccup.