Required packages for image builder and firmware selector broken for the last few days?

FanOfOpenWRT · February 25, 2025, 6:11pm

Apologies if this has already been noted or discussed... Couldn't find any mention in the forums except for previous incidents.

Edit: Is it my package selection causing the problem? (I add luci-app-statistics collectd-mod-thermal nano to the build every time.)

Edit 2: No, it's not my package selection; I reverted to defaults and it still errors.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r28821-9fb44638ba
 -----------------------------------------------------
root@OpenWrt:~# owut check
ASU-Server     https://sysupgrade.openwrt.org
Upstream       https://downloads.openwrt.org
Target         qualcommax/ipq807x
Profile        linksys_mx4300
Package-arch   aarch64_cortex-a53
Version-from   SNAPSHOT r28821-9fb44638ba (kernel 6.6.77)
Version-to     SNAPSHOT r28896-7b401a9d95 (kernel 6.6.79)
16 packages missing in target version, cannot upgrade
1 packages were downgraded
95 packages are out-of-date
Package build failures affect this device: details at
  https://downloads.openwrt.org/snapshots/faillogs/aarch64_cortex-a53/

ERROR: Checks reveal errors, do not upgrade

brada4 · February 25, 2025, 6:52pm

How imagebuilder is broken when it clearly says some of your packages failed to build inviting you to examine build logs?

Maybe use stable release to at least 24.10-SNAPSHOT to avoid daily evolving build/packaging system?

FanOfOpenWRT · February 25, 2025, 7:00pm

You may have missed the second edit.

This seems like the "wait a bit for packages to catch up" recurring issue, just confirming it affected everyone and not just me.

I updated the title of this thread to indicate it's the packages, not the image builder's fault.

brada4 · February 25, 2025, 7:10pm

Nobody gets notifications on your random edits and in general they get ignored.
If collectd was not built (due to broken autotools) your totaly non-defult package selection will not upgrade.
You are supposed to get around trivial issues if using bleeding edge snapshot on your spare routers.
Cannot stress more that 24.10.0 has 1-2 build failures, mostly in very exotic corners if ever. owut check -V 24.10-SNAPSHOT

belarios · February 27, 2025, 5:18pm

I'm in the same boat. I have an mx4300 which is not in stable yet, so relying on SNAPSHOT.

You can't custom build a router because nftables-json is part of fw4 and it is not available.

You can download a premade firmware that does have nfttables-json but there are a number of packages that you can't add after you install the premade successfully. For me that would be unbound, nftables and htop.

I guess we keep waiting and eventually those packages will be fixed.

Unless anyone has any other suggestions. Thanks.

brada4 · February 27, 2025, 5:54pm

Yes, that happens with snapshot now and then. Even release-snapshot sometimes include something broken, though significantly less often. Just wait it out (and hope for next stable release)

FanOfOpenWRT · February 27, 2025, 8:20pm

I was setting up a brand new Linksys E8450 and was able to use the pre-built SNAPSHOT OpenWrt image from Firmware Selector, but it either doesn't include Luci or my browser wouldn't let me connect to the non-secure Luci so I had to SSH to the device afterwards, run apk update and then apk add luci luci-ssl nano to get things rolling.

brada4 · February 27, 2025, 8:33pm

The build failures will happen, best is to go conservative with stable release and carefully curated package feed.

belarios · February 28, 2025, 5:07pm

Is it possible to request a rebuild of https://downloads.openwrt.org/snapshots/packages/aarch64_cortex-a53/

belarios · February 28, 2025, 9:09pm

Or I should be able to use packages from https://downloads.openwrt.org/snapshots/packages/aarch64_generic/ ?

hnyman · March 1, 2025, 6:01am

It has been tried to be rebuilt several times in the last few days, but it fails always...
https://buildbot.openwrt.org/main/packages/#/builders/1

Possible reasons are e.g.

an interactive prompt in uboot-mediatek since an upgrade week ago. This is already known by developers and @ansuel and @robimarko are looking for fixing it
a race condition between some packages, where the build of one package first creates a library noticed by the second package, triggering some config prompt in that. Looking at the build history of cortex a53, there is a "timeout bug" that has started already 4 months ago, much earlier than the recent uboot-mediatek problem. So, there is possibly something hidden deep in some package's build config options.

robimarko · March 1, 2025, 9:33am

It cannot be U-boot since that is nonshared and thus built in phase1 and not during packages

belarios · March 1, 2025, 2:41pm

Thanks for the info. It's interesting how it only affects this target and not the other similar ones.

hnyman · March 2, 2025, 8:28am

I did some debugging today. I suspect Node bump in Nov 2024...

github.com/openwrt/packages

Node 22 version bump with ABI possibly causing severe timeout problem in buildbot

opened 08:25AM - 02 Mar 25 UTC

hnyman

cc @robimarko @Ansuel @nxhack @ianchi We have a frequent timeout/stall problem… in the packages buildbot, which timeout is destroying quite many builds due to a hangup like `failed 'make -j12 ...' (failure) (timed out`. * https://buildbot.openwrt.org/main/packages/#/builders/1 * https://buildbot.openwrt.org/main/packages/#/builders/4 * https://buildbot.openwrt.org/main/packages/#/builders/37 ``` make[3] -C feeds/packages/admin/zabbix compile command timed out: 3600 seconds without output running [b'make', b'-j12', b'IGNORE_ERRORS=n m y', b'BUILD_LOG=1', b'CONFIG_AUTOREMOVE=y', b'CONFIG_SIGNED_PACKAGES='], attempting to kill process killed by signal 9 program finished with exit code -1 ``` About 1/3 of the builds in the affected targets end in timeout. As it does not affect all builds of the same target architecture and 2/3 times the build succeeds, it is likely some kind of concurrency/race problem, so that the building order of the packages (or submodules) affects the features detected by a second package, causing a config prompt, or something like that. That seems to have started approx 3 months ago. The oldest failures that I have spotted are from around `November 23, 2024` * https://buildbot.openwrt.org/main/packages/#/builders/36/builds/30 * https://buildbot.openwrt.org/main/packages/#/builders/3/builds/27 The failures happen on aarch64, arm, i386, x86 But not on armeb, arm_xscale, mips, powerpc, loongarch It is hard to figure out what is happening, as the buildbot compile step logs available for casual users is just the launch of each package's compilation. And due to concurrent building, the packages are built in sligthly different order each time, so there is not direct diff possibility of the 4000+ line logs. However, I did debugging by sorting the compile step output, and then comparing from the same target a recent ok build. I noticed that from both analysed targets (x86_64, arm8vfpv3), the exact same package lines were missing from the timeouted build: ``` *** 1164,1183 **** make[3] -C feeds/packages/lang/node clean-build make[3] -C feeds/packages/lang/node compile make[3] -C feeds/packages/lang/node host-compile - make[3] -C feeds/packages/lang/node-arduino-firmata clean-build - make[3] -C feeds/packages/lang/node-arduino-firmata compile - make[3] -C feeds/packages/lang/node-cylon clean-build - make[3] -C feeds/packages/lang/node-cylon compile - make[3] -C feeds/packages/lang/node-hid clean-build - make[3] -C feeds/packages/lang/node-hid compile - make[3] -C feeds/packages/lang/node-homebridge clean-build - make[3] -C feeds/packages/lang/node-homebridge compile - make[3] -C feeds/packages/lang/node-javascript-obfuscator clean-build - make[3] -C feeds/packages/lang/node-javascript-obfuscator compile - make[3] -C feeds/packages/lang/node-serialport clean-build - make[3] -C feeds/packages/lang/node-serialport compile - make[3] -C feeds/packages/lang/node-serialport-bindings clean-build - make[3] -C feeds/packages/lang/node-serialport-bindings compile make[3] -C feeds/packages/lang/node-yarn host-compile make[3] -C feeds/packages/lang/perl clean-build make[3] -C feeds/packages/lang/perl compile ``` The node main compilation is started and also node-yarn host-compile gets started (as the first node module?). But then there is no trace that compiling other modules ever starts, until a timeout kills the whole buildbot build round. So, my guess for the reason is https://github.com/openwrt/packages/pull/25435 : `node: upgrade to 22.11.0 LTS` on 23 Nov 2024 , which commit in addition to the major version bump, also added ABI versioning to node modules. Node is restricted with ` DEPENDS:=@HAS_FPU @(i386||x86_64||arm||aarch64)` to build on the affected targets, which increasingly points out to node being the reason for the major timeouts. So for some reason, the node builds likely fails 1/3 of the times, but succeeds 2/3. Curious. Sorted logs: [sort x86 ok stdio.txt](https://github.com/user-attachments/files/19042166/sort.x86.ok.stdio.txt) [sort x86 error stdio.txt](https://github.com/user-attachments/files/19042165/sort.x86.error.stdio.txt) Original: [x86 ok stdio.txt](https://github.com/user-attachments/files/19042164/x86.ok.stdio.txt) [x86 error stdio.txt](https://github.com/user-attachments/files/19042167/x86.error.stdio.txt)

Ps. cortex a53 finally managed to succeed last night.

robimarko · March 2, 2025, 10:13am

Awesome, it took like 4 attempts to get rebuilt, but Nginx for some reason is still falling with the broken patch (Which I fixed couple of days ago) but locally when using the SDK it builds just fine

belarios · March 2, 2025, 1:52pm

Successful build from firmware-selector and successful flash.

Thanks much.

system · March 12, 2025, 1:52pm

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.