21.02.0-rc2 - Build Error on libubus20210215

I built the 21.02.0-rc-2 the other day with the image builder "openwrt-imagebuilder-21.02.0-rc2-mvebu-cortexa9.Linux-x86_64.tar.xz" for the WRT3200ACM and want to build it again today with a couple of changes but I got the following error:

Collected errors:

  • pkg_hash_fetch_best_installation_candidate: Packages for libiwinfo20210106 found, but incompatible with the architectures configured
  • satisfy_dependencies_for: Cannot satisfy the following dependencies for luci:
  • libubus20210215
  • opkg_install_cmd: Cannot install package luci.

My build command is:

make image PROFILE=linksys_wrt3200acm PACKAGES="luci luci-app-advanced-reboot block-mount kmod-usb-storage luci-app-samba4 luci-app-ddns luci-app-uhttpd luci-app-openvpn openvpn-openssl luci-app-statistics collectd-mod-thermal vsftpd-tls ca-certificates -dnsmasq dnsmasq-full luci-app-vpn-policy-routing wget curl hostapd-common wget-ssl -ppp -ppp-mod-pppoe"

I look at the problem package and it is now a new dated package:

"releases / 21.02.0-rc2 / packages / arm_cortex-a15_neon-vfpv4 / base /"
"libubus20210603_2021-06-03-a8cf6782-1_arm_cortex-a15_neon-vfpv4.ipk"

Am I doing something wrong or is it an issue with the Image Builder?

1 Like

read here for info

1 Like

There has already been an upgrade of ubus after rc2, just like you noticed:

That may have invalidated the dependencies on normal userspace packages that depend on ubus & libubus. Or at least the packages built by the phase2 packages buildbot may produce different ones than those included in rc2, and the target-specific packages built by phase1 images buildbot may be out of sync.

Your best bet is probably to download the current 21.02 HEAD imagebuilder, instead of the week-old 21.02.0-rc2

Alternatively, you might wait 2-3 days and see if the buildbot produced packages are again in sync.

Ps.
I get a bit confused why you talk about WRT3200ACM but look at the IPQ806x related packages download dir arm_cortex-a15_neon-vfpv4. You did that just for example???

1 Like

they are identical... rc1 = rc2 = SNAPSHOT(21) all share the same packages... (edit: ?)

Not quite true.

Yes, all non-target related packages are shared.
But there are some target-specific packages in "targets/.../.../packages/"
E,g, https://downloads.openwrt.org/releases/21.02-SNAPSHOT/targets/mvebu/cortexa9/packages/

Those inlucde iwinfo and libiwinfo which cause the problem for @FCS001FCS

  • libiwinfo from 21.02-SNAPSHOT:
Package: libiwinfo20210106
Version: 2021-01-31-4a32b33e-1
Depends: libc, libnl-tiny1, libuci20130104, libubus20210603
...

From rc2:

Package: libiwinfo20210106
Version: 2021-01-31-4a32b33e-1
Depends: libc, libnl-tiny1, libuci20130104, libubus20210215

EDIT:
That is a weird dependency chain, as non-target specific ubus & libubus have been updated, but the target-related iwinfo & libiwinfo is taken from the old rc2 build and expectd a different version of ubus.

3 Likes

yeah should have put a question mark after that statement and happy to be better informed...

kernel related dependencies will also not be identical due to the kernel being bundled within the IB...

I will write to the developer mailing list and propose that ubus would be marked to be target-specific. This is not the first time when this issue hits users, but that usually happens in master, not in the release branches.

EDIT:
https://lists.openwrt.org/pipermail/openwrt-devel/2021-June/035436.html
https://lists.openwrt.org/pipermail/openwrt-devel/2021-June/035442.html

5 Likes

Sorry, picked the wrong reference.

As far as I know releases are not rebuild so what is build is build, if something is wrong it is wrong to next release.
It is only the snapshots that are rebuild and updated more or less on a daily basis.

Only partially true.
Firmware images and target-specific packages for a release are fixed and are not rebuilt frequently, but the normal user-space shared packages are constantly rebuilt by the 21.02 buildbot. I was first hoping that the error might be among that kind of packages, and would correct itself in the next build round.

This package rebuild activity can be seen from https://downloads.openwrt.org/releases/21.02.0-rc2/
Note that

  • release targets have a dedicated directory (like /releases/21.02.0-rc2/targets/) and are fixed, but
  • release packages have a separate directory, which is common to all 21.02 releases (21.02.0-rc1, -rc2, final, ongoing 21.02 snapshots). Note the symlink from
    /releases/21.02.0-rc2/packages/ -> ../packages-21.02. There the normal packages are constantly updated and rebuilt, even during the release lifetime.

(and there are also the semi-hidden regularly updated 21.02-snapshot builds for images, not only packages)

If normal packages are updated, their mutual dependencies do get corrected in the next build rounds.

But as target-specific release rc2 packages are fixed and not rebuilt, they continue to depend on the outdated versions of normal packages if there is ABI specification in the normal package so that you want an exact version. That is the error now, and the rc2 imagebuilder will remain broken. (If the depdency would be to a non-ABI-versioned package, the dependency would not be tied to a specific version and things would be ok.)

My fix proposal attempts to prevent similar errors in future, by marking six "normal" ABI versioned packages as target specific, so that in future they are built together with the target specific packages. So the fixed release build will contain also them.

3 Likes

thanks for your work on this @hnyman

imagebuilder does seem to have picked up popularity over the last year or two due to 1G>cpu demand... chef/auc ... dynamism of landscape and higher need for 'master' (new devices, breaking support fixes etc.)

wondering what (if any) streamlining can be done at the buildbot level to better sync / snappify the stage1/2 process... (particularly upon release which you mention in the PR)

obviously... there are cost / competing demands on available resources and it's not a simple question...

but thought worthy of mention...

There aren't that often ABI breaking updates in the release branches, and my proposed changed will mitigate that.

(In the last few years, the iwinfo - libubus dependency has been the problematic one, at least 2-3 times including the current one)

2 Likes

As a frequent imagebuilder user, I have been wondering the same thing. It would be nice if shared packages were snapshotted together with the non-shared ones, but I can see how this would be costly on the mirrors.

As a user, I think the workable options are to either privately mirror packages around the time of releases, or abandon image builder and switch to full builds instead.

As a note, the 19.07.7 imagebuilder is currently also broken for wrt1200ac target, ironically due to https://github.com/openwrt/openwrt/commit/616fff2a94837f1372e78342b78edc7588a7bb9a (the package is not in shared anymore, and is not in non-shared yet / until 19.07.8 gets tagged...)

2 Likes

good points and thinking aloud (these are not directed at you)

i'm guessing over the last year... at least 100 users have resorted to this... (creates alot of load of infrastructure)

1G x 100 = 100G

alot of people aren't capable of running a buildroot...

chef libremesh right now just says...

asu.build.PackageSelectionError

no build log or indication why... this is with no luci / -luci on rc1 rc2

it's openwrt branded despite it's arguable officialism, and a public reflection of the brand...

I'm not sure if the following would be practical, but for the sake of brainstorming:

  • Continue to have the builders rebuild every package (because it's hard not to with buildroot), but do not immediately upload the built packages
  • After packages have been rebuilt, do some filtering by comparing them against the current published versions. If there are meaningful differences (as identified by a new version number, or modified dependencies), upload the new package (to be published on openwrt.org and mirrors). Otherwise, discard it assuming any binary differences are just the result of non-reproducible builds. (I'm worried that this filtering step might be the weak point of this proposal....)
  • Due to the filtering of packages before upload, the rate of package changes would be greatly reduced - most packages wouldn't change from one minor release to the next - and it might become feasible to create package snapshots for every release (linking the unmodified packages to a common location, so that the actual package files do not have to be duplicated).
1 Like
  • dont upload phase1 until the associated phase2 is complete
  • (partially?) prioritise phase2 based on recent phase1 completions (instead of round robin)

pleased to say for master at least... the situation has improved(considerably)... guessing phase2 worker allocation (3days->2max) and reversion of abi stuff (very few ib failures due to deps even when out of phase)

thanks to whomever optimized the setup...

du:20210715-1511 stg1:20210715-0429 stg2:20210714-0903 r17127-99a22d48f2  phaseout
du:20210716-0514 stg1:20210715-2050 stg2:20210716-0545 r17130-b1b16bddb0  phaseok
du:20210717-0435 stg1:20210716-0743 stg2:20210716-0545 r17131-479a2a90f7  phaseout
du:20210717-1841 stg1:20210716-0743 stg2:20210716-0545 r17131-479a2a90f7  phaseout
du:20210717-2117 stg1:20210716-0743 stg2:20210716-0545 r17131-479a2a90f7  phaseout
du:20210718-0309 stg1:20210716-0743 stg2:20210718-0402 r17131-479a2a90f7  phaseok
du:20210718-1242 stg1:20210718-0710 stg2:20210718-1124 r17143-c1a3eff3ac  phaseok
du:20210719-0313 stg1:20210719-0021 stg2:20210718-1124 r17147-7270d768da  phaseout
du:20210719-1448 stg1:20210719-0021 stg2:20210718-1124 r17147-7270d768da  phaseout
du:20210720-0443 stg1:20210719-1846 stg2:20210720-0333 r17160-32adbfc789  phaseok
du:20210721-0412 stg1:20210720-2248 stg2:20210720-0333 r17174-f7ab41acc9  phaseout
du:20210722-1505 stg1:20210722-0959 stg2:20210722-0656 r17181-b309248730  phaseout