Hello everyone! I just got out of a deep rabbit-hole of trying to figure out why my image builder stopped working all of the sudden.
On the 1st of August, buildbot deployed new changes to the arm_cortex-a7_neon-vfpv4 package set for 19.07, I assume for the coming launch of 19.07.8.
In this upgrade, libubus20191227 was removed from the global 19.07 branch of packages and libubus20210603 was added in it's place. However, the core packages at targets/ipq40xx/generic/packages where not re-built for the 19.07.7 branch of packages so they were still expecting libubus20191227 which does not exist anymore. This has now broken image builder for me for that version set for me, and as I see for others as well.
Is this expected behavior, where for four days the advertised image builder will not work?
I can't imagine this is acceptable though? The downloads page has been, for several days now, pointing people towards a now-broken release that downloads.openwrt.org shows as being the only "stable" release. Maybe at the very least there should be a warning of some kind, and a link to the newer release. I know there's a troubleshooting guide on the wiki that says "wait a few hours or a day" but that isn't really the same as "use the new version that hasn't been released yet".
I'm not trying to be overly mean, OpenWRT is an amazing project, and the Image Builder is a blessing. Anytime one sees a successful FOSS project, they got to be impressed. I would just hate to see the many people who are getting confused by this and think they're doing something wrong, as even a couple months ago, I certainly would have been confused myself.
fwiw... i added that note a few days ago... mostly to catch sluggish phase2 buildbots on critical dependency change (or just whacky upload states)... (with master or XYZ-SNAPSHOT in mind)
the phenomenon of stable dependancy breakages is relatively new... (and way beyond the scope of a normal imagebuilder guide)
and as best I can tell... only really understood by a handfull of people...
Ah okay, the wording of some of the earlier comments on this thread made me feel like this is more of a common occurrence that the community has lived with for quite a while.
For my curiosity, have the only dependency breakages been made by ABI-breaking updates between the minor versions?
probably should have said 'in an ideal world'... but I tend to agree with you... the sooner we know that those ib's are perma-toast or not... we should throw a notice on the wiki regarding this ( or just remove them )
The "normal" way to build is to build from the source code, which has worked all the time. You download/update the main and feed sources, and build everything, and that succeeds as all dependencies are current. Works also in the release branches.
The imagebuilder is a shortcut, cooking up a firmware from prebuilt binaries, so it is more fragile to package upgrades.
Two observations:
Somewhat more core package fixes are backported to the release branches than earlier.
Imagebuilder has got new popularity in the last few months (after 10+ years of minor usage)
The combination of those two items causes problems arising from the two-step build system (fixed target-specific packages, continuously rebuilt user-space vanilla packages) are surfacing more than earlier.
The move to the two-step build system was done with the LEDE split in 2016, and it tackled the inefficiency of building a full release (with all packages) once and then not upgrading anything for that release. That was the default upto Chaos Calmer 15.05.1 in March 2016. Upto that release, all packages downloads were fixed without any later upgrades.
The current system works mainly well, enables upgrading packages during the life of a release, but faces imagebuilder/opkg difficulties when certain ABI versioned packages are upgraded in a release branch after a release. It enables the majority of packages to get upgraded after a release, but is not perfect.
indeed... and proper / more use of 'abi-versioning' has shifted symptomatology from an installed and perhaps partly functional binary to a hard-fault at the opkg download stage...
fwiw... ipq8065 builds perfectly ok with default package list...
avoid selecting including 'iwinfo' or anything that may pull it in and you may have better luck
while i've not recommended this in the past... I suggest you make a local clone of all 19.07.8 repositories relating to your device (preferably from a mirror) when it is released, as an interim measure that will allow you to avoid such issues in the future...
My usual image builder workflow is run echo $(opkg list_installed | awk '{ print $1 }') to get all packages in the current image before upgrading with a custom sysupgrade image. I pass this to the imagebuilder PACKAGES option with make to install all packages. The package list posted in my reply is what opkg list_installed returned, should I remove libubox packages from the imagebuilder list and let them be installed via dependencies?
Thanks, that worked after removing any ABI package from image builder packages list.
I must have been lucky, it is the first time I've hit this as I've used every image builder through 19.07, because I have a lot of customisations and additional packages and it's just nice to be able to run your own custom sysupgrade with all the packages in the image so there's no need to opkg install anything after. I'll note to avoid lib ABI versioned packages in future usage going forward though.
I've added a small note to the Wiki around image builder and ABI packages, it seems like it is a good idea to not directly include these (even if provided by opkg list_installed) and instead let image builder install them via package dependencies. Just flashed the 19.07.8 I built without them in and can see from looking at the image, they were automatically installed with the required ABI versions at image builder make.