A stable release is not really stable…

I have the same issue while installing unbound in a fresh v22.03.4 install on x86-64

Downloading https://downloads.openwrt.org/releases/22.03.4/packages/x86_64/base/libopenssl1.1_1.1.1t-2_x86_64.ipk
Collected errors:
 * opkg_download: Failed to download https://downloads.openwrt.org/releases/22.03.4/packages/x86_64/base/libopenssl1.1_1.1.1t-2_x86_64.ipk, wget returned 8.
 * opkg_install_pkg: Failed to download libopenssl1.1. Perhaps you need to run 'opkg update'?
 * opkg_install_cmd: Cannot install package libopenssl.

libopenssl seems to be missing on x86-64.

No issue on 2 TP-Link archer C5 v1.2.

22.03-SNAPSHOT build of x86-64 fails with the same error.

@jow @tmomas @hnyman

Do you know downloads[DOT]OpenWrt[DOT]org FailLog URL for 22.03.4 x86_64 ?

Not much there.

https://downloads.openwrt.org/releases/faillogs-22.03/x86_64/packages/

1 Like

I had already checked those logs and there was nothing there. I could understand the missing files for 22.03-SNAPSHOT, but somehow they got deleted from 22.03.4, which worked just fine two days ago: I built the image several times then with all the same packages.
The files for 22.03.4 were there originally, but not any more.

It is there now.

I suspect that that you were just hit with a bad timing, as buildbot was then uploading an updated version of the package (after first deleting the old one). Openssl got two bugs fixed two days ago, so a new version has been automatically compiled.

22.03.4 shares the vanilla packages with the ongoing 22.03 branch, so even release packages can disappear from the download server.

There seems to be a problem with the buildbot or download site, so that some buildbot workers take a lot of time in uploading the new packages to the download server.

Last x86-64 build took over 12 hours for uploading. (So there might have been 12 hours of "missing package" from deletion to the upload of the new one)

See the upload step in https://buildbot.openwrt.org/openwrt-22.03/packages/#/builders/32/builds/313

Older mention of that problem:

Cc @ynezz @aparcar

4 Likes

It would appear it is missing from the directory https://downloads.openwrt.org/releases/22.03.4/packages/x86_64/base where openwrt downloads the package from.

I’m in the same situation trying to install OpenVPN

Does not that defeat the purpose of a “stable release”? What you said means that every time I make an image of 22.03.4 with IB, I get a different result and different packages.

Is it possible to delete the superseded packages only after the upload process completes? That keeps the time that the packages are "missing" to as short as possible. It looks like rsync, so maybe using --delete-after instead?

that could be suggested to @ynezz @jow and other actual buildbot managers

1 Like

You are right, it is all working now. Although, it is unfortunate that the stable builds are not reproducible.

They are reproducible, if you build from the sources and git checkout the exact 22.03.4 tag with locked feed commits.

But the quick-and-dirty imagebuilder uses current 22.03 packages for the "vanilla" packages and only kmods etc. are locked to the release tag download repo.
See the split in https://downloads.openwrt.org/releases/22.03.4/

  • Direct link to the targets, images and kmods and sensitive packages
  • Symlink to ongoing 22.03 package builds for the general packages.

Why "dirty"? It is officially supported.

Same error with openssh-sftp-server and curl packages right now. And build stuck in error as I can see (https://buildbot.openwrt.org/openwrt-22.03/packages/#/builders/32/builds/314).

Compared to "build everything from source" it is a clear shortcut, as it uses ready-compiled packages, and the coherence of the packages is not similarly sure as with full compilation.

Phase 1 images buildbot builds kernel, kmods, kernel related ABI versioned packages and the core packages and compiles the basic firmware image. And it compiles the SDK. Build cycle is maybe once per day.
Phase 2 packages buildbot builds vanilla packages using the SDK from phase1. Build cycle is maybe once per two days.

It could well happen in the final image cooking with imagebuilder that packages are still built with the older SDK, while the core image and kernel have already changed and might cause different packages compilation in the next build round. Usually there is no problem, but sometimes there is. (e.g. we have seen breakages when iwinfo/libiwinfo has been upgraded by phase1 but some client packages from phase2 still expect the older ABI version)

Respectfully, you are hung up on explaining "how this currently works" instead of trying to understand end user's experience. It does not matter if a user is using an image builder or not. Here is a use case without IB: two different users download the same "stable" image, update their routers, reboot, and proceed to download all additional packages. If they do this several weeks apart, they end up with different firmware. Which might theoretically lead to different behaviours or stability issues even though they had most likely thought they ended up with identical firmware.
All I am saying is that OpenWrt "stable" releases are not actually stable, they mutate as time goes on and are not much different from 22.03-SNAPSHOOT. If that is the goal, I wish I saw it explained somewhere. It is what it is and I understand the supported and unsupported use cases now.

1 Like

I understand your point, but I think it’s comparable to installing the original stable release and finding the list of upgradable packages growing over time (e.g. opkg update; opkg list-upgradable).

1 Like

Well, there has been a lot of warnings in this forum not to blindly update all packages this way, because they can cause instability issues. I would bet I saw this warning on some official wiki page as well.
Yet, IB process is called “dirty” while it is actually doing exactly same as a regular user. Ironic, right?

It is hard for me to reconcile this warning with how stable releases get polluted with new packages: the process goes against the warning from the same folks :wink:

How is a buildbot issue related to soft bricking a device from installing software that's available?