Imagebuilder 19 - Packages rebuilt daily - why?

Hi all,

I have noticed that the openwrt packages under https://downloads.openwrt.org/releases/packages-19.07/ are reuploaded on a daily basis - that is, the file for a given package version is reuploaded every day, and the files have different sizes and checksums every time.

I was wondering why that is the case ?

Part of the reason I noticed this is that I am frequently building images, and was trying to avoid downloading the same packages over and again as I do so. However, I found out that I have to download the new package files every time I update the Packages file, as it references the size and hashes of every expected package file, and those keep changing on a daily basis...

What is the reason you are frequently building?

Maybe that will be considered odd, but I like to generate an image (using the imagebuilder) with my choice of packages and configurations built in. Basically I find that doing version control for my configs is easier on the build side than on the target devices.

So, what is the reason you are frequently building? And how often is frequently?

Varies. Sometimes I'll build a dozen images in a day as I'm experimenting with things, and sometimes I won't build any for a couple months. But I think this is getting sidetracked ?

Let me see if i'm understanding you right. You want to use image builder but you don't want to use the latest packages? Are you under some kind of download restriction?

I'm not under any severe download restrictions, but having local copies of the packages I use is still faster than downloading them every time.

For example, look at today's info for a random package:
Package: wireguard-tools
Version: 1.0.20191226-1
Depends: libc, libmnl0, ip
License: GPL-2.0
Section: opt
Architecture: arm_cortex-a9_vfpv3-d16
Installed-Size: 26528
Filename: wireguard-tools_1.0.20191226-1_arm_cortex-a9_vfpv3-d16.ipk
Size: 27229
SHA256sum: d487c8780ac8b4bc102824b901582d9ae71fd9392684552b4fd7c182a0668ca4
Description:

Compare with yesterday's info for the same package:
Package: wireguard-tools
Version: 1.0.20191226-1
Depends: libc, libmnl0, ip
License: GPL-2.0
Section: opt
Architecture: arm_cortex-a9_vfpv3-d16
Installed-Size: 26518
Filename: wireguard-tools_1.0.20191226-1_arm_cortex-a9_vfpv3-d16.ipk
Size: 27266
SHA256sum: d46ea05d8e2b9fb6d45c7208455cdbfd1f7f3185978f493080a6899167dfc2e0
Description:

This is the same package version in both cases, but today's ipk file is different from yesterday's. It's kinda unusual for distributions to re-upload new builds of the same package version, and I was wondering why openwrt does that, and if there is any reason why a user would want to get the new build of a package if the version number hasn't changed, and if there is any convenient way to not do that if there are no such reasons :slight_smile:

The reason is that the builders are rebuilding the feeds daily in order to provide updated binary packages in case something has been updated in the sources. The differing sizes and checksums hint at a problem with the reproducible build pipeline, they‘re likely caused by timestamp changes in the outer or inner ipk tar archive.

It would be useful to obtain a copy of two different variants of a package with the same version in order to compare them with something like vbindiff.

I suspect an unnoticed regression in the reproducible build handling.

3 Likes

Looked a little bit more into the ipk files.

Looking at two different builds of the same version of the wireguard-tools package:

  • The ./usr/bin/wg binaries within the data.tar.gz file are not identical (actually have different sizes). I'm not sure how to dig into that one, but it does look like reproducible builds are not working there.
  • The ./control files within the control.tar.gz file are not identical - they differ in the Installed-Size field. This seems like a direct consequence of the above.
  • The time stamps embedded in the control.tar.gz and data.tar.gz, as well as within the ipk file itself, differ between the builds.

Thanks for looking into the details. Can you provide me with the two copies of the two .ipk archives? I‘d like to investigate what is causing the wg binaries to differ