Build reproducibility - definitive guide available?

Trying to reproduce build(s) and failing, and attempts take quite a while, so hoping someone can point to a definitive working example, of 19.07.7 would be nice (as that's running on my r6350), but current 21.02.0-rc3 might be acceptable as well (altho I have not yet tried that build on my device)...
More detail of my efforts...
For a netgear r6350, I have been trying to locally produce a build from source that I can determine is verifiably the same as a release build by comparing sh256sum values of what I build to those posted with the release, but withOUT success.
I have tried by 19.07.7 and 21.02.0-rc3, both without success.
I have found various pages with posts dating from 2018 into 2021 (.seed ==> .buildinfo changes, etc.), and the build system page, and so forth.
No where have I found a step-by-step for any given specific build that might be followed to success.
Is there one I've overlooked?

The steps I have synthesized from the various posts are contained in the following:

git clone
cd openwrt
# tag
git checkout v${TARGVERSION}
# create local branch
git checkout -b v${TARGVERSION}

set -x
curl -O${TARGVERSION}/targets/ramips/mt7621/config.buildinfo
curl -O${TARGVERSION}/targets/ramips/mt7621/feeds.buildinfo
curl -O${TARGVERSION}/targets/ramips/mt7621/version.buildinfo

cp config.buildinfo .config
cp feeds.buildinfo feeds.conf

set +x

./scripts/feeds update -a
./sripts/feeds install -a

diff version version.buildinfo

make defconfig

make download world

make V=s -j4

(I have tried some variations as far as 'make defconfig' or not (when trying steps manually) but so far always with failure to reproduce, equal sha256sum values.)

Where is a correct example and/or what am I doing wrong in the above?


(I also saw mention that perhaps matching sums might not be possible without exact same toolsets used to perform archive construction/compression, but wasn't sure if I was correctly understanding those comments.)

Maybe you are not using the same options as the buildbots making the release builds?

make menuconfig

Global Build Settings

Set build defaults for automatic builds (e.g. via buildbot)

should set the config to what the buildbots use (and will take a while to compile because it's compiling a bunch of kmod packages as modules)

Thanks for replying.

I thought obtaining the conf.buildinfo from the release I'm attempting to duplicate and copying that into '.config' was supposed to assure that I was using the same options... ?
And likewise, obtaining the feeds.buildinfo and copying that into feeds.conf, ensuring the same source(s) would be accessed...

Quite possible of course that I'm "thinking wrong"...

Hence my desire for a concrete working example to mimic...

@bobafetthotmail, have tried to follow your suggestion and what I found is that already seems to be selected.
" [*] Set build defaults for automatic builds (e.g. via buildbot)"


netgear_r6350-squashfs-factory.img	c69fbcaab02f4bfee2fe261a64e8f5b002ba4a00653c7cedddd19fe7fb71f8eb	6100.4 KB

Having used basic script above with TARGVERSION=21.02.0-rc3, different results as below

sha256sum ./bin/targets/ramips/mt7621/openwrt-21.02.0-rc3-ramips-mt7621-netgear_r6350-squashfs-factory.img
459a9dd83c5980f176f6d0da2aafc61f82857bf622dc1f4c3e2a3462c7994417  ./bin/targets/ramips/mt7621/openwrt-21.02.0-rc3-ramips-mt7621-netgear_r6350-squashfs-factory.img

In middle of that build effort, did have following WARNING: (with some context):

 make[2] -C package/utils/px5g-wolfssl download
 make[2] -C package/utils/spidev_test download
 make[1] target/download
 make[2] -C target/linux download
WARNING: your configuration is out of sync. Please run make menuconfig, oldconfig or defconfig!
 make[1] world
 make[2] tools/compile
 make[3] -C tools/flock compile

Looking back, I see one of changes I had made from 'basic' script above commented out the 'make defconfig', perhaps that is cause for the WARNING, trying again with that enabled, also removed the 'world' leaving just 'make download' before actual make.

Maybe this document about the Linux kernel reproducibility could explain part of why it's not possible to easily recreate an identical bit-for-bit firmware, at least when it comes to the Linux kernel part, unless you take care of overriding each of those "external references".

For reference purpose, would you have a link to that mention?

(I had been hunting for myself as well, but just now finally found it,
tho I'm not sure if its referring to the hash I'm trying to match...)

possibly stupid question: if the issue is the file compression, try comparing the extracted binary files

Those should have the same hash, if all went well during compile

reasonable stupid question, can explore that, are the .img/.bin files compressed archives, guess I can figure that out...

Also occurred to me that I can try building with the same steps twice and see if I get results with matching hashes - if not then I can conclude that the build system in its current state canNOT reproduce bit-for-bit duplicated results, unless there are simply other options I'm not yet aware of and utilizing. (The page linked by Fiouz on kernel reproducibility certainly casts doubt on the possibility of total reproducibility.)

packages are gzip archives and can be uncompressed, that's the easier thing to check.
See in your device's feeds where it's pulling packages from exactly
or search for it in the target list
and download them manually from here

then you ungzip them down to the bare binaries and see if both official ones and your own build have same checksum

I suspect that img files are full system images and may contain other stuff like release keys or signatures that are not included in a build you make on your system, since you are not OpenWrt official buildbots see
If that's the case then it explains why the firmware image in itself is not reproducible

From 2018 there's also this comment
"Measuring 'Reproducible Build' seems elusive"
on this page

diff the package version list (manifest) and compare the vermagic...

anything else as mentioned above will either contain artifacts or symbols

if you want to get technical... match the host binaries to the buildbots...

Have now done same steps via script file utilizing same build environment (though different shell window instances) and did not get bit-wise same builds, many differences to be seen by comparing 'sha256sums' files from both build trees.

So unless there are other options I'm not using to apply to the fetch/build process, appears bit-wise identical build reproduction is not currently possible.

script file used contained:

git clone
cd openwrt
make distclean
git checkout v${TARGVERSION}
git checkout -b v${TARGVERSION}
curl -O${TARGVERSION}/targets/ramips/mt7621/config.buildinfo
curl -O${TARGVERSION}/targets/ramips/mt7621/feeds.buildinfo
curl -O${TARGVERSION}/targets/ramips/mt7621/version.buildinfo

cp config.buildinfo .config
cp feeds.buildinfo feeds.conf

./scripts/feeds update -a
./scripts/feeds install -a

diff version version.buildinfo

make defconfig

make download

#make V=s -j4

There seems to be continuous tests from the Reproducible Builds project covering OpenWrt.

After taking a look at some results:

(community packages are not in the scope of the original post, as it is referring to official releases, but it's worth noting for people building their own custom firmware)

Digging a bit in the commit history, there are also some improvements that are specific to the oncoming 21.02 release such as those commits for build path independence.

So it seems there is an interest in having bit-for-bit identical builds from core developers, as this commit from 2016 mentions:

The motivation for this option is to achive reproducible (bit-for-bit
identical) kernel builds of official openwrt releases.

Being unable to reproduce official release builds looks like a bug/regression (or a work-in-progress).