Build scripts that document changes and enable easy re-creation of build environment

hnyman · May 22, 2017, 8:36am

I have used a set of build scripts to build my community builds for WNDR3700 and later also for R7800. I thought to document the scripts here in case somebody finds them useful. I have polished these scripts for some five years, and they make a regular build process pretty easy.

The key characteristics for the script set is that it documents all personal changes during the build process and creates not only the binary firmware but also the set of patches that can be applied to a clean master. It also contains a self-creation script that clones a clean new repo and patches it with the made changes.

Benefits of the script set:

It enables me to publish diff/patch files with each release of the community release making the exact contents of my build wholly transparent. Anybody can easily rebuild the same firmware from scratch with the scripts.
This enables me to e.g. transfer my whole build environment to a new clean Ubuntu buildhost in a few minutes
Having extensive diff about all personal changes in main source and all feeds (and in .config, diffconfig etc.) enables also an easy comparison of unexpected changes between builds. I use a visual diff tool (WinMerge, Meld, etc.) to compare diffconfig and the patches to the previous builds' same files.

I placed an example of a firmware build and the related files in "x Example build release" directory in Dropbox.

Prerequisites:

personal changes are not committed to git, but just left added by "git add". That enables "git diff" usage. (The same could likely be achieved via git commits and "git format-patch", but I have selected to use uncommitted changes)
Use a ".config seed" to set the package/config menuconfig selections. I use a file .config.init that I always copy to be the new .config before a new build. "make defconfig" then expands it to a full .config during the build. Using such file is not quite mandatory, but enables better control of package selections in light of possibly changing defaults.
scripts expect that they are located in "hnscripts" directory in buildroot

Scripts and their short explanations:

Main scripts:

updateNmake - Update source code and make the build (via parallelcompile). Update git sources, timestamp the build, build it, and finally create the patches and re-creation script into the bin directory
newBuildroot - Creates Openwrt/LEDE build environment with the current directory as the root. Installs prerequisite packages into Ubuntu x64. An updated version of the script is created into the bin directory along the firmware at each build
mountNcopy - Mount the Virtualbox shared folder and copy files to PC. Just for transferring files, nothing really special

Compile scripts:

parallelcompile - Compile using a multiple threads
singlecompile - Compile using a single thread. Useful for debugging
kernelcompile - Clean & compile kernel using a single thread. Useful for debugging

Called from singlecompile & parallelcompile:

timestampVersion - Collect source version info and insert it into firmware before compilation. Embedd exact version info of all sources and feeds in the build.
createbuildinfo - Create info on current config and source code changes. Run at the end of the build process. Creates the patch files, the re-creation script, diffconfig etc.

Usage

The creation script runs pretty automatically. The needed few steps are:

Create the base directory (I use /OwrtLEDE) and make it writable by your normal user account (non-root)
Download from my newest firmware the newBuildroot.sh file and the four *.patch files to /OwrtLEDE
Run newBuildroot.sh. It creates the complete build environment and applies the patches
Build firmware with hnscripts/updateNmake.sh

Detailed explanation of the steps in the build environment creation process can be found in my WNDR3700 build thread:

Steps in the actual firmware build process:

Copy .config.init as the new .config to initialize the build profile: cp .config.init .config
("make defconfig" will expand the recipe to a full .config . You can run that command also manually.)
Do the actual make: hnscripts/updateNmake.sh
Transfer files from bin/targets/... to wherever you need them. I use the script hnscripts/mountNcopy.sh

Download link:

hnyman · May 22, 2017, 8:36am

Main scripts

updateNmake.sh

#!/bin/bash
#
# update & make  -  Update source code and continue to make the build

umask 0022
echo "...update main source..."
git pull
[ "$?" -ne 0 ] && echo "Updating the main OpenWrt source code failed." && exit 1
echo "...update feeds..."
./scripts/feeds update -a
[ "$?" -ne 0 ] && echo "Updating the feeds failed." && exit 1
echo "...install feeds..."
./scripts/feeds install -a
echo "...make defconfig..."
make defconfig
#echo "...make menuconfig..."
#make menuconfig
echo "...download new source packages..."
make download
echo "...make the firmware..."
hnscripts/parallelcompile.sh

parallelcompile.sh

#!/bin/bash
#
# parallelcompile  -  Compile using multiple threads

echo "...create version info file..."
hnscripts/timestampVersion.sh
echo "...make world..."
mkdir -p logs
make -j 5 V=s 2>&1 | tee logs/build.log | grep -i -E "^make.*(error|[12345]...Entering dir)"
[ ${PIPESTATUS[0]} -ne 0 ] && exit 1
echo -n "...create build info: "
hnscripts/createbuildinfo.sh
echo done.

createbuildinfo.sh

#!/bin/sh
#
# createbuildinfo  -  Create info on current config and source code changes

getGitInfo() {
#params: directory patchfile infofile
 echo "\n######################################################\n" >> $3
 (cd $1
  git diff HEAD > $2
  git remote -v show | grep fetch >> $3
  git branch --list >> $3
  git show --format="%cd %h %s" --abbrev=7 --date=short | head -n 1 | cut -b1-60 >> $3
  git status --porcelain >> $3
 )
}

BinDir=$PWD/bin/targets/ipq806x/generic
Device=R7800
Prefix=openwrt-ipq806x-generic-netgear_r7800
Branch=master

VersTime=$Branch-`scripts/getver.sh`-`date +%Y%m%d-%H%M`
TFile=$BinDir/$Device-$VersTime

echo process $Branch...

# cleanup old binaries & patches
rm -f $BinDir/$Device-*

# remove unnecessary files
rm -f $BinDir/*root.img $BinDir/*vmlinux.elf $BinDir/*initramfs-uImage

# create status info and patches
echo "$VersTime" > $TFile-status.txt
getGitInfo . $TFile-main.patch $TFile-status.txt
getGitInfo feeds/luci $TFile-luci.patch $TFile-status.txt
getGitInfo feeds/packages $TFile-packages.patch $TFile-status.txt
#getGitInfo feeds/routing $TFile-routing.patch $TFile-status.txt
sed -i -e 's/$/\r/' $TFile-status.txt

# collect config info
cp .config $TFile.config
cp .config.init $TFile.config.init
scripts/diffconfig.sh > $TFile.diffconfig 2>/dev/null

# copy buildroot creation script and patch timestamp info
cp hnscripts/newBuildroot.sh $TFile-newBuildroot.sh
sed -i "s/^FILESTAMP=.*/FILESTAMP=$Device-$VersTime/" $TFile-newBuildroot.sh

# cleanup checksum files
grep -sh $Prefix.*-squashfs $BinDir/md5sums $BinDir/sha256sums \
  | sed -e 's/$/\r/' -e 's/\*'$Prefix'/'$Device'/' -e 's/squashfs-//' \
  > $TFile-checksums.txt
rm -f $BinDir/md5sums $BinDir/sha256sums

# rename manifest and firmware files
cd $BinDir
mv *.manifest $Device-$VersTime.manifest
mv $Prefix-squashfs-sysupgrade.bin $Device-$VersTime-sysupgrade.bin
mv $Prefix-squashfs-factory.img $Device-$VersTime-factory.img

timestampVersion.sh

#!/bin/bash
#
# timestampVersion  -  Collect source version info and insert it into firmware

STATUSFILE=files/etc/Compile_info.txt
Nickname=master

echo OpenWrt $Nickname `scripts/getver.sh` / `date "+%F %H:%M"` > $STATUSFILE
echo "---" >> $STATUSFILE
echo "main      `(git log -1 --format="%cd %h %s" --abbrev=7 --date=short | head -n 1 | cut -b1-60)`" >> $STATUSFILE
echo "luci      `(cd feeds/luci && git log -1 --format="%cd %h %s" --abbrev=7 --date=short | cut -b1-60)`" >> $STATUSFILE
echo "packages  `(cd feeds/packages && git log -1 --format="%cd %h %s" --abbrev=7 --date=short | cut -b1-60)`" >> $STATUSFILE
echo "routing   `(cd feeds/routing && git log -1 --format="%cd %h %s" --abbrev=7 --date=short | cut -b1-60)`" >> $STATUSFILE
cat $STATUSFILE
git add $STATUSFILE

# Override git/svn timestamp after r48583-48594, set initial clock to now
date +%s > version.date

newBuildroot.sh

#!/bin/sh
#
# newBuildroot.sh
#
# Creates the build environment with the current directory as the root
# To avoid problems with long paths, something like /Openwrt is preferable
#
# Script creates subdir for trunk or the release branch, and dl for downloads
# Creates main source repository and luci, packages & routing feeds

### Target definitions
TARGET=master
GITREPO=https://git.openwrt.org/openwrt/openwrt.git

## Current version
FILESTAMP=R7800-master-r11094-a03219ba09-20190922-1521

### Prerequisites for buildroot
sudo apt-get install build-essential subversion libncurses5-dev zlib1g-dev
sudo apt-get install gawk gcc-multilib flex git-core gettext libssl-dev

### Prerequisite for 18.06 and 19.07 on Ubuntu 17.10+ as it has python3 by default
# sudo apt-get install python

### Prerequisite for master on Ubuntu as master needs python3 libs
sudo apt-get install python3-distutils

### Prerequisites for being able to send patches to openwrt-devel
sudo apt-get install git-email

### Newly patched Ubuntu may not yet have the correct kernel headers.
# sudo apt-get install linux-headers-$(uname -r)

### set the preferred umask (allowed: 0000-0022)
umask 0022

### download directory (outside main directory to protect from make distclean)
mkdir -p dl

### main directory
mkdir -p $TARGET

### checkout/clone and change to directory
git clone $GITREPO $TARGET
cd $TARGET

### create symlink to dl (after git clone)
ln -s ../dl dl

### patch main source first to set feeds correctly
### update the feeds, apply patches to feeds
### re-create index to find new packages, finally install
patch -p1 -i ../$FILESTAMP-main.patch
scripts/feeds update -a
(cd feeds/luci;     patch -p1 -i ../../../$FILESTAMP-luci.patch)
(cd feeds/packages; patch -p1 -i ../../../$FILESTAMP-packages.patch)
#(cd feeds/routing;  patch -p1 -i ../../../$FILESTAMP-routing.patch)
scripts/feeds update -i
scripts/feeds install -a

### chmod known script files executable
chmod -f 755 files/etc/*.sh
chmod -f 755 files/etc/rc.button/*

### chmod buildscripts executable
chmod -f 755 hnscripts/*.sh

### add created/modified files in main repo to version control
git add -f files
git add -A

### add created/modified files in feeds to version control
(cd feeds/luci;     git add -A)
(cd feeds/packages; git add -A)
#(cd feeds/routing;  git add -A)

### initialise .config
cp .config.init .config

EDIT:
Updated in Nov 2019 with the current versions.

hnyman · May 22, 2017, 8:36am

reserved 2

realopty · May 25, 2017, 7:15am

Very useful! Thank you.

hnyman · February 11, 2018, 7:25pm

Scripts updated to the current versions, with post-merge Openwrt repo locations etc.

(Note: the up-to-date scripts can naturally be always obtained from my most recent community builds for R7800 and WNDR3700. Those have also 17.01-tailored versions)

JuggernauT · June 22, 2018, 10:04am

This looks like excellent stuff, will try to make this work for my particular device... zsun wifi reader.

hnyman · November 22, 2019, 7:04pm

Scripts updated to the current versions in Nov 2019.

Note: the up-to-date scripts can naturally be always obtained from my most recent community builds for R7800 and WNDR3700. Those have also 19.07-tailored versions of the scripts.