Programmatically (non-interactively) configure the OpenWrt build

Hi all:

I am writing a script to build several OpenWrt versions for several targets. I cannot realistically do a manual "make menuconfig" every time.

Automating "make menuconfig" appears to be neglected both in the Linux Kernel and in OpenWrt, or at least I could not find any official and comprehensive documentation about it yet.

The scripted solution I have at the moment is as follows:

Step 1) Generate a .config file like this with the following kind of content for each target:

CONFIG_TARGET_x86=y
CONFIG_TARGET_x86_64=y
CONFIG_TARGET_x86_64_Generic=y

Step 2) Expand file .config to a full configuration with "make defconfig".

Step 3) Use script "scripts/config" from the Linux kernel to set some configuration options I wish. For example:

config --set-val "CONFIG_DOWNLOAD_FOLDER" "/my/download/folder"

config --set-val "GRUB_TIMEOUT" "0"

config --enable "CONFIG_CCACHE"

Setting option CONFIG_DOWNLOAD_FOLDER could be problematic, because it actually needs option CONFIG_DEVEL=y beforehand, at least when manually doing a "make menuconfig". I actually found this requirement confusing when using the menus. Option CONFIG_DOWNLOAD_FOLDER seems to work fine without CONFIG_DEVEL=y anyway.

I then realised that OpenWrt was being built without Luci, so I tried to enable it in the same way. The trouble is, enabling this option is not so straightforward:

Symbol: PACKAGE_luci [=m]
Type : tristate
Prompt: luci................... LuCI interface with Uhttpd as Webserver (default)
Location:
-> LuCI
-> 1. Collections
Defined at tmp/.config-package.in:62363
Selects: PACKAGE_libiwinfo-lua [=m] && PACKAGE_libpthread [=y] && PACKAGE_rpcd-mod-rrdns [=m] && PACKAGE_uhttpd [=m] && PACKAGE_luci-app-firewall [=m] && \
PACKAGE_uhttpd-mod-ubus [=m] && PACKAGE_luci-proto-ipv6 [=m] && PACKAGE_luci-app-opkg [=m] && PACKAGE_luci-theme-bootstrap [=m] && PACKAGE_libc [=y] && \
PACKAGE_librt [=y] && PACKAGE_luci-proto-ppp [=m] && PACKAGE_luci-mod-admin-full [=m] && PACKAGE_libssp [=n]
Selected by: PACKAGE_luci-ssl [=n] || PACKAGE_luci-ssl-openssl [=n] || PACKAGE_luci-app-noddos [=n]

That means that setting CONFIG_PACKAGE_luci=y actually enables many other options automaticallly, even more than the ones listed above. I guess some sort of dependency recursion is going on for such options.

I would like to programmatically simulate what the user is doing in "make menuconfig" when he clicks on the "luci" setting. All dependent options should then be automatically set in the same way.

If the option no longer exists, it should error. Script "scripts/config" from the Linux kernel is very simple and does not seem to be doing any validation at all.

I have seen that there are other ways to handle the OpenWrt configuration with commands like "make oldconfig", but I have not quite understood what they do yet. None seems a good fit for my purposes anyway.

What is the best way to achieve the automation I am looking for?

I guess that OpenWrt release build scripts must be doing this kind of automation somehow. After all, the official x86_64_Generic binary (openwrt-19.07.2-x86-64-combined-ext4.img.gz) does ship with Luci enabled, which is not the default when running "make menuconfig".

Thanks in advance,
rdiez

The buildbots are based on a minimal config seed, which is then extended by make defconfig (using catch-all symbols like ALL_KMODS and ALL_NONSHARED).

The easiest thing is probably to manually craft a short .config recipe for each target, and then tailor it for each OpenWrt version.

The .config recipe can easily be originally based on ./scripts/diffconfig.sh output

I am building several community builds with that approach. (WNDR3700 with ar71xx and ath79 , R7800 ipq806x), and privately also WRT3200ACM mvebu, and I build master and 19.07 for all of them.

Pretty easy, once you get the .config recipe for master. It usually needs only slight changes for 19.07.

I have also crafted build scripts that pretty much automate the actual building process and even generates copyable regeneration scripts of my build env. See the community build section. E.g. Build for Netgear R7800

In the recipes, you can set also more exotic options related to the build system config.
See example of my R7800 master below.

But you can't really craft AI for reacting to all possible changes in sources. There are so many ways the config, packages, dependencies etc. can change, that you can't avoid manual checking.

After a build I always compare the source diffs, packages selections, diffconfig etc. to the ones from the previous builds in order to see if there are any unexpected changes.

Example of my R7800 master recipe:

# Use "make defconfig" to expand this to a full .config
CONFIG_TARGET_ipq806x=y
CONFIG_TARGET_ipq806x_generic=y
CONFIG_TARGET_ipq806x_generic_DEVICE_netgear_r7800=y

# Per-package build logs in <buildroot>/logs
CONFIG_DEVEL=y
CONFIG_BUILD_LOG=y

# Debugging options (build gdbserver, include package debug)
CONFIG_PACKAGE_gdbserver=m
CONFIG_DEBUG=y

# Include package list in build
CONFIG_INCLUDE_CONFIG=y

# exfat is patented
CONFIG_BUILD_PATENTED=y

# Longer waiting for failsafe button push
CONFIG_IMAGEOPT=y
CONFIG_PREINITOPT=y
CONFIG_TARGET_PREINIT_TIMEOUT=5

# Busybox tweaks
CONFIG_BUSYBOX_CUSTOM=y
#CONFIG_BUSYBOX_CONFIG_DIFF=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVEHISTORY=y
CONFIG_BUSYBOX_CONFIG_FEATURE_EDITING_SAVE_ON_EXIT=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_FLAGS=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_REGEXP=y
CONFIG_BUSYBOX_CONFIG_FEATURE_LESS_WINCH=y

# Add-on programs
CONFIG_PACKAGE_htop=y
CONFIG_PACKAGE_nano=y
CONFIG_PACKAGE_ccrypt=y
CONFIG_PACKAGE_curl=y
CONFIG_LIBCURL_OPENSSL=y
#CONFIG_PACKAGE_vsftpd-tls=y
CONFIG_PACKAGE_wget=y
CONFIG_PACKAGE_patch=y
CONFIG_PACKAGE_diffutils=y
CONFIG_PACKAGE_tree=y
CONFIG_PACKAGE_irqbalance=y
CONFIG_DROPBEAR_ECC=y

# USB device mount & file systems support
CONFIG_PACKAGE_block-mount=y
CONFIG_PACKAGE_kmod-usb-storage=y
CONFIG_PACKAGE_kmod-fs-cifs=y
CONFIG_PACKAGE_kmod-fs-exfat=y
CONFIG_PACKAGE_libblkid=y
CONFIG_PACKAGE_kmod-fs-ext4=y
CONFIG_PACKAGE_kmod-fs-hfsplus=y
CONFIG_PACKAGE_kmod-fs-msdos=y
CONFIG_PACKAGE_kmod-fs-vfat=y
CONFIG_PACKAGE_ntfs-3g=y
CONFIG_PACKAGE_kmod-nls-cp1250=y
CONFIG_PACKAGE_kmod-nls-cp437=y
CONFIG_PACKAGE_kmod-nls-cp850=y
CONFIG_PACKAGE_kmod-nls-iso8859-1=y
CONFIG_PACKAGE_kmod-nls-iso8859-15=y
CONFIG_PACKAGE_kmod-nls-utf8=y

# IPv6 support
CONFIG_PACKAGE_6in4=y
CONFIG_PACKAGE_6to4=y
CONFIG_PACKAGE_6rd=y

# IPv6 NAT support (ip6tables NAT extensions, ipt-nat6 and nf-nat6 kmods)
###CONFIG_PACKAGE_ip6tables-mod-nat=y

# WLAN/WPS support
CONFIG_PACKAGE_hostapd-utils=y
#CONFIG_WPA_MSG_MIN_PRIORITY=2
CONFIG_WPA_MSG_MIN_PRIORITY=4
CONFIG_PACKAGE_wpad-openssl=y
# CONFIG_PACKAGE_wpad-basic is not set

# SSL certificates
CONFIG_PACKAGE_ca-certificates=y

# Luci (SSL from OpenSSL)
CONFIG_PACKAGE_luci-ssl-openssl=y
CONFIG_PACKAGE_luci-mod-admin-full=y
CONFIG_PACKAGE_luci-app-commands=y
CONFIG_PACKAGE_luci-app-ddns=y
CONFIG_PACKAGE_luci-app-upnp=y
CONFIG_PACKAGE_luci-app-wol=y

# Luci statistics
CONFIG_PACKAGE_luci-app-statistics=y
CONFIG_PACKAGE_collectd-mod-conntrack=y
CONFIG_PACKAGE_collectd-mod-cpufreq=y
CONFIG_PACKAGE_collectd-mod-entropy=y
CONFIG_PACKAGE_collectd-mod-ping=y
CONFIG_PACKAGE_collectd-mod-thermal=y
CONFIG_PACKAGE_collectd-mod-uptime=y

# QoS selection - currently SQM
CONFIG_PACKAGE_luci-app-sqm=y

# Build material and openwrt-2020 themes, default is still bootstrap
CONFIG_PACKAGE_luci-theme-material=y
CONFIG_PACKAGE_luci-theme-openwrt-2020=y

# kernel support for tunnels, VPNs
CONFIG_PACKAGE_kmod-tun=y

# wireguard
CONFIG_PACKAGE_luci-app-wireguard=y

# PPTP support
CONFIG_PACKAGE_luci-proto-ppp=y
CONFIG_PACKAGE_ppp-mod-pptp=y

# iptables add-on ipsec
CONFIG_PACKAGE_iptables-mod-ipsec=y

# Support for IETF BCP38
CONFIG_PACKAGE_luci-app-bcp38=y

# Adblock package with Luci support, initially disabled
CONFIG_PACKAGE_luci-app-adblock=y

# BanIP package with Luci support, initially disabled
CONFIG_PACKAGE_luci-app-banip=y

# nlbwmon app
CONFIG_PACKAGE_luci-app-nlbwmon=y

# chaoskey TRNG USB dongle
CONFIG_PACKAGE_kmod-chaoskey=y

For ath79 I build several devices with difference in flash sizes, so the device section is a bit more complicated with different packages being selected to different devices. I show here only the interesting part highlighting that:

# Use "make defconfig" to expand this to a full .config
CONFIG_TARGET_ath79=y
CONFIG_TARGET_ath79_generic=y
CONFIG_TARGET_MULTI_PROFILE=y
CONFIG_TARGET_DEVICE_ath79_generic_DEVICE_netgear_wndr3700=y
CONFIG_TARGET_DEVICE_PACKAGES_ath79_generic_DEVICE_netgear_wndr3700="-wpad-basic"
CONFIG_TARGET_DEVICE_ath79_generic_DEVICE_netgear_wndr3700-v2=y
CONFIG_TARGET_DEVICE_PACKAGES_ath79_generic_DEVICE_netgear_wndr3700-v2="-wpad-basic patch diffutils tree kmod-fs-hfsplus kmod-fs-ntfs luci-app-wol luci-app-upnp luci-app-commands luci-theme-material luci-theme-openwrt-2020"
CONFIG_TARGET_DEVICE_ath79_generic_DEVICE_netgear_wndr3800=y
CONFIG_TARGET_DEVICE_PACKAGES_ath79_generic_DEVICE_netgear_wndr3800="-wpad-basic patch diffutils tree kmod-fs-hfsplus kmod-fs-ntfs luci-app-wol luci-app-upnp luci-app-commands luci-theme-material luci-theme-openwrt-2020"

CONFIG_TARGET_PER_DEVICE_ROOTFS=y

# Per-package build logs in <buildroot>/logs
CONFIG_DEVEL=y
CONFIG_BUILD_LOG=y
...
# nlbwmon app
CONFIG_PACKAGE_luci-app-nlbwmon=y

# packages only for WNDR3700v2 and WNDR3800
CONFIG_PACKAGE_patch=m
CONFIG_PACKAGE_diffutils=m
CONFIG_PACKAGE_curl=m
CONFIG_LIBCURL_OPENSSL=y
#CONFIG_PACKAGE_vsftpd-tls=m
CONFIG_PACKAGE_tree=m
CONFIG_PACKAGE_kmod-fs-hfsplus=m
CONFIG_PACKAGE_kmod-fs-ntfs=m
CONFIG_PACKAGE_luci-app-wol=m
CONFIG_PACKAGE_luci-app-upnp=m
CONFIG_PACKAGE_luci-app-commands=m
CONFIG_PACKAGE_luci-theme-material=m
CONFIG_PACKAGE_luci-theme-openwrt-2020=m


2 Likes