Including packages with Image Builder automatically

For Developers
1

I’m confused how to effectively use the OpenWRT image builder. I’m compiling the whole tree from source, including the image builder, and I’d like to use the image builder so that later I can bake in configs and other files.

However, I’m unsure how to properly get the image builder to include all of the packages for which I have PACKAGE_name=y.

For example, I have LuCI set to be installed:
root@dd46ebed4ab6:/workdir/openwrt-imagebuilder-mediatek-filogic.Linux-x86_64# cat .config | grep PACKAGE_luci | head
CONFIG_PACKAGE_luci=y

and the .ipk is in the packages directory, but when I run make image, it doesn’t install it:
Building images for mediatek - Bananapi BPi-R4
Packages: base-files ca-bundle dnsmasq dropbear e2fsprogs f2fsck firewall4 fitblk fstools kernel kmod-crypto-hw-safexcel kmod-eeprom-at24 kmod-gpio-button-hotplug kmod-hwmon-pwmfan kmod-i2c-mux-pca954x kmod-leds-gpio kmod-mt7996-233-firmware kmod-mt7996-firmware kmod-nft-offload kmod-phy-aquantia kmod-rtc-pcf8563 kmod-sfp kmod-usb3 libc libgcc libustream-mbedtls logd mkf2fs mt7988-wo-firmware mtd netifd nftables odhcp6codhcpd-ipv6only opkg ppp ppp-mod-pppoe procd-ujail uboot-envtools uci uclient-fetch urandom-seed urngd wpad-basic-mbedtls

If I set PACKAGES to everything in the packages directory, I get conflicts, presumably because some things are built as modules and thus their packages are still in there.

How can I get the image builder to create me the same build that I get from my .config and just running make world?

2

Maybe there are some dependencies missing. You should be able to check with make menuconfigwhich LuCI parts are selected. Does this differ from your .config file?

3

I don’t think there should be any missing. I have luci, luci-light, luci-ssl-openssl, luci-base, etc.

If I add luci to PACKAGES manually, it installs it and all the dependencies fine.

My .config in the image builder folder is the same as my .config that I used to make world.

Is the Image Builder supposed to add everything automatically? Because I am also following this guide https://git.defensec.nl/?p=selinux-policy.git;a=blob;f=README;h=8c6fb2bac542ae0d5dd68a111db06d069b946713;hb=HEAD where I have to add a custom feed, and even though the custom selinux-policy-next package is added to the menuconfig menu, it’s still required that I add that manually to PACKAGES. So I’m just not sure how it’s meant to work.

4

Are you sure the image builder is using the .config for the packages to install? When looking at it a few years back, I believe which packages are included by default were controlled by the .target_info.

grep them from file and add them as PACKAGE= in CLI?

5

Are you sure the image builder is using the .config for the packages to install?

No, not at all. In fact I suspect it doesn’t use that, and indeed doesn’t add any other packages than what’s included by default.

My target profile has:
Target-Profile: DEVICE_bananapi_bpi-r4
Target-Profile-Name: Bananapi BPi-R4
Target-Profile-Packages: kmod-hwmon-pwmfan kmod-i2c-mux-pca954x kmod-eeprom-at24 kmod-mt7996-firmware kmod-mt7996-233-firmware kmod-rtc-pcf8563 kmod-sfp kmod-usb3 e2fsprogs f2fsck mkf2fs mt7988-wo-firmware
Target-Profile-hasImageMetadata: 1
Target-Profile-SupportedDevices: bananapi,bpi-r4

but that doesn’t account for all of the packages added (seems like it only specifies some very specific ones there).

So I tried this with grep '^CONFIG_PACKAGE_' .config | grep '=y' | sed 's/CONFIG_PACKAGE_//;s/=y//' which gives me a list, and I can pass that to packages:
Packages: base-files busybox-selinux ca-bundle dnsmasq dropbear firewall4 fitblk fstools fwtool getrandom jsonfilter libc libgcc libpthread librt logd mtd netifd openwrt-keyring opkg procd-seccomp procd-selinux procd-ujail rpcd rpcd-mod-file rpcd-mod-iwinfo rpcd-mod-ucode selinux-policy selinux-policy-next ubox ubus ubusd uci urandom-seed urngd usign wifi-scripts trusted-firmware-a-mt7981-ram-ddr3 trusted-firmware-a-mt7981-ram-ddr4 trusted-firmware-a-mt7986-ram-ddr3 trusted-firmware-a-mt7986-ram-ddr4 trusted-firmware-a-mt7988-emmc-comb trusted-firmware-a-mt7988-ram-comb trusted-firmware-a-mt7988-sdmmc-comb trusted-firmware-a-mt7988-spim-nand-ubi-comb u-boot-mt7988_bananapi_bpi-r4-emmc u-boot-mt7988_bananapi_bpi-r4-sdmmc u-boot-mt7988_bananapi_bpi-r4-snand libiwinfo-data eip197-mini-firmware mt7988-wo-firmware wireless-regdb kmod-crypto-aead kmod-crypto-authenc kmod-crypto-ccm kmod-crypto-cmac kmod-crypto-crc32c kmod-crypto-ctr kmod-crypto-des kmod-crypto-gcm kmod-crypto-geniv kmod-crypto-gf128 kmod-crypto-ghash kmod-crypto-hash kmod-crypto-hmac kmod-crypto-hw-safexcel kmod-crypto-manager kmod-crypto-md5 kmod-crypto-null kmod-crypto-rng kmod-crypto-seqiv kmod-crypto-sha1 kmod-crypto-sha256 kmod-crypto-sha3 kmod-crypto-sha512 kmod-gpio-button-hotplug kmod-hwmon-core kmod-hwmon-pwmfan kmod-i2c-core kmod-i2c-mux kmod-i2c-mux-pca954x kmod-leds-gpio kmod-lib-crc-ccitt kmod-lib-crc32c kmod-nls-base kmod-nf-conntrack kmod-nf-conntrack6 kmod-nf-flow kmod-nf-log kmod-nf-log6 kmod-nf-nat kmod-nf-reject kmod-nf-reject6 kmod-nfnetlink kmod-nft-core kmod-nft-fib kmod-nft-nat kmod-nft-offload kmod-libphy kmod-phy-aquantia kmod-phylink kmod-sfp kmod-ppp kmod-pppoe kmod-pppox kmod-slhc kmod-eeprom-at24 kmod-regmap-core kmod-regmap-i2c kmod-rtc-pcf8563 kmod-usb-core kmod-usb-xhci-hcd kmod-usb-xhci-mtk kmod-usb3 kmod-cfg80211 kmod-mac80211 MAC80211_DEBUGFS MAC80211_MESH kmod-mt76-connac kmod-mt76-core kmod-mt7996-233-firmware kmod-mt7996-firmware kmod-mt7996-firmware-common kmod-mt7996e ucode ucode-mod-fs ucode-mod-math ucode-mod-nl80211 ucode-mod-rtnl ucode-mod-ubus ucode-mod-uci ucode-mod-uloop libmbedtls libopenssl libopenssl-conf jansson libblkid libblobmsg-json libcomerr libe2p libext2fs libf2fs libiwinfo libjson-c liblucihttp liblucihttp-ucode libmnl libnftnl libnl-tiny libpcre2 libselinux libsepol libss libubox libubus libuci libuclient libucode libudebug libustream-openssl libuuid musl-fts rpcd-mod-luci rpcd-mod-rrdns luci luci-light luci-ssl-openssl luci-base luci-mod-admin-full luci-mod-network luci-mod-status luci-mod-system luci-app-firewall luci-app-package-manager luci-theme-bootstrap luci-proto-ipv6 luci-proto-ppp nftables-json knot-resolver_dnstap cgi-io uhttpd uhttpd-mod-ubus hostapd-common wpad-basic-mbedtls iw odhcp6c odhcpd-ipv6only ppp ppp-mod-pppoe uclient-fetch uboot-envtools e2fsprogs f2fsck mkf2fs iwinfo jshn libjson-script openssl-util ubi-utils ucode-mod-html base-files ca-bundle dnsmasq dropbear e2fsprogsf2fsck firewall4 fitblk fstools kernel kmod-crypto-hw-safexcel kmod-eeprom-at24 kmod-gpio-button-hotplug kmod-hwmon-pwmfan kmod-i2c-mux-pca954x kmod-leds-gpio kmod-mt7996-233-firmware kmod-mt7996-firmware kmod-nft-offload kmod-phy-aquantia kmod-rtc-pcf8563 kmod-sfp kmod-usb3 libc libgcc libustream-mbedtls logd mkf2fs mt7988-wo-firmware mtd netifd nftables odhcp6c odhcpd-ipv6only opkg ppp ppp-mod-pppoe procd-ujail uboot-envtools uci uclient-fetch urandom-seed urngd wpad-basic-mbedtls

and other than the fact that this includes some packages that aren’t meant to be installed:

opkg_install_cmd: Cannot install package u-boot-mt7988_bananapi_bpi-r4-snand. opkg_install_cmd: Cannot install package MAC80211_DEBUGFS.
opkg_install_cmd: Cannot install package MAC80211_MESH.
opkg_install_cmd: Cannot install package knot-resolver_dnstap.

and needing to remove some default packages, it does build and work with:

make image PACKAGES="$(while read -r name; do ls packages/"${name}"_*.ipk 2>/dev/null; done < built_in_packages.txt | tr '\n' ' ') $(printf '%s ' $(ls ../openwrt/bin/packages/aarch64_cortex-a53/custom/*.ipk) luci -libustream-mbedtls -selinux-policy)"

I guess this is a solution… but is this really the intended way to use this? It feels brittle.

6

The intended way is to ignore all automatically included default packages, and only add the additional top-level packages to the list.

You do not even need to include all "lower" dependency packages, as they are pulled in via dependencies. E.g. no need to add "sqm-scripts", if you have the LuCI app "luci-app-sqm" which pulls in the underlying sqm-scripts.

I usually build via full toolchain, but here is example of a shell script to build via imagebuilder.

make image \
 PROFILE="dynalink_dl-wrx36" \
 PACKAGES="ca-bundle ccrypt diffutils gdbserver htop irqbalance \
  mtr-nojson nano-full openssh-sftp-server patch tcpdump-mini tree wget-ssl \
  block-mount kmod-usb-storage kmod-fs-cifs kmod-fs-exfat libblkid \
  kmod-fs-ext4 kmod-fs-msdos kmod-fs-ntfs3 kmod-nls-cp437 kmod-nls-iso8859-1 \
  kmod-nls-utf8 hostapd-utils wpad-openssl \
  luci-ssl-openssl \
  luci-app-adblock luci-app-banip luci-app-bcp38 luci-app-commands \
  luci-app-irqbalance luci-app-nlbwmon luci-app-sqm luci-app-uhttpd \
  apk-openssl luci-app-package-manager \
  luci-app-statistics collectd-mod-conntrack collectd-mod-cpufreq \
  collectd-mod-ping collectd-mod-thermal collectd-mod-uptime \
  kmod-tun luci-proto-wireguard unetd unet-cli luci-proto-unet \
  iptables-nft ip6tables-nft ipset \
  -wpad-basic-mbedtls -libustream-mbedtls -libmbedtls -apk-mbedtls" \
 FILES="../files"
7

One really offensive way to configure packages to be included in all full toolchain builds and in all built imagebuilders (and thus images built with it), would be to directly modify the deepest level default packages inclusion in source code, before compiling the imagebuilder itself.

The short "ultimate default" list is in include/target.mk

https://github.com/openwrt/openwrt/blob/main/include/target.mk

DEFAULT_PACKAGES:=\
	base-files \
	ca-bundle \
	dropbear \
	fstools \
...

Adding packages there makes them to be included in all possible builds (unless explicitly de-selected).

8

The guide works in the custom feed differently than I do, so I can’t speak to its specifics. My process is usually as follows:

  • Clone Openwrt git
  • Add filesfolder with a couple of overrides
  • cp feeds.conf.default feeds.conf
  • echo my repo into that file (or a local package branch in some cases)
  • comment out feeds I don’t need (LuCI, video, telephony)
  • update and install all feeds
  • Now, when using make menuconfig I have my feed as main folder (below Administration in my case), and can enter it and enable the package (model dependent) I want to build.

That approach has never failed to build my custom packages into images, even without specifying them in the make command. I worked a couple of times with the .config edit approach, but have defaulted back to using make menuconfig only.

What is irritating though, is that it fails to include a package like LuCI for you and not the custom stuff.

9

I’ve answered this question for you. :wink:

That’s a separate issue which @hnyman has already addressed. Looks like you may want to keep two separate lists of packages:

  • list of packages to make in toolchain
  • list of packages to be added to an image builder
10

It is already a default package.

11

I know. That is surplus in my example.

(Echos from old days. It needed to be explicitly selected until a few years back, when OpenWrt started to provide Https support by default)

12

Is there an easy way to sort the packages by what is a dependency of what? I’m thinking of something like a tree view where dependencies are children, and that way I can see what top-level packages I need to add and what they’ll bring in automatically.