[Solved] Unable to install packages with opkg on CONFIG_TARGET_DEVICE_PACKAGES images

I am using the CONFIG_TARGET_DEVICE_PACKAGES feature quite happily for a few ar71xx devices, but it seems like I am unable to install packages through opkg on firmwares built with this neat feature.

I e.g. build for my (ramips) DIR-860L and (x86) APU2. That's just one device per architecture. On both, I can get opkg to download and verify my own package lists just fine:

root@PC Engines APU2:~# opkg update
Downloading https://volatilesystems.org/dl/lede/17.01/latest/targets/x86/64/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/designated_driver_core
Downloading https://volatilesystems.org/dl/lede/17.01/latest/targets/x86/64/packages/Packages.sig
Signature check passed.
[...]

On any ar71xx device (WNDR3700, UniFi AP AC Pro, ...), however, I get a failed signature check, despite them all having been built in one run. Packages included may differ between the targets, but they all use the same binary packages I'd think.

root@UniFi AP AC Pro 1:~# opkg update
Downloading https://volatilesystems.org/dl/lede/17.01/latest/targets/ar71xx/generic/packages/Packages.gz
Updated list of available packages in /var/opkg-lists/designated_driver_core
Downloading https://volatilesystems.org/dl/lede/17.01/latest/targets/ar71xx/generic/packages/Packages.sig
Signature check failed.
Remove wrong Signature file.
[...]

Are the entries in /etc/opkg/keys/ identical in both cases?

Hi jow. Thanks for chiming in.

There's something weird going on there. The UniFi e.g. has its keys in that path, but other ar71xx devices have it in a subdir. The UniFi is missing a 'local build' key, but the other ar71xx devices have it.

root@UniFi AP AC Pro 1:/etc/opkg/keys# cat *
untrusted comment: LEDE usign key of Hans Dedecker
RWRRUfaUIMP1CAL9wvk3ABBHdUM+3SjMvIuJlK68b3b04Pw3wiaiAfxX
untrusted comment: LEDE usign key of Jo-Philipp Wich
RWRypX8hkbIR4FLhtx5pjXcAIsI1iPUIcI5bMG8jZoiCkrwTstECBPqL
untrusted comment: LEDE 17.01 "Reboot" public key
RWR5LZ2bOfGA3FGliZosEDhodiAKDOISmQs/mmjo4rhcbFtqkibJqMzo
untrusted comment: LEDE usign key of Álvaro Fernández Rojas
RWSe9GlCCBAsQwI5+wztnWKHfBlvPFP2G00FvZyx+Wfv9AwSViUwo/s2
untrusted comment: LEDE usign key for unattended build jobs
RWS1BD5w+adc3j2Hqg9+b66CvLR7NlHbsj7wjNVj0XGt/othDgIAOJS+
untrusted comment: LEDE usign key of Ted Hess
RWTazp1N8WiWvy7rYxstJqaMzGiS4XfW1oyYrk2vwJMRBeBF+8xEA+EZ
untrusted comment: LEDE usign key of John Crispin
RWTdbeDQa709heyMmwDZjWmlhcTCUv/q+3TBYDPdJAGRuys6xcxE09fp

The TL-WR1043ND v2 has them in /etc/opkg/target-dir-1a37ae4d.opkg/keys, and has an extra 'local build key'. My WNDR3700 has a similar subdir, and a local build key as well.

root@LEDE:/etc/opkg/target-dir-1a37ae4d.opkg/keys# cat *
untrusted comment: LEDE usign key of Hans Dedecker
RWRRUfaUIMP1CAL9wvk3ABBHdUM+3SjMvIuJlK68b3b04Pw3wiaiAfxX
untrusted comment: Local build key
RWQFa8bW97CKKBvikUN2uWwGnHs1tfACHrFR+GWu2gXGFFFskcHrHdlY
untrusted comment: LEDE usign key of Jo-Philipp Wich
RWRypX8hkbIR4FLhtx5pjXcAIsI1iPUIcI5bMG8jZoiCkrwTstECBPqL
untrusted comment: LEDE 17.01 "Reboot" public key
RWR5LZ2bOfGA3FGliZosEDhodiAKDOISmQs/mmjo4rhcbFtqkibJqMzo
untrusted comment: LEDE usign key of Álvaro Fernández Rojas
RWSe9GlCCBAsQwI5+wztnWKHfBlvPFP2G00FvZyx+Wfv9AwSViUwo/s2
untrusted comment: LEDE usign key for unattended build jobs
RWS1BD5w+adc3j2Hqg9+b66CvLR7NlHbsj7wjNVj0XGt/othDgIAOJS+
untrusted comment: LEDE usign key of Ted Hess
RWTazp1N8WiWvy7rYxstJqaMzGiS4XfW1oyYrk2vwJMRBeBF+8xEA+EZ
untrusted comment: LEDE usign key of John Crispin
RWTdbeDQa709heyMmwDZjWmlhcTCUv/q+3TBYDPdJAGRuys6xcxE09fp

The weird thing is all three devices are running the same build (17.01-SNAPSHOT, r3805+4-9934231670). I know for sure the TP-Link and WNDR3700 had a clean flash, but the UniFi might have been flashed with a vanilla 17.01 build first before I flashed mine over it. That seems to make a difference.

If I move the keys/ subdir into /etc/opkg/ directly then opkg update runs fine on both the TP-Link and the Netgear. Adding the local build key to the UniFi's keys makes the UniFi validate the signatures as well.

Do you need a diffconfig for my buildroot? I don't know where the target-dir weirdness comes from.

Yes, a diffconfig would be helpful. Do you have any scripting business or overrides in files/ ?

The extra subdirectory is created in include/image.mk, specifically this section:

target-dir-%: FORCE
        rm -rf $(mkfs_cur_target_dir) $(mkfs_cur_target_dir).opkg
        $(CP) $(TARGET_DIR_ORIG) $(mkfs_cur_target_dir)
        -mv $(mkfs_cur_target_dir)/etc/opkg $(mkfs_cur_target_dir).opkg
        echo 'src default file://$(PACKAGE_DIR_ALL)' > $(mkfs_cur_target_dir).conf
        $(if $(mkfs_packages_remove), \
                -$(call opkg,$(mkfs_cur_target_dir)) remove \
                        $(mkfs_packages_remove))
        $(if $(call opkg_package_files,$(mkfs_packages_add)), \
                $(opkg_target) update && \
                $(opkg_target) install \
                        $(call opkg_package_files,$(mkfs_packages_add)))
        $(call prepare_rootfs,$(mkfs_cur_target_dir))
        -mv $(mkfs_cur_target_dir).opkg $(mkfs_cur_target_dir)/etc/opkg
        rm -f $(mkfs_cur_target_dir).conf

Can you remove the - from both -mv calls and issue a make target/linux/install V=s ?
I wonder if you have some permission or file system problems on your host or we face a race condition here.

I am overriding /etc/opkg/distfeeds.conf through files/ (so it points to my repo).

I have some stuff in files/etc/uci/defaults/, but that does not touch opkg (it configures IP, DHCP, unbound, firewall and dropbear settings, timezone and zonename). I can pastebin those as well though if you suspect they'd interfere nonetheless.

I will edit tomorrow the include/image.mk code and report back to you.

My diff is below. At first glance, I don't se anything in the diff that might interfere, but then again...

CONFIG_TARGET_ar71xx=y
CONFIG_TARGET_ar71xx_generic=y
CONFIG_TARGET_MULTI_PROFILE=y
CONFIG_TARGET_DEVICE_ar71xx_generic_DEVICE_wndr3700=y
CONFIG_TARGET_DEVICE_PACKAGES_ar71xx_generic_DEVICE_wndr3700="ethtool htop iperf3 kmod-sched-cake kmod-usb-net-cdc-ether kmod-usb-net-rndis luci-ssl luci-app-ddns luci-app-ntpc luci-app-sqm luci-app-statistics luci-app-unbound luci-app-wireguard luci-proto-wireguard luci-theme-bootstrap netperf tmux usbutils unbound unbound-anchor"
CONFIG_TARGET_DEVICE_ar71xx_generic_DEVICE_wndr3700v2=y
CONFIG_TARGET_DEVICE_PACKAGES_ar71xx_generic_DEVICE_wndr3700v2="ethtool htop iperf3 kmod-sched-cake kmod-usb-net-cdc-ether kmod-usb-net-rndis luci-ssl luci-app-ddns luci-app-ntpc luci-app-sqm luci-app-statistics luci-app-unbound luci-app-wireguard luci-proto-wireguard luci-theme-bootstrap netperf tmux usbutils unbound unbound-anchor"
CONFIG_TARGET_DEVICE_ar71xx_generic_DEVICE_tl-wr1043nd-v1=y
CONFIG_TARGET_DEVICE_PACKAGES_ar71xx_generic_DEVICE_tl-wr1043nd-v1="ca-certificates iperf3 kmod-sched-cake kmod-usb-net-cdc-ether kmod-usb-net-rndis luci -mtr netperf usbutils"
CONFIG_TARGET_DEVICE_ar71xx_generic_DEVICE_tl-wr1043nd-v2=y
CONFIG_TARGET_DEVICE_PACKAGES_ar71xx_generic_DEVICE_tl-wr1043nd-v2="ethtool htop iperf3 kmod-sched-cake kmod-usb-net-cdc-ether kmod-usb-net-rndis luci-ssl luci-app-ddns luci-app-ntpc luci-app-sqm luci-app-statistics luci-app-unbound luci-app-wireguard luci-proto-wireguard luci-theme-bootstrap netperf tmux usbutils unbound unbound-anchor"
CONFIG_TARGET_DEVICE_ar71xx_generic_DEVICE_tl-wr841-v7=y
CONFIG_TARGET_DEVICE_PACKAGES_ar71xx_generic_DEVICE_tl-wr841-v7="-ca-certificates -dmesg -ipset -kmod-ppp -kmod-ppp-pppoe -kmod-ppp-pppox -kmod-usb-core -kmod-usb-ledtrig-usbport -kmod-usb-net -kmod-usb-net-cdc-ether -kmod-usb-net-rndis -kmod-usb-ohci -kmod-usb2 -libevent2 -libncursesw -mtr -ppp-mod-pppoe -ppp -terminfo"
CONFIG_TARGET_DEVICE_ar71xx_generic_DEVICE_ubnt-unifiac-pro=y
CONFIG_TARGET_DEVICE_PACKAGES_ar71xx_generic_DEVICE_ubnt-unifiac-pro="alfred ath10k-firmware-qca988x batctl -ddns-scripts -ddns-scripts-_no-ip_com ethtool htop iperf3 kmod-ath10k kmod-batman-adv luci-ssl luci-app-meshwizard luci-app-ntpc luci-app-statistics luci-theme-bootstrap netperf tmux usbutils"
CONFIG_DEVEL=y
CONFIG_TOOLCHAINOPTS=y
CONFIG_TARGET_PER_DEVICE_ROOTFS=y
# CONFIG_BINUTILS_USE_VERSION_2_25_1 is not set
CONFIG_BINUTILS_USE_VERSION_2_27=y
CONFIG_BINUTILS_VERSION="2.27"
CONFIG_BINUTILS_VERSION_2_27=y
CONFIG_CCACHE=y
CONFIG_DROPBEAR_ECC=y
# CONFIG_GCC_USE_VERSION_5 is not set
CONFIG_GCC_USE_VERSION_6=y
CONFIG_GCC_VERSION="6.3.0"
CONFIG_IMAGEOPT=y
# CONFIG_KERNEL_KALLSYMS is not set
CONFIG_KMOD_BATMAN_ADV_BLA=y
CONFIG_KMOD_BATMAN_ADV_DAT=y
CONFIG_KMOD_BATMAN_ADV_DEBUGFS=y
CONFIG_KMOD_BATMAN_ADV_MCAST=y
CONFIG_LIBCURL_COOKIES=y
CONFIG_LIBCURL_FILE=y
CONFIG_LIBCURL_HTTP=y
CONFIG_LIBCURL_MBEDTLS=y
CONFIG_LIBCURL_NO_SMB="!"
CONFIG_LUCI_SRCDIET=y
CONFIG_OPENSSL_WITH_DEPRECATED=y
CONFIG_OPENSSL_WITH_EC=y
CONFIG_OPENSSL_WITH_EC2M=y
CONFIG_OPENSSL_WITH_NPN=y
CONFIG_OPENSSL_WITH_PSK=y
CONFIG_OPENSSL_WITH_SRP=y
CONFIG_PACKAGE_ALFRED_VIS=y
CONFIG_PACKAGE_ATH_DEBUG=y
CONFIG_PACKAGE_alfred=m
CONFIG_PACKAGE_batctl=m
CONFIG_PACKAGE_ca-certificates=y
CONFIG_PACKAGE_collectd=m
CONFIG_PACKAGE_collectd-mod-conntrack=m
CONFIG_PACKAGE_collectd-mod-cpu=m
CONFIG_PACKAGE_collectd-mod-dns=m
CONFIG_PACKAGE_collectd-mod-interface=m
CONFIG_PACKAGE_collectd-mod-iptables=m
CONFIG_PACKAGE_collectd-mod-iwinfo=m
CONFIG_PACKAGE_collectd-mod-load=m
CONFIG_PACKAGE_collectd-mod-logfile=m
CONFIG_PACKAGE_collectd-mod-memory=m
CONFIG_PACKAGE_collectd-mod-network=m
CONFIG_PACKAGE_collectd-mod-ntpd=m
CONFIG_PACKAGE_collectd-mod-openvpn=m
CONFIG_PACKAGE_collectd-mod-rrdtool=m
CONFIG_PACKAGE_curl=y
CONFIG_PACKAGE_ddns-scripts=y
CONFIG_PACKAGE_ddns-scripts_no-ip_com=y
CONFIG_PACKAGE_dmesg=y
CONFIG_PACKAGE_etherwake=m
CONFIG_PACKAGE_ethtool=m
CONFIG_PACKAGE_hostapd=m
CONFIG_PACKAGE_hostapd-utils=m
CONFIG_PACKAGE_htop=m
CONFIG_PACKAGE_ip-tiny=m
CONFIG_PACKAGE_iperf3=y
CONFIG_PACKAGE_ipset=y
CONFIG_PACKAGE_iptables-mod-conntrack-extra=m
CONFIG_PACKAGE_iptables-mod-ipopt=m
CONFIG_PACKAGE_iptables-mod-iprange=y
CONFIG_PACKAGE_kmod-batman-adv=m
CONFIG_PACKAGE_kmod-crypto-crc32c=m
CONFIG_PACKAGE_kmod-crypto-hash=m
CONFIG_PACKAGE_kmod-ifb=m
CONFIG_PACKAGE_kmod-ipt-conntrack-extra=m
CONFIG_PACKAGE_kmod-ipt-filter=y
CONFIG_PACKAGE_kmod-ipt-ipopt=m
CONFIG_PACKAGE_kmod-ipt-iprange=y
CONFIG_PACKAGE_kmod-ipt-ipset=y
CONFIG_PACKAGE_kmod-leds-wndr3700-usb=y
CONFIG_PACKAGE_kmod-lib-crc16=m
CONFIG_PACKAGE_kmod-lib-crc32c=m
CONFIG_PACKAGE_kmod-lib-textsearch=y
CONFIG_PACKAGE_kmod-mii=y
CONFIG_PACKAGE_kmod-nfnetlink=y
CONFIG_PACKAGE_kmod-nls-base=y
CONFIG_PACKAGE_kmod-sched=m
CONFIG_PACKAGE_kmod-sched-cake=m
CONFIG_PACKAGE_kmod-sched-core=m
CONFIG_PACKAGE_kmod-tun=m
CONFIG_PACKAGE_kmod-udptunnel4=m
CONFIG_PACKAGE_kmod-udptunnel6=m
CONFIG_PACKAGE_kmod-usb-core=y
CONFIG_PACKAGE_kmod-usb-ledtrig-usbport=y
CONFIG_PACKAGE_kmod-usb-net=y
CONFIG_PACKAGE_kmod-usb-net-cdc-ether=y
CONFIG_PACKAGE_kmod-usb-net-rndis=y
CONFIG_PACKAGE_kmod-usb-ohci=y
CONFIG_PACKAGE_kmod-usb2=y
CONFIG_PACKAGE_kmod-wireguard=m
CONFIG_PACKAGE_libcurl=y
CONFIG_PACKAGE_libevent2=y
CONFIG_PACKAGE_libexpat=m
CONFIG_PACKAGE_libiptc=m
CONFIG_PACKAGE_libiwinfo-lua=m
CONFIG_PACKAGE_libltdl=m
CONFIG_PACKAGE_liblua=m
CONFIG_PACKAGE_liblzo=m
CONFIG_PACKAGE_libmbedtls=y
CONFIG_PACKAGE_libmnl=y
CONFIG_PACKAGE_libncurses=y
CONFIG_PACKAGE_libopenssl=m
CONFIG_PACKAGE_libpcap=m
CONFIG_PACKAGE_librrd1=m
CONFIG_PACKAGE_librt=y
CONFIG_PACKAGE_libubus-lua=m
CONFIG_PACKAGE_libuci-lua=m
CONFIG_PACKAGE_libunbound=m
CONFIG_PACKAGE_libustream-mbedtls=m
CONFIG_PACKAGE_lua=m
CONFIG_PACKAGE_luci=m
CONFIG_PACKAGE_luci-app-commands=m
CONFIG_PACKAGE_luci-app-ddns=m
CONFIG_PACKAGE_luci-app-diag-core=m
CONFIG_PACKAGE_luci-app-firewall=m
CONFIG_PACKAGE_luci-app-meshwizard=m
CONFIG_PACKAGE_luci-app-ntpc=m
CONFIG_PACKAGE_luci-app-openvpn=m
CONFIG_PACKAGE_luci-app-sqm=m
CONFIG_PACKAGE_luci-app-statistics=m
CONFIG_PACKAGE_luci-app-unbound=m
CONFIG_PACKAGE_luci-app-watchcat=m
CONFIG_PACKAGE_luci-app-wireguard=m
CONFIG_PACKAGE_luci-app-wol=m
CONFIG_PACKAGE_luci-base=m
CONFIG_PACKAGE_luci-lib-ip=m
CONFIG_PACKAGE_luci-lib-jsonc=m
CONFIG_PACKAGE_luci-lib-nixio=m
CONFIG_PACKAGE_luci-mod-admin-full=m
CONFIG_PACKAGE_luci-proto-ipv6=m
CONFIG_PACKAGE_luci-proto-ppp=m
CONFIG_PACKAGE_luci-proto-wireguard=m
CONFIG_PACKAGE_luci-ssl=m
CONFIG_PACKAGE_luci-theme-bootstrap=m
CONFIG_PACKAGE_meshwizard=m
CONFIG_PACKAGE_mtr=y
CONFIG_PACKAGE_netperf=m
CONFIG_PACKAGE_ntpclient=m
CONFIG_PACKAGE_px5g-mbedtls=y
CONFIG_PACKAGE_rpcd=m
CONFIG_PACKAGE_rrdtool1=m
CONFIG_PACKAGE_sqm-scripts=m
CONFIG_PACKAGE_tc=m
CONFIG_PACKAGE_terminfo=y
CONFIG_PACKAGE_tmux=m
CONFIG_PACKAGE_uhttpd=m
CONFIG_PACKAGE_uhttpd-mod-ubus=m
CONFIG_PACKAGE_unbound=m
CONFIG_PACKAGE_unbound-anchor=m
CONFIG_PACKAGE_unbound-control=m
CONFIG_PACKAGE_watchcat=m
CONFIG_PACKAGE_wifitoggle=m
CONFIG_PACKAGE_wireguard-tools=m
CONFIG_PACKAGE_wpad=m
CONFIG_PACKAGE_zlib=m
CONFIG_PACKAGE_zoneinfo-europe=y
# CONFIG_PER_FEED_REPO_ADD_DISABLED is not set
CONFIG_PREINITOPT=y
CONFIG_TARGET_PREINIT_BROADCAST="10.0.0.255"
CONFIG_TARGET_PREINIT_IP="10.0.0.1"
CONFIG_WPA_SUPPLICANT_INTERNAL=y
# CONFIG_KMOD_BATMAN_ADV_BATMAN_V is not set
# CONFIG_LIBCURL_FTP is not set
# CONFIG_LIBCURL_PROXY is not set

@jow I have pasted the make target/linux/install V=s output here. This is with the modified include/image.mk.

Thanks @jow for saving the day once more. This patch fixes it :smiley:.