"Vendoring" source archives in packages while still checking hashes

For Developers
1

I have a private package feed/repository for some in-house applications that run on OpenWRT. Mostly these are either the usual Git repo based ie. a Makefile with PKG_SOURCE_URL set, or occasionally with the source directly in packagename/src (which the buildroot just "knows" what to do with via default rules).

However, we have also accumulated a few vendor tools eg. a manufacturer's debug log capturing utility, a modified fastboot... other things. We get these as archives of source code (so not pre-built). I extract them, create source-based packages from them and check them in.

I want to move to packaging the archives directly ie. have a structure something like:

vendor-tool/
  Makefile
  vendortool_20220607.tar.gz

The pros I see are:

  • no manual extraction/commit steps between "thing the vendor sends me" and "thing in the repo" that might drop or change something
  • can check the hash of the archive in the repo against eg. the last archive you think the vendor sent
  • automated hash checks in builds mean you can be sure you're building the right thing (in CI or locally)

I'm having trouble implementing it though, because OpenWRT's buildroot is made for downloading archives from an external location, not referring to local archives. I don't really want to set up a https archive for a handful of vendor packages (I already have to manage too many git repos for this project).

So far what I've done is followed the instructions for download overrides and written something like:

include $(TOPDIR)/rules.mk

PKG_NAME:=vendor-tool
PKG_VERSION:=1
PKG_RELEASE:=1

PKG_SOURCE:=vendortool_v$(PKG_VERSION).tar.gz
# PKG_HASH:=???

include $(INCLUDE_DIR)/package.mk

UNPACK_CMD=gzip -dc $(PKG_SOURCE) | $(TAR) -C $(1) -xf -
MAKE_PATH:=src

define Package/vendor-tool
    SECTION:=utils
    CATEGORY:=Vendor Cruft
    TITLE:=Vendor Tool
endef

define Package/vendor-tool/description
    Vendor-supplied version of thing
endef

define Package/vendor-tool/install
    $(INSTALL_DIR) $(1)/usr/bin
    $(INSTALL_BIN) $(PKG_BUILD_DIR)/$(MAKE_PATH)/bin/vendor-tool $(1)/usr/bin/vendor-tool
endef

$(eval $(call BuildPackage,vendor-tool))

Without the UNPACK_CMD, the buildroot looks in /dl (or $(DL_DIR)) for $(PKG_SOURCE), when it's actually in the package directory itself.

The above works okay, but there seems to be no easy way to check the hash, which is part of the downloading logic. But overriding the downloading step seems extremely difficult when there's no external URL to point at.

Is there a path of less resistance for managing these kinds of vendored source distributions?

2

Took me a while, but I figured it out.

First, I needed to intervene a little earlier in the process: the download stage instead of the unpack stage. It's nearly impossible to actually override the download stage though, but I can make it copy instead of downloading by setting

# Two slashes, not three, because you want the current directory.
PKG_SOURCE_URL:=file://./

Note that it has to be a directory and not the full path to the file (like it would be for literally any other URL) because there's a check in the download script (scripts/download.pl) that special-cases file:// schemes which the documentation actually says but I never noticed before.

The next thing to note is that the buildroot usually deals with archives with a leading directory component eg. packagename-1.1.1/actual_code_in_here (because usually it's cloning a repo into a subdir referred to by PKG_SOURCE_SUBDIR). So the default un-tar command does a -C $(1)/.. which extracts everything into the directory containing all the package build directories eg. build_dir/target-mipsel_24kc_musl. So I also needed:

TAR_OPTIONS:=-C $(PKG_BUILD_DIR) $(TAR_OPTIONS)

The only thing I couldn't figure out is how to get scripts/download.pl to stop hitting the official mirrors if something didn't work. I don't think it's possible without bypassing it completely.

Anyway, the resulting package Makefile looks something like this:

include $(TOPDIR)/rules.mk

PKG_NAME:=vendor-tool
PKG_VERSION:=1
PKG_RELEASE:=1

PKG_SOURCE:=vendortool_v$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=file://./
PKG_HASH:=NNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNNN

include $(INCLUDE_DIR)/package.mk

TAR_OPTIONS:=-C $(PKG_BUILD_DIR) $(TAR_OPTIONS)
MAKE_PATH:=src

define Package/vendor-tool
    SECTION:=utils
    CATEGORY:=Vendor Cruft
    TITLE:=Vendor Tool
endef

define Package/vendor-tool/description
    Vendor-supplied version of thing
endef

define Package/vendor-tool/install
    $(INSTALL_DIR) $(1)/usr/bin
    $(INSTALL_BIN) $(PKG_BUILD_DIR)/$(MAKE_PATH)/bin/vendor-tool $(1)/usr/bin/vendor-tool
endef

$(eval $(call BuildPackage,vendor-tool))
2 Likes
3

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.