Need Help with Host/ package creation

I'm working to bring Rust to OpenWrt. I've hit many walls, but I'm getting to the end of the process (I hope).

Currently, I create tarball installation files and then install them (rust has an issue this corrects). This allows me to save the tarball files and use them instead of recompiling the rustc/cargo toolchains every build.

Does anyone know how the build system stores and validates tar.xz files in ./dl/? I'm sure I'll be digging through the includes, but if anyone has insight, I'll take it. (poke @hnyman)

There are several files that are extracted and installed..

build-manifest-nightly-x86_64-unknown-linux-gnu.tar.xz  rustc-nightly-src.tar.xz
cargo-nightly-x86_64-unknown-linux-gnu.tar.xz           rustc-nightly-x86_64-unknown-linux-gnu.tar.xz
clippy-nightly-x86_64-unknown-linux-gnu.tar.xz          rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
llvm-tools-nightly-x86_64-unknown-linux-gnu.tar.xz      rust-nightly-x86_64-unknown-linux-gnu.tar.xz
miri-nightly-x86_64-unknown-linux-gnu.tar.xz            rust-src-nightly.tar.xz
rust-analysis-nightly-x86_64-unknown-linux-gnu.tar.xz   rust-std-nightly-mips64-unknown-linux-muslabi64sf.tar.xz
rust-analyzer-nightly-x86_64-unknown-linux-gnu.tar.xz   rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz

These files should not change, unless the version changes.

Anyone have any suggestions on how to handle something like this?

Below is what I've got so far. It checks the dist directory to see if it exists and then installs anything in there. Ideally, this would be either in ./dl/ as a single file that I can extract, and then do the work on.

define Host/Prepare
	if [ -d $(HOST_BUILD_DIR)/build/dist ]; then
	   $(call Host/Install)
	   $(call Host/Prepare/Default)

	   # This is our CARGO_HOME, so make sure it's there to receive files
	   [ -d $(CARGO_HOME) ] || (mkdir -p $(CARGO_HOME))

	# Allows outside packages to call $$(BUILD_DIR_HOST)/rust as the dir
	# rather than needing the version number.
	[ -L $(BUILD_DIR_HOST)/rust ] || (cd $(BUILD_DIR_HOST); ln -s "$(PKG_NAME)-$(PKG_VERSION)" rust)
define Host/Install
	cd $(HOST_BUILD_DIR)/build/dist/ && \
	   find -iname "*.xz" -exec tar -x -J -f {} ";"
	cd $(HOST_BUILD_DIR)/build/dist/ && \
	   find ./* -type f -name -execdir sh {} --prefix=$(CARGO_HOME) --disable-ldconfig \;

This sounds like building the version in the name of files (or having a reference version.txt or calling --version on a binary) and checking for existence beforehand based on the "expected version" would work. (correct version, keep going - bad version, remove bad one and build correct one).

"version" can be a SHA256, a commit ID, a timestamp or some internal version to Rust...

1 Like

Do you have a package example that does something like that? that stores internal archives aside from the package one?

No, I don't know of such a package (not that I know too many). It was more trying to be a generic answer...

If you're interested in generating a unique signature for a specific set of files that you can list (for example the ones that identify your package), you could take the sha256 of the files sha256 (for example to take a snapshot of a zip file content before zipping).

Something along the lines of:

find . -type f -exec sha256sum {} \; | sort -k 2 | sha256sum > ../my_package.sha
1 Like

I know the build system has provisions for sha256 checking against a file in ./dl/, I've just not dug into how yet :slight_smile: I'll have to go digging if someone doesn't know offhand. Thank you @VincentR!

usb-modeswitch might be closest with "download, verify and apply external files" approach

the download & verification logic is in include/ and scripts/


This isn't exactly what I need, but it is close enough to show me how to do it, I believe :slight_smile:

@jow, the system allows for checking static hashes, but do you have a way for the build system (Makefile) to generate the combined tar.xz file, sha256sum, and record it somewhere that can be accessed the next time it's built?

Or, does it matter? Ideally for security, we'd ensure the files built were the same and not corrupt or changed from build to build, but.... is it worth it? isn't a great question, but realistically, is it expected to change? Does the buildbot use ./dl/? Or is it just instantiated each time?

Ok.. So, I'm trying to figure out what to do next :slight_smile:

The situation is, I am creating a packed file of compressed folders.

define Host/CreateInstallBinaries
	$(info In CreateInstallBinaries)

	cd $(HOST_BUILD_DIR)/build/dist && \
	   $(RM) *.gz && \
	   $(call dl_tar_pack,$(DL_DIR)/$(RUST_INSTALL_FILE_NAME),.)

I will need to create a sha256 file after creation (manually? by function already defined?), OR, a sub-Makefile that I can run $(SCRIPT_DIR)/ $(CURDIR)/Makefile fix-hash $(3) $(call gen_sha256sum,$(1)) $(2) against to change the hash value when the new install archive is created.. Suggestions?

I need to do the following:

  1. If the $(DL_DIR)/$(RUST_INSTALL_FILE_NAME) exists, I need to run check_hash or verify the hash of the archive, and if so, extract/extract/run to install the already compiled rust package, bypassing the rest of the Host/Configure and Host/Compile, jumping to Host/Install

    • The Host/Install should handle the code for extracting and installing of the final dist archives.
define Host/InstallBinaries
	$(info In InstallBinaries)
	[ -d $(RUST_TMP_DIR) ] \
	   rm -rf $(RUST_TMP_DIR)

	mkdir -p $(RUST_TMP_DIR)


	cd $(RUST_TMP_DIR) && \
	   find -iname "*.xz" -exec tar -x -J -f {} ";" && \
	   find ./* -type f -name -execdir sh {} --prefix=$(CARGO_HOME) --disable-ldconfig \;

define Host/Install
	$(info In Install)
	$(call Host/InstallBinaries)
  1. If the file doesn't exist, or, doesn't match for some reason, toss it and build out the package as normal

    • This would go through the entire process, including creating the new install file, hash, etc.

Edit / Update:

In the first point above, I've changed the Creation function to use existing internal functions

define Host/CreateInstallBinaries
	$(info In CreateInstallBinaries)

	# This creates the installation binary in downloads
	cd $(HOST_BUILD_DIR)/build/dist && \
	   $(RM) *.gz && \
	   $(call dl_tar_pack,$(DL_DIR)/$(RUST_INSTALL_FILE_NAME),.)

	# This updates the Makefile to the new SHA256 Hash for the new
	# binary archive
     $(SCRIPT_DIR)/ $(CURDIR)/Makefile fix-hash \
	   $(call gen_sha256sum,$(RUST_INSTALL_FILE_NAME))

I figure it might be good to track some of the discoveries I've made, since they can be .. interesting to find if you don't already know what you're supposed to be looking for :smiley:

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.