I'm working to bring Rust to OpenWrt. I've hit many walls, but I'm getting to the end of the process (I hope).
Currently, I create tarball installation files and then install them (rust has an issue this corrects). This allows me to save the tarball files and use them instead of recompiling the rustc/cargo toolchains every build.
Does anyone know how the build system stores and validates tar.xz files in ./dl/? I'm sure I'll be digging through the includes, but if anyone has insight, I'll take it. (poke @hnyman)
There are several files that are extracted and installed..
These files should not change, unless the version changes.
Anyone have any suggestions on how to handle something like this?
Below is what I've got so far. It checks the dist directory to see if it exists and then installs anything in there. Ideally, this would be either in ./dl/ as a single file that I can extract, and then do the work on.
define Host/Prepare
if [ -d $(HOST_BUILD_DIR)/build/dist ]; then
$(call Host/Install)
else
$(call Host/Prepare/Default)
# This is our CARGO_HOME, so make sure it's there to receive files
[ -d $(CARGO_HOME) ] || (mkdir -p $(CARGO_HOME))
fi
# Allows outside packages to call $$(BUILD_DIR_HOST)/rust as the dir
# rather than needing the version number.
[ -L $(BUILD_DIR_HOST)/rust ] || (cd $(BUILD_DIR_HOST); ln -s "$(PKG_NAME)-$(PKG_VERSION)" rust)
endef
define Host/Install
cd $(HOST_BUILD_DIR)/build/dist/ && \
find -iname "*.xz" -exec tar -x -J -f {} ";"
cd $(HOST_BUILD_DIR)/build/dist/ && \
find ./* -type f -name install.sh -execdir sh {} --prefix=$(CARGO_HOME) --disable-ldconfig \;
endef
This sounds like building the version in the name of files (or having a reference version.txt or calling --version on a binary) and checking for existence beforehand based on the "expected version" would work. (correct version, keep going - bad version, remove bad one and build correct one).
"version" can be a SHA256, a commit ID, a timestamp or some internal version to Rust...
No, I don't know of such a package (not that I know too many). It was more trying to be a generic answer...
If you're interested in generating a unique signature for a specific set of files that you can list (for example the ones that identify your package), you could take the sha256 of the files sha256 (for example to take a snapshot of a zip file content before zipping).
I know the build system has provisions for sha256 checking against a file in ./dl/, I've just not dug into how yet I'll have to go digging if someone doesn't know offhand. Thank you @VincentR!
This isn't exactly what I need, but it is close enough to show me how to do it, I believe
@jow, the system allows for checking static hashes, but do you have a way for the build system (Makefile) to generate the combined tar.xz file, sha256sum, and record it somewhere that can be accessed the next time it's built?
Or, does it matter? Ideally for security, we'd ensure the files built were the same and not corrupt or changed from build to build, but.... is it worth it? isn't a great question, but realistically, is it expected to change? Does the buildbot use ./dl/? Or is it just instantiated each time?
The situation is, I am creating a packed file of compressed folders.
define Host/CreateInstallBinaries
$(info In CreateInstallBinaries)
cd $(HOST_BUILD_DIR)/build/dist && \
$(RM) *.gz && \
$(call dl_tar_pack,$(DL_DIR)/$(RUST_INSTALL_FILE_NAME),.)
endef
I will need to create a sha256 file after creation (manually? by function already defined?), OR, a sub-Makefile that I can run $(SCRIPT_DIR)/fixup-makefile.pl $(CURDIR)/Makefile fix-hash $(3) $(call gen_sha256sum,$(1)) $(2) against to change the hash value when the new install archive is created.. Suggestions?
I need to do the following:
If the $(DL_DIR)/$(RUST_INSTALL_FILE_NAME) exists, I need to run check_hash or verify the hash of the archive, and if so, extract/extract/run to install the already compiled rust package, bypassing the rest of the Host/Configure and Host/Compile, jumping to Host/Install
The Host/Install should handle the code for extracting and installing of the final dist archives.
define Host/InstallBinaries
$(info In InstallBinaries)
[ -d $(RUST_TMP_DIR) ] \
rm -rf $(RUST_TMP_DIR)
mkdir -p $(RUST_TMP_DIR)
tar -C $(RUST_TMP_DIR) -xJf $(DL_DIR)/$(RUST_INSTALL_FILE_NAME)
cd $(RUST_TMP_DIR) && \
find -iname "*.xz" -exec tar -x -J -f {} ";" && \
find ./* -type f -name install.sh -execdir sh {} --prefix=$(CARGO_HOME) --disable-ldconfig \;
endef
define Host/Install
$(info In Install)
$(call Host/InstallBinaries)
endef
If the file doesn't exist, or, doesn't match for some reason, toss it and build out the package as normal
This would go through the entire process, including creating the new install file, hash, etc.
Edit / Update:
In the first point above, I've changed the Creation function to use existing internal functions
define Host/CreateInstallBinaries
$(info In CreateInstallBinaries)
# This creates the installation binary in downloads
cd $(HOST_BUILD_DIR)/build/dist && \
$(RM) *.gz && \
$(call dl_tar_pack,$(DL_DIR)/$(RUST_INSTALL_FILE_NAME),.)
# This updates the Makefile to the new SHA256 Hash for the new
# binary archive
$(SCRIPT_DIR)/fixup-makefile.pl $(CURDIR)/Makefile fix-hash \
$(RUST_INSTALL_FILE_HASH) \
$(call gen_sha256sum,$(RUST_INSTALL_FILE_NAME))
endef
I figure it might be good to track some of the discoveries I've made, since they can be .. interesting to find if you don't already know what you're supposed to be looking for
I'll have to go digging if someone doesn't know offhand. Thank you 
