Rust-lang (rustc/cargo) for OpenWrt - testing needed

Community Builds, Projects & Packages
62

Hi, strange that it is giving the errors on your side. I got it working without errors on my side even without the configure arguments. I don't have rust installed on the host, only in the toolchain through your lang/rust package. I'm now working to build the std crate package and disable static linking.

My update on github is now working perfect, even pulling in external crates and building them. I only need to have the Cargo.toml and source files in the src directory. In my setup the IPK is build ok and I can install using opkg which is good.

I didn't use the ac_cv_path_ variables nor CONFIGURE_VARS as I'm not using autoconf, cmake or not even GNU Make but pure Rust Cargo. The key was to use cargo rustc instead of cargo build and pass in the correct compiler and linker with the $(CARGO_HOME) variable.

I think the Makefile is now quite universal and we could send in a patch in the OpenWRT build system to have a rust.mk for rust packages which define these build functions. What do you think?

I now read that Rust is not ABI stable which is off course bad for OpenWRT where binary size is of the utmost importance. However as OpenWRT is compiled from source this should not be a huge issue as upgrading rust just means to recompile all rust binaries in the toolchain and doing a sysupgrade.

My first attempts as creating the libstd-**.so are not successful, makefile here: daanpape/rust-libstd: OpenWRT package which provides the libstd-*.so shared library (github.com)

UPDATE
I now realized that the libstd-**.so is already compiled by the rust package itself and it's living in the staging_dir of the toolchain. I have updated the makefile of the rust-libstd package to create an IPK file of it. I also updated the rust-hello-world example to make use of this. It runs perfectly on my MIPS softfloat target (Atheros Qualcomm AR9331). I think this is a good setup for OpenWRT devices as you can now write multiple rust applications without having having a ton of RAM and flash usage. The only downside to this is that for every crate that you import a separate package should be made which also compiles this external crate with dynamic linking, for some crates this could become tedious. It would be handy if you could select which crates to link in dynamically (libstd) and which statically, I don't know if this is possible?

Current sizes:

rust-hello-world 4204B (4.11kB)
libstd-61e1ef472d25ade8.so 2029380B (1981.82kB)

The libstd thus is very very large, as a comparison libc (musl) is 626692B (612KB)

63

AFAIK, MUSL is dynamically linked, and I set that in the linux_musl.rs file.

I am not a Programmer, nor do I understand Rust, but I wanted a package (Suricata6) that required it, so I made it work :slight_smile:

Ny test-bed is a MIPS64 device, so I can't test for other platforms. I also was looking into creating rustc/cargo artifacts for the target systems, but LLVM doesn't seem to like MIPS64, so I'm at a dead-end for that. I will revisit in the future. For now, this package is strictly used as a HOST toolchain.

I'm seeing the error and you aren't because you are hard-coding your links

define Package/rust-hello-world/install
        $(INSTALL_DIR) $(1)/usr/sbin
        $(INSTALL_BIN) $(PKG_BUILD_DIR)/target/mips-openwrt-linux-musl/release/main $(1)/usr/sbin/rust-hello-world
endef

$(PKG_BUILD_DIR)/target/mips-openwrt-linux-musl/release/main doesn't exist for me, and is giving the error because of it. but, that is to be expected.

Also, any suggestions from those who actually use Rust would be a tremendous help. I'm currently at the point of trying to shrink the dist file footprint to set as a "default" for compiler options, and then let people break it/balloon it as they see fit.

64

Hi @Grommish,

For not being a programmer you are doing a heck of a job for which I'm very grateful. The error is indeed a bad mistake from my side and I have now fixed it. The examples I've made should now work.

I'm now focussing on optimizing size as I think this is more important than speed optimizations on our OpenWRT platforms. I have just tested and rust is doing what I want out of the box:

  • The libstd is now always linked in dynamically
  • All libraries that rust can't find in the toolchain will be linked in statically.

This is very nice behavior as this let's you choose which library you make available as an OpenWRT package and which not. This is exacly what you want on an embedded device, so far so good.

I' now convinced that we should work on the lang/rust package that you made and optimize the prebuilt libstd with the following:

  • Abort on Panic instead of doing the backtrace
  • Remove all debug symbols (this is done already in my example)
  • Compile with optimization level z
  • Strip all unnecessary symbols, just doing a string command on the precompiled lib shows that there is a lot to win.

I'm going to try and find some time to do this. Maybe you already know where the optimization options can be set?

65

In the Config.in file, it holds the switches exposed by ./configure, with the exception of --set, which can change anything in the config.yaml - for good or ill, I've had some feedback in the PR about actually being able to turn certain switches off, for example.

I'm still working on reducing the dist installation file size for the host (currently ~350Mb for my x86_64-mips64 toolchain, down from 550Mb). I just want to find a default that's as minimal as possible.

Keep in mind that I've yet to compile (for example) a rustc/cargo stage2 for MIPS64 itself (Cross-compile is fine) because of the LLVM issue, but in theory, if the OpenWrt build system will build on the host, you should be able to make rust

66

updated PR - added mipsel support

67

Is there an example of an OpenWrt Makefile for a package written in Rust?

68

This started so I could play with Suricata6, which uses Rust. See the PR below

1 Like
69

I have updated the PRs for Rust, Suricata 6, and LibHTP

Rust now splits out the Host and Target Rust toolchains, and dealing with multiple target toolchains are now handled better

Suricata saw updates to the Build Logic and DEPENDS, moved LibHTP external to Suricata package

LibHTP introduced external to Suricata.

Any questions/issues/etc, let me know.

2 Likes
70

Hello,
I was able to install Rust using the standard rustup script.

root@OpenWrt:~# curl https://sh.rustup.rs -sSf | sh
info: downloading installer
Warning: Not enforcing strong cipher suites for TLS, this is potentially less secure

Welcome to Rust!
...
Current installation options:


   default host triple: x86_64-unknown-linux-musl
     default toolchain: stable
               profile: minimal
  modify PATH variable: yes

info: default toolchain set to 'stable-x86_64-unknown-linux-musl'

  stable-x86_64-unknown-linux-musl installed - rustc 1.53.0 (53cb7b09b 2021-06-17)


Rust is installed now. Great!

To get started you may need to restart your current shell.
This would reload your PATH environment variable to include
Cargo's bin directory ($HOME/.cargo/bin).

To configure your current shell, run:
source $HOME/.cargo/env
root@OpenWrt:~# source $HOME/.cargo/env
root@OpenWrt:~# rustc --version
rustc 1.53.0 (53cb7b09b 2021-06-17)
root@OpenWrt:~# cargo --version
cargo 1.53.0 (4369396ce 2021-04-27)

However, I get a linker error when I try to compile a simple "hello world" app

root@OpenWrt:~# cargo new testapp
     Created binary (application) `testapp` package
root@OpenWrt:~# cd testapp/
root@OpenWrt:~/testapp# cargo run
   Compiling testapp v0.1.0 (/root/testapp)
error: linker `cc` not found
  |
  = note: No such file or directory (os error 2)

error: aborting due to previous error

error: could not compile `testapp`

To learn more, run the command again with --verbose.
root@OpenWrt:~/testapp#

Please how do I fix this? Thanks.

71

Anyway, I am able to cross-compile using cargo build --target=x86_64-unknown-linux-musl on Linux (Ubuntu) and then run the application in OpenWRT.

72

This is for creating the rust toolchain to be included as part of the OpenWrt, not for installing rustc/cargo on the device itself. While it potentially be done, it's something for much rather down the line.

This is because there is limited gcc support for OpenWrt. It isn't designed to be able to be a build environment.

You are able to run the rustup because you are using an x86_64 device. The grand majority of devices, unfortunately, aren't. If you are building on x86_64 for x86_64 using the host's x86_64 rustc/cargo installation, then yes, it should work fine. But that's beyond the rust-lang package because it relies on the host machine having a given package. You could always have cross-compiled rust applications for OpenWrt if you were willing to accept making rust a build-environment requirement.

You should also be aware that Openwrt reports it's own triple in rust as x86_64-openwrt-linux-musl

I can't help you with compiling rust applications or using rustc/cargo on the OpenWrt device. At the very least, the missing build tools will be a hinderence, and musl is dynamically linked (and rustup is statically linked)

If you ever want a real shot of OpenWrt having rust-lang support, I'd check out the PR. Although I've shifted my efforts to other things, I'm still kicking around on rust-lang and suricata

73

I agree with you. Installing the standard rustc/cargo binaries on a OpenWRT device is not worth the stress right now because cross-compilation on a Linux machine works perfectly.
At least, I was able to easily cross-compile a warp web server application on a Linux machine (it uses tokio, serde, hyper and a ton of other Rust crates) and then run the application smoothly on OpenWRT.

For now, I will stick with the cross-compilation option. Please what's the current status of the Rust toolchain for OpenWRT? I will be great to have Rust by default under the language packages.

1 Like
74

It is in Draft PR status because it needs more wide-range testing. It works for the devices I have to actually test it, and overall should probably be redone to slim it down and see what other ways I can do it. It's still a WIP but you're welcome to test it.

75

I've been able to start back on rust-lang, and have been pushing updates.

I think I've finalized aarch64, arm, armv7, mips, mipsel, mips64, and x86_64. The arm/armv7 should now correctly determine if the target needs hard-float or not.

powerpc has been dropped from the package for now, as it seems to have LLVM issues. For anyone who wants to test rust-lang for powerpc, I've left provisions in the Makefile to pass the correct defines (--D__ppc__) to LLVM for powerpc targets if it's called directly (make package/feeds/packages/rust/host/{clean,compile}), but it isn't enabled in the rust-lang package DEPENDS

This is simply at the stage rust-lang toolchain compiles, and I've not tested that they successfully cross-compile, or that the cross-compiled package actually works correctly, for any ARCH other than mips64 and mipsel. I have no reason to suspect it won't work, but early on I ran into issues with mips64 where it compiled both the toolchain and suricata6 but would immediately SIGILL because of the static vs dynamic linking (musl issue). I had no indication of an issue until it just died on the device.

If anyone would like to test and see if rust-lang builds for your target, I'd appreciate the feedback. If anyone has a target ARCH not listed, let me know (I don't play outside of the very confined world of the Octeon MIPS64 branch often) as I'm not up on the other ARCHs.

  21M dl/rust-1.56.1-aarch64-unknown-linux-musl-install.tar.xz
  20M dl/rust-1.56.1-arm-unknown-linux-musleabihf-install.tar.xz
  20M dl/rust-1.56.1-armv7-unknown-linux-musleabihf-install.tar.xz
  22M dl/rust-1.56.1-mips64-unknown-linux-muslabi64-install.tar.xz
  20M dl/rust-1.56.1-mipsel-unknown-linux-musl-install.tar.xz
 364M dl/rust-1.56.1-x86_64-unknown-linux-gnu-install.tar.xz
  22M dl/rust-1.56.1-x86_64-unknown-linux-musl-install.tar.xz
 109M dl/rust-1.56.1.tar.xz

I'm also verifying arm-unknown-linux-musleabi and expect it should compile correctly.

Appreciate any help or feedback!

1 Like
76

Hi @Grommish thanks a ton for your work on this! I'm working on building Rust code for a couple of devices running OpenWRT, and what you've done here is hugely valuable.

That said, when running what I believe is your latest, I came across a small issue. I think I've maybe addressed it, but am still waiting for the build to complete, so figured I'd post here in the meantime

What I did was to apply your patch to the version of openwrt/packages (@ this fairly recent commit) that's used by the Teltonika SDK, which is relevant for the routers I'm building for. Although this is probably not 100% the envisaged approach, it feels like it ought to work. And it mostly does. I can request to build a sample Rust program, which triggers building of the toolchain, and that mostly builds, but then I get to an error such as

Building stage1 std artifacts (x86_64-unknown-linux-gnu -> mips-openwrt-linux-musl)
error: failed to run `rustc` to learn about target-specific information

Caused by:
  process didn't exit successfully: `/home/ubuntu/rutos-ath79-rut9-gpl/build_dir/hostpkg/rust-1.57.0/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names -Zsymbol-mangling-version=legacy -Zmacro-backtrace '-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Ctarget-feature=-crt-static -Zsave-analysis -Cprefer-dynamic -L native=/home/ubuntu/rutos-ath79-rut9-gpl/staging_dir/toolchain-mips_24kc_gcc-8.4.0_musl/lib -Cembed-bitcode=yes '-Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/nightly/")' --target mips-openwrt-linux-musl --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg` (exit status: 1)
  --- stderr
  error: Error loading target specification: Could not find specification for target "mips-openwrt-linux-musl". Run `rustc --print target-list` for a list of built-in targets

Now, mips-openwrt-linux-musl is indeed not a valid target that's recognized by rustc. That should be mips-unknown-linux-musl, right? I've not really been able to track down where that discrepancy arises, but it feels to me like $RUSTC_TARGET_ARCH gets incorrectly set to the "raw" target triple, rather than the Rust-specific one. From what I can tell the solution is to switch the commenting towards the end of rust_targets.mk, so that

RUSTC_TARGET_ARCH:=$(ARCH)-unknown-linux-$(patsubst "%",%,$(RUST_TARGET_SUFFIX))

I'm now trying to build with that. Initially got a weird ld error about mismatch in "VFP register arguments" (potentially related to the hard vs soft float issue you mentioned), but have now cleaned the build tree and am building from scratch just to be sure it wasn't a conflict with a previous build.

Finally, do you still have a method for using rustup to install the Rust toolchain? I used that to build binaries for the architectures that I'm interested in (armv7 and mips) outside of the OpenWRT tree, and it worked fine - so I'm wondering if there's a way to integrate this into the OpenWRT build process (for supported architectures), to avoid building rustc from source.

1 Like
77

Hi!

Couple of things:

  1. rustup won't work because rustup is statically linked, and OpenWrt MUSL requires dynamically linked libraries. I wish we could use rustup..

  2. OpenWrt uses xxxx-openwrt-linux-musl for the tuples. I'm working on the best way to integrate this into Openwrt and Rust-lang. I have access to a MIPS64 device, so that's what I'm currently testing. Previously I had MIPS/MIPS64/Aarch64/x86_64 working (ARM, ARMv5/v6/v7 is proving a pain).

I've gone from cross-linking tuples from xxxx-openwrt-linux-musl to the built-in rust-lang tuples (xxxx-unknown-linux-muslxxx) to just upstream'ing the tuples as Tier 3 targets (https://github.com/rust-lang/rust/pull/92300)

I've been trying to figure out the best way to integrate long-term into the build system and potentially be able to introduce rustc and cargo natively on-device for the higher-end devices that might need it (although that will depend on LLVM adding support for the target Arch)

Compiling rust-lang takes about 4 hours on my device from a clean slate, so it just takes time to build, test, fail, repeat :smiley:

ARM Targets are going to be tricky because OpenWrt uses multiple ways to define ARM functionality (like VFP/NEON) across the trees, so that'll have to come later.

grommish@DESKTOP-AW:~/openwrt/staging_dir/host/bin$ file rustc
rustc: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=76008b88647eb342385535fc311bfaf0d769c225, for GNU/Linux 3.2.0, not stripped

This is my cross-compiled rustc for the build system (used to cross).

I'll probably push an update here soon, as I've been trying to organize the Makefiles to better be readable and organized.

I can push a MIPS enabled commit, if you'd like to test

78

I should add: I'm not a programmer and I don't know rust. So, if you actually know rust, it would be very helpful if you'd be willing to help test that what actually builds, builds correctly.

79

Thanks for the quick response! A few things to unpack here; let's maybe start with this:

To be clear, the second part of this isn't an objective of mine. I agree it would be neat to do that, but I'm currently simply trying to cross-compile Rust applications to run on constrained OpenWRT devices. And indeed to do that as part of the OpenWRT build and firmware packaging process. [The cross-compilation of individual binaries outside of the OpenWRT build process is quite doable, by e.g. following https://github.com/japaric/rust-cross - but that's besides the point here.]

I wasn't familiar with this, but after a bit of reading (e.g. here), I think I'm kind of getting the picture. Is the actual issue that the rustc compiler that's been installed via rustup will always produce statically-linked binaries on musl? (I assume it's not about the rustup/rustc executables themselves being statically linked)

While I agree it's not ideal to have statically linked binaries, it does seem that such binaries would nonetheless run fine on the target (at least they do in my tests). In that regard I'm not sure what you mean when you say that OpenWRT "requires dynamically linked libraries". Can you elaborate, or would you be able to point me to any materials that I can use to educate myself on this? (am a bit of an OpenWRT novice).

The upstreaming seems like a worthwhile effort, but potentially also a lot of work to implement and maintain across all relevant tuples? I don't have a clear sense of what's best here. From a practical perspective, given that no openwrt tuples are currently recognized by rustc upstream, having some mechanism to match to a valid RUSTC_TARGET_ARCH seems like the only viable near-term option.

On that note, I've made some updates to your rust_targets.mk to correctly parse mips (not mips64) and at least some arm architectures. For example on one device I have

CONFIG_CPU_TYPE="cortex-a7+neon-vfpv4"

which the original rust_targets.mk did not recognize as having hard-float capabilities. I changed some instances of $(filter ... to $(findstring ... so that the neon and vfp keywords would be picked up even if not surrounded by whitespace. But yes, maybe trying to do this reliably for all tuples would end up being quite brittle - to your point, especially for all the ARM variants.

I think I've got it sorted (with the above), but may come back to you on that - thanks for the offer!

Yeah, similar here unfortunately - hence the allure of rustup :slight_smile: .

As well as being an OpenWRT noob, I'm also a bit of a Rust noob, so probably not the best person to give a definitive stamp of approval :). But in guess in both our cases we're running applications with defined functionality, so hopefully it should be fairly easy to tell if they do what's expected or not. (I would also be a bit surprised if everything compiles successfully, but then - silently - does something different to what it's supposed to during execution)

1 Like
80

I'm a hard-agree with this, BTW. The ability to create the packages is the goal, and one that works fine. My issue has been getting the cross-compiled package (in this case, suricata6) to work to validate rust-lang. It's the second part I've been having issues with.

Upstreaming the tuples isn't any harder than maintaining the local patches/ to add the OpenWrt specific tuples. The patch file I created for the rust PR could be dropped into feeds/packages/lang/rust/patches (and, in fact, was before I cloned rust-lang to submit the PR)

As you can see, at one point I worked to cross-map tuples between Openwrt's and the GNU standard tuple that rust-lang (and LLVM) use. Trying to sort out the ABI was reasonable on everything except ARMv7, and you saw the spaghetti waterfall of checks. Ideally, just using the tuple Openwrt wants to use internally keeps everything the same across the entire build environment. The only downside is the PR to rust-lang vs an update PR to openwrt-packages. But, it isn't like that would be an issue outside of rust-lang version updates, which would require an update anyway.

There were a few issues.. You can read them on the rustup PR (https://github.com/openwrt/packages/pull/13040)

My method has always been make it work and then figure out how to make it work while doing it right. I honestly couldn't tell you if a -mhard-float target would work with rustup because it was a non-starter once I learned it.

When I asked about the dynamic vs static linking, below is what I was told and have gone with.

I'm always open to suggestions and comments!

1 Like
81

Fair enough. I was thinking more about doing simple "matching" of tuples that are already supported by rust-lang, but I take your point that that can be a mess also. I guess it largely comes down to whether it's easier to get PRs merged into rust-lang or openwrt/packages :).

Thanks - this is really helpful context. To be honest I'm not sure just how different the rustup behavior is from the way that other languages are built. I do feel like there is often an element of binary downloads (e.g. of dependencies), though maybe it's the caching element that rustup doesn't do well. My gut feeling is that the ultimate solution would be to have the rustup behavior improved in a way that brings it in line with what's acceptable and desirable within the OpenWRT build process, but maybe that's a pipe dream.

Anyway, happy to stick with building from source for now. On that note, I've been playing around with trying to reduce the build time to something less ridiculous. E.g. by removing builds of unneeded tooling (rls, rustfmt, docs, linters, etc), to only leave rustc and cargo. This still doesn't help with (by far) the biggest resource-hog, which is building LLVM from source. In principle the best way to address this would be to set the config option llvm.download-ci-llvm to true or if-available, and just get a pre-built LLVM binary. This is available for all Tier 1 and Tier 2 target tuples (list), which is a nice start though obviously not exhaustive.

All that said - I cannot get the LLVM binary download to work currently, for what seems like a silly reason. Attempting to use that option triggers some git commands in bootstrap.py (here) to obtain the currently checked out commit hash, which is used to figure out the correct LLVM build to download. But since in the OpenWRT build the source tree comes from an archive, and not a checked-out git repo, the git commands fail. Short of patching bootstrap.py, I'm not sure how to address this.

Anyway, I do feel like I'm close to getting the compiler to build properly. Once that's done I'll look into the dynamic vs static linking a bit more. Among the many discussions on the topic, it's not always clear whether people are talking about (a) building a dynamically linked rustc compiler on a musl host, or (b) having rustc build dynamically linked binaries for a musl target. Obviously it's the latter that's most relevant here.