Rust-lang (rustc/cargo) for OpenWrt - testing needed

Community Builds, Projects & Packages
82

Hm I'm hitting an error that I'm not sure what to make of. This is during

Building stage1 compiler artifacts (x86_64-unknown-linux-gnu -> armv7-unknown-linux-musleabihf)

The error message itself is incredibly long, but boils down to

   Compiling rustc_driver v0.0.0 (/home/ubuntu/rutos-mdm9x07-trb1-gpl/build_dir/hostpkg/rust-1.58.0/compiler/rustc_driver)
error: linking with `arm-openwrt-linux-muslgnueabi-gcc` failed: exit status: 1
  |
  = note: "arm-openwrt-linux-muslgnueabi-gcc" [...very long list of options, including:] "-Wl,-Bdynamic" "-lLLVM-13-rust-1.58.0-stable" "-Wl,-rpath,$ORIGIN/../lib"
  = note: /home/ubuntu/rutos-mdm9x07-trb1-gpl/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-8.4.0_musl_eabi/lib/gcc/arm-openwrt-linux-muslgnueabi/8.4.0/../../../../arm-openwrt-linux-muslgnueabi/bin/ld: cannot find -lLLVM-13-rust-1.58.0-stable
          collect2: error: ld returned 1 exit status

At this stage I have several relevant-looking LLVM libraries present at the following paths (under the OpenWRT root):

./build_dir/hostpkg/rust-1.58.0/build/x86_64-unknown-linux-gnu/llvm/lib/libLLVM-13-rust-1.58.0-stable.so
./build_dir/hostpkg/rust-1.58.0/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVM-13-rust-1.58.0-stable.so
./build_dir/hostpkg/rust-1.58.0/build/x86_64-unknown-linux-gnu/stage1/lib/libLLVM-13-rust-1.58.0-stable.so
./build_dir/hostpkg/rust-1.58.0/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-13-rust-1.58.0-stable.so

I can't quite tell if I just don't have the correct environment set (seems more likely), or there's a missing LLVM artifact. @Grommish any ideas/pointers? Not sure if you faced anything like this before.

There's some discussion here: https://github.com/rust-lang/rust/issues/68714 which indicates that it's an environment variable issue. To be honest - and this is maybe the bigger issue - I have very little idea how to troubleshoot this, given how far-removed this step is from the original environment in which the build is run. I don't for example know what the value of $ORIGIN is - it doesn't seem to appear anywhere else in the build log. I also wouldn't even know where to start in terms of trying to set $LIBRARY_PATH (suggestion from the linked issue). Finally, it doesn't seem that I can just attempt to re-run the command in my shell (to try and narrow down the root cause), since it relies on a linker script in a temporary directory that gets cleared after the error.

Any pointers on how I might be able to isolate and troubleshoot this?

[Finally, @Grommish, just wanted to point out that I've slightly tweaked the original options to (a) build version 1.58.0 rather than 1.57.0 and (b) use from the stable rather than the nightly channel. This was actually in an attempt to get some more alignment between the versions, since I noticed that trying to build 1.57.0 with your original script was attempting to build against LLVM-13-rust-1.59.0-nightly - and crashing with the error above - and my initial assumption was that maybe the error was due to a version mismatch]

83

I've updated the PR. This better organizes and uses internal build functions that I was unaware of during the first (of many) iterations.

I attempted to pare back the rust-lang toolchain to a minimum, but I don't know enough about it to do it effectively. I can tell you that working in the Fakeroot adds.. issues.. for things like make install and how it installs itself to the fakeroot and then registers itself. To get around this, I build the dist and then extract them directly to $(STAGING_DIR_HOST). This makes them available to the build system.

I've also added a mips-openwrt-linux-musl patch to the PR so you can test build, although I've not tested it as of this writing because it takes so long.. Should be fine though. I'll edit this with an update once I run it.

Once you've got the toolchain built, anything you need to build with it will need the following:

PKG_BUILD_DEPENDS:=rust/host
TARGET_CONFIGURE_OPTS += CARGO_HOME="$(STAGING_DIR_HOST)/.cargo"

CONFIG_HOST_SUFFIX:=$(shell cut -d"-" -f4 <<<"$(GNU_HOST_NAME)")
RUSTC_HOST_ARCH:=$(HOST_ARCH)-unknown-linux-$(CONFIG_HOST_SUFFIX)

CONFIGURE_ARGS += \
        --target=$(REAL_GNU_TARGET_NAME) \
        --host=$(REAL_GNU_TARGET_NAME) \      
        --build=$(RUSTC_HOST_ARCH)

The CONFIGURE_ARGS flags will change depending on what your package configure will handle. Although I need to verify this, I believe the build system defaults default to --target=$(REAL_GNU_TARGET_NAME)

I can say I'll need to do more cleanup on the whole thing since I'm no longer cross-directing tuples back and forth, but it works for now

84

Thanks for sharing @Grommish - I did finally work out the issue, and I can see it's also fixed in your latest version. Namely, in the previous version, in the build configuration both target and host were set to $RUSTC_TARGET_ARCH, which meant that Rust would build the LLVM library for ARM (or MIPS, etc), and would then try to link against that when building rustc (potentially also for that platform?...not sure), which would fail. Of course the host should be set to $RUSTC_HOST_ARCH. When I changed that it all worked out. FWIW the version I ended up with is here.

I think it's pretty close to your latest - though I'm still going with the OpenWRT<=>Rust/LLVM tuple matching in the makefiles, rather than patching the Rust codebase. I'm still personally averse to the idea of maintaining patches.

I was curious about one aspect of your latest PR: although you define $HOST_CONFIGURE_ARGS here, that variable is not used anywhere. In the previous version there was a ./configure <configure args> stage which would pick up the arguments and generate the config.toml file used by x.py. Is this no longer necessary?

Anyway, my current state is that rustc and cargo are installed and work more or less fine. I'm encountering an issue with one of the dependencies for the specific application I'd like to build, but that seems secondary.

85

HOST_CONFIGURE_ARGS and CONFIGURE_ARGS are automatically applied in Host/Configure when it runs ./configure. Check out TOPDIR/include/host-build.mk and TOPDIR/include/package-defaults.mk to see what is defined be default for those as well as the Prepare, Configure, Compile, and Install defaults for Build/ and Host/

In the latest PR, anything HOST_ or Host/ relates to the Host-Toolchain install. The CONFIGURE_ARGS are used for Build/ runs.

With TARGET_CONFIGURE_OPTS += CARGO_HOME="$(STAGING_DIR_HOST)/.cargo"

If you are building a Host Package, you would also add a

HOST_CONFIGURE_OPTS += CARGO_HOME="$(STAGING_DIR_HOST)/.cargo"

To set CARGO_HOME. My latest revision was still trying out different locations, so I defined it in the Makefile, but that'll just change.

The advantage to not cross-tying the tuples is that I don't have to import/copy the "sorting logic" to each project that uses rust-lang and remains LLVM compliant.

I haven't done testing on 1.58, but will once I finish testing 1.57.0. If I switch revisions, I have to start over :smiley:

You may have problems with stable versus nightly, as nightly is where the Experimental tuples are (like the ones we are building). Keep that in mind if you run into issues building or using the resulting toolchain. I know I set it nightly for a reason, but it was so early on I couldn't tell you exactly why.

If you extract and install the dist archive into STAGING_HOST_DIR, there isn't a need to try and set -lLIBPATH or -Iinclude_stuff

However, if you WANT to play with it, you can use the HOST_ and TARGET_ flags as below:

HOST_CFLAGS +=
HOST_CXXFLAGS +=
HOST_LDFLAGS +=

Which will pass on those flags to the build system.

Example:

TARGET_CFLAGS += -D_GNU_SOURCE
TARGET_CXXFLAGS += -latomic
TARGET_LDFLAGS += -latomic
1 Like
86

Thanks @Grommish for the advice here! I did get it working ultimately, though it was definitely not straightforward. In particular, for someone who's not very familiar with it, the amount of "magic" (through various inter-dependencies, semi-hidden steps, and environment-setting) that the OpenWrt build process does is pretty overwhelming. So I found the number of potential levers to play with when trying to troubleshoot anything (e.g. a linker error) pretty mind-boggling.

Anyway, it seems to work ok now. My version of the Rust host build is here: https://github.com/svet-b/packages/tree/69690bf32ddabee479c96272229b36c37f2197eb/lang/rust. It's a slight tweak on yours, in particular to download the pre-built LLVM binary from CI, rather than spend ~3 hours building it locally (and yes, I know that's cheating by OpenWrt standards, but :man_shrugging: eh).

I also struggled making much sense of the Suricata Makefile since I really just need a simple cargo build, but ultimately managed to figure out what that needs to look like. For anyone interested, below is my minimal Makefile for a Rust Hello World. I also have some more complex sample code that works with an MQTT broker through the paho-mqtt library, and that's based on a nearly identical Makefile - example here.

Sample Makefile for Hello World:

include $(TOPDIR)/rules.mk

PKG_NAME:=rust_helloworld
PKG_VERSION:=0.1.0
PKG_RELEASE:=1

PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/nindalf/helloworld.git
PKG_SOURCE_VERSION:=0ee163afbeeba166d6a163bb08da358306542a5e

PKG_BUILD_DEPENDS:=rust/host

include $(INCLUDE_DIR)/package.mk

CARGO_HOME := $(STAGING_DIR_HOST)/.cargo
RUSTFLAGS="-C linker=$(TARGET_CC_NOCACHE) -C ar=$(TARGET_AR)"

CONFIGURE_VARS += \
        CARGO_HOME=$(CARGO_HOME) \
        RUSTFLAGS=$(RUSTFLAGS)

define Build/Compile
        cd $(PKG_BUILD_DIR) && \
          $(CONFIGURE_VARS) cargo build --release --target=$(REAL_GNU_TARGET_NAME)
endef

define Package/rust_helloworld
        SECTION:=examples
        CATEGORY:=Examples
        TITLE:=Rust Hello World
endef

define Package/rust_helloworld/description
  Hello World in Rust for OpenWrt
endef

define Package/rust_helloworld/install
        $(INSTALL_DIR) $(1)/usr/bin
        $(INSTALL_BIN) $(PKG_BUILD_DIR)/target/$(REAL_GNU_TARGET_NAME)/release/helloworld $(1)/usr/bin
endef

$(eval $(call BuildPackage,rust_helloworld))
87

That is pretty much the experience I had with dealing with the build system. I'd figure out how to do what I needed it to do thru brute force, then figure out where/how what I needed was already in the system, just not documented (or spread out).

If you don't mind, can I use your helloworld package for testing? I don't know rust, but if you do, and would, can the rust Hello World that also testing correct handling of Float (which seems to be the biggest issue I am facing with targets)? Having that be cross-compiled and run on a target device as a verification of the toolchain would be fantastic.

88

Yeah I certainly struggled with the documentation. E.g. it would be great to have a reference for the standard variables that are available for use in Makefiles, like $(TARGET_CC_NOCACHE) or $(TOOLCHAIN_DIR). Thankfully the community support is first-class :slight_smile: .

For sure! The repo it's based on is someone else's by the way - I just figured there was a git repo out there where someone has simply run cargo new helloworld and committed (and indeed there were plenty).

Do you know what a good test for this would be (conceptually)? As part of my sample MQTT pub/sub code, I convert a UTF-8 encoded string to a float (e.g. "1.1" -> 1.1), then multiply that float by 2.0 and convert it back to a string (here). So there's obviously a float operation in there, but I'm not sure if it really pushes the right buttons for testing. I'd certainly be happy to come up with a one-liner - e.g.

println!("Double a float = {:?}", 1.1 * 2.0);

if you have any idea as to the exact requirements. By the way I can confirm that this runs great on a MIPS unit (no hf), after cross-compilation via the toolchain here.

Finally, can also confirm that the resulting binaries are dynamically linked:

root@openwrt-host:~# ldd /usr/bin/helloworld
	/lib/ld-musl-mips-sf.so.1 (0x77da8000)
	libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x77d0e000)
	libc.so => /lib/ld-musl-mips-sf.so.1 (0x77da8000)

That said, the ones that I can build with the current version of the standard Rust toolchain (downloaded with rustup) are also dynamically linked, so this isn't necessarily a significant achievement.

89

This is why we can't cheat and install rustup on the build-host and by-pass the build system :laughing: You run into issues no one ever expected someone building to be in and assumptions by the dev-team on the far end get made (I ran into a similar Suricata5 issue with cross-compiling assumptions on auto-confs)

Also, it looks like the first *-openwrt-linux-musl* rust-lang toolchain was accepted as a Tier 3 target:

1 Like
90 
  Compiling helloworld v0.1.0 (/home/sda3/openwrt/build_dir/target-x86_64_musl/rust_helloworld-0.1.0)
error: linking with `x86_64-openwrt-linux-musl-gcc` failed: exit status: 1
  |

Compile error with make file

91

Hi @qiuzi ..

I've not tested anything x86_64 as a target, and I'm not even sure if the Makefile will support that platform at the moment.

I am making changes to the entire package and how it's outlined. I'm working to validate a working toolchain and that it integrates properly before I start adding other arches.

That being said, I'm more than happy to enable x86_64 if it isn't already when I push the next update to the rust-lang PR.

Are you the one that emailed me? If so, I did receive it, but I've not had a chance to respond yet.

I'm doing some more testing before I send up the latest batch of changes, including developing a Makefile package template to use with Rust packages.

DEPENDS:=@BROKEN @(aarch64||arm||mips||mips64||mipsel||x86_64)

So I did have x86_64 included in the package, however, there is NO toolchain support for anything other than Mips64 and potentially mips at the moment. I will need to update the PR for a x86_64 target.

I have created a template (https://github.com/Itus-Shield/openwrt-rustlang-template/blob/master/Makefile) that can be used for creating packages that include rust-lang.

1 Like
92

I've added ripgrep package for those with rust-lang

93

Thanks to @neg2led, I have a test suite for testing rust-lang cross-compile output for specific outlier cases.

If you want to help test, pick your ARCH below, download and install the ipk file and let me know if it works. It'll show the output below, including the fractal pattern, or, it will SIGILL and exit.

root@OpenWrt:/tmp# /bin/float_test
running on linux mips64 with 0x0 terminal, aspect ratio 0.5
                                                        @

                                                  .
                                              .    @@
                                               .@@@@@@@+
                                               @@@@@@@@..
                                               +@@@@@@@
                                  .     :.  * #::~:@:@=* ..:  .
                                  .@@    .@@@@@@@@@@@@@@@@@@@@     .
                                  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@.
                                 .:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                             .+. ~@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@.
                               .@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                            #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                .            @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
         .@@ .@@@@=@       *@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
          @@@@@@@@@@@@@    .@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
       .@@@@@@@@@@@@@@@@@  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
       @@@@@@@@@@@@@@@@@@@.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  +@@@.@@@@@@@@@@@@@@@@@@@:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
  +@@@.@@@@@@@@@@@@@@@@@@@:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@%
       @@@@@@@@@@@@@@@@@@@.@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
       .@@@@@@@@@@@@@@@@@  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
          @@@@@@@@@@@@@    .@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
         .@@ .@@@@=@       *@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                .            @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                            #@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                               .@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                             .+. ~@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@.
                                 .:@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
                                  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@.
                                  .@@    .@@@@@@@@@@@@@@@@@@@@     .
                                  .     :.  * #::~:@:@=* ..:  .
                                               +@@@@@@@
                                               @@@@@@@@..
                                               .@@@@@@@+
                                              .    @@
                                                  .

root@OpenWrt:/tmp#

I will be adding arches as I generate and begin testing for the various other arches.

Anyone who can help test, it would be greatly appreciated. Once I can verify that the toolchain actually works, I can upstream the tuples. I don't have hardware to test on though, other than mips64.

Edit: A point was raised in the testing, if you test and report back, please let me know what branch you tested on.

mips_24kc on 19.07 was a successful test - Archer C7v5

94

ath79/generic master OK
ramips/mt7621 master OK

2 Likes
95

Added aarch64 float-test ipk to the repo.

96

For anyone who wants to do their own testing, and not use the compiled ipk packages for float_test.

This is the feed you can add for the float_test package after you've pulled the rust-lang PR. It shows up under Testing/ category.

97

Added float-test_0.8.5-1_arm_cortex-a9_neon for armv7-openwrt-linux-muslgnuegabi

98

At this point (March 2023), can I cross-build Rust packages for armv7-openwrt-linux-muslgnuegabi?
(In my case, target is Turris Omnia). If so, how do I get the right rust backend installed?

99

You probably want: rustup target add armv7-unknown-linux-musleabihf
Build with: cargo build --target=armv7-unknown-linux-musleabihf

100

I've been using the rust package quite a lot over the last few weeks for my project, and it works great! Thanks a lot for adding this :slight_smile:

However, I run into issues when compiling to multiple targets within the same environment (using ./scripts/env to switch between configuration). I have tried to document the issue as best as I could here:

In summary, I think rust/host compiles with target support for the selected OpenWrt configuration once, causing issues on OpenWrt configuration changes which requires a new target.

101

Well.. Guess who is back after wandering the wilds :upside_down_face:

For anyone who has tried to email me, I was on sabbatical and foregoing tech. That being said, while I don't know the current status of things (as I'm only now getting back into things and don't even have a real Internet connection yet), assuming no one has actually implemented rust-lang, I'll look back into picking the project up. I've had ideas on a better way to deal with things within the existing Openwrt framework (as it stood when I left)

That is correct. The assumption (and only way it worked at the time) was that anyone making multiple targets would wipe everything out and build the new target from the start to ensure things didn't get mucked up between the two. This led to issues, especially since the host stuff was only compiled once which limited the targets it was aware of. I was thinking on ways of dealing with it when I stepped away.

So, besides the irony of me having to ask the community what the current status of a project I started was - Anyone know?