Building OpenWrt ON OpenWrt - A Howto

It's been noted before that OpenWrt isn't self hosting. Well, with only a very few additions (but with a bunch of bug workarounds) it can be! My BPI-R64 running snapshot just made its own build. This was a learning experience about the OpenWrt build system, and it exposed a LOT of bugs in things like LLVM, musl, perl, and in OpenWrt itself. But it was fun. This is still, of course, experimental. But I'll go into the steps you can use to replicate this:

First: Why Build OpenWrt on OpenWrt?

  1. Becoming self-hosting is an important milestone for any OS
  2. It makes building OpenWrt more accessible for those with less resources.
  3. Continuously building on a mono-culture (x86-based full blown Linux systems) lets a lot of bugs slide under the radar. Building on a different system can (and did) expose a lot of issues.
  4. For the brownie points of knowing (or bragging) that the OS on your device was BUILT on your device.

Hardware Requirements

  • Device with 1GHz+ CPU, preferable dual core or better
  • 512MiB RAM per core you intend to use**1
  • 2GiB swap space on non-built-in storage (sd card if your playform supports it, or USB stick) **2
  • 2GiB of space on whatever your rootfs is
  • ~26GiB storage space for the build, **2

Note1: Building on 64-bit systems is more RAM-hungry than building on 32-bit systems. You can get away with a little less ram-per-core on 32-bit systems. You can also get away with a little less ram-per-core if you can use a pre-built LLVM.
Note2: This should be a fairly good quality storage device. Certain steps in the process will be somewhat heavy on the swap. A good quality storage device will handle the wear better. A larger storage device will spread the wear more.

Setting Up The Dev Environment
You need a whole lot of packages:

  • Some good-to-have packages to make life easier:
    opkg install joe joe-extras bash htop whereis less file findutils findutils-locate chattr lsattr wget-ssl xxd
  • The dev tools themselves:
    opkg install pkg-config make gcc diffutils autoconf automake bzip2 check git git-http patch libtool-bin
  • Perl:
    opkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b **3
  • A bunch of coreutils, since busybox versions aren't good enough in some cases, or aren't there:
    opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq
  • Other miscellaneous tools:
    opkg install grep rsync tar python3 getopt procps-ng-ps gawk sed xz unzip gzip flock
  • A few libs:
    opkg install libncurses-dev zlib-dev musl-fts
  • We need a softlink for libncurses:
    ln -s libncursesw.a /usr/lib/libncurses.a

Note3: My system had a lot of perl installed already, so there may be more perl packages necessary that aren't listed here.

Before we can go further, we have a few Initial bugs to fix:

  • OpenWrt Bug: OpenWrt doesn't put execute permissions on the scripts for the automake, autoconf, and libtool packages. This prevents a lot of build systems from working correctly. To fix:
    chmod +x /usr/share/automake-1.16;chmod -x /usr/share/automake-1.16/COPYING /usr/share/automake-1.16/INSTALL
    chmod +x /usr/share/autoconf/Autom4te/*
    chmod +x /usr/share/libtool/build-aux/*

Now we can install the three dev libraries that OpenWrt doesn't support. One of them is included in OpenWrt as a non-dev library, but it's stripped in such a way as you can't compile against it. These three libs are all to support GNU LIBC extensions that musl doesn't have and are required mostly for elfutils. Switch to your ~/devel folder (or wherever you're going to use to build things) and do the following:

  • libargp:
    git clone https://github.com/xhebox/libuargp.git
    cd libuargp
    make
    make prefix=/usr install
    ln -s libargp.so /usr/lib/libargp.so.0
    cd ..
  • libfts: We installed the non-dev version above already, so this will overwrite it with one that can be compiled against. We installed the non-dev version first so that it doesn't get installed later and overwrite your dev version with a lobotomized one:
    git clone https://github.com/void-linux/musl-fts.git
    cd musl-fts
    ./bootstrap.sh
    ./configure --prefix=/usr
    make
    make install
    cd ..
  • libobstack:
    git clone https://github.com/void-linux/musl-obstack.git
    cd musl-obstack
    ./bootstrap.sh
    ./configure --prefix=/usr
    make
    make install
    cd ..

Now a few more bug fixes and issue workarounds:

  • OpenWrt Workaround: Not a bug per se, though one might call it a bug by omission. The bzip2 binary also acts as bunzip2 and bzcat, but it can only do so if the proper softlinks are made when it's installed, which the OpenWrt package doesn't do. So:
    ln -s bzip2 /usr/bin/bunzip2
    ln -s bzip2 /usr/bin/bzcat
  • Elements of the build system don't like to be built as root. OpenWrt is essentially single-user, so to get around this:
    export FORCE_UNSAFE_CONFIGURE=1
    You might want to add the above to your ~/.profile
  • Speaking of .profile, I use bash as my shell. I suspect that the busybox Ash isn't good enough for some of the scripts in the build system, so I recommend installing bash and setting it as your /bin/sh
    opkg install bash (already included above)
    ln -sf bash /bin/sh

The rest of the bug workarounds, and the installation of the one tool we're still missing are all done in the OpenWrt source tree itself. So let's get it:

Get and Build OpenWrt
This is based on the wiki instructions, with a bunch of deviations to fix various bugs.

Make sure you're in your devel directory.

  • First the preliminaries:
    git clone https://git.openwrt.org/openwrt/openwrt.git
    cd openwrt
    git checkout master
    git pull
    ./scripts/feeds update -a
    ./scripts/feeds install -a
  • Get the config file for your architecture and save it as .config. For me (BPI-R64) it is this:
    wget https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/config.buildinfo -O .config
  • Configure your build:
    make defconfig
    make menuconfig
    Things to change in the config: set it up to build for only your platform, read the next step below about LLVM and configure whether or not you're going to use a pre-built one, and then make any other tweaks you normally do.
  • EITHER get a pre-made LLVM (preferable)...
    Compiling LLVM is the single most time and resouce-intensive step in the whole process. It can easily take two days by itself. I have so far built it on two platforms: aarch64 and armv7l. If you have one of those platforms, you can grab my pre-made LLVM and untar it into the openwrt build folder. If not, you will have to...
  • ...OR Build LLVM
    BE CAREFUL HERE! Early in my experiments I burned out a Sandisk Extreme 64GB microSD card because I did a multi-core build and the memory requirements to build LLVM pushed the system so far into swapping that it did about 18 hours of non-stop writes. A single-core build will take a long time, but is safer. This is going to take a long time, so best use nohup in case you get disconnected - you don't want compilation to stop. And start getting into the habit of using V=sc and logging everything, because this whole process is still experimental:
    nohup nice -10 make -j 1 V=sc tools/llvm-bpf/compile > ~/llvmcomp01.out &
    Periodically check htop to make sure it is going well, and check the log when it stops. Hopefully there's no error, but if there is, welcome to the experiment. Debug, rinse and repeat and report it here for the rest of us.
  • Now we build the rest of the toolchain. Now that LLVM is out of the way, if you have multiple cores and enough RAM per core, then you should be ok to use multi-core compiling. Keep an eye on it, though and htop is your friend here. I am still going to put -j 1 in all the copy/paste commands for safety - but feel free to up the core count at your discretion.
    nohup nice -10 make -j 1 V=sc toolchain/install > ~/toolchain01.out &
  • Now that the toolchain is done we can do a make download. A bug in qosify's makefile prevents make download from working before the toolchain is complete in some cases.
    make download
  • We will now build the last tool we're missing: rev, which is a little-known and littler used unix tool found in util-linux. OpenWrt doesn't make a package out of it, even though it's built, so we're going to build util-linux then copy it out. This is quick, prolly no need to log it:
    make -j 1 package/util-linux/compile
    cp $(find . -wholename \*ipkg-install/usr/bin/rev) /usr/bin/rev
  • Now we're going to fix some bugs. One which causes a problem with perl on musl, one where recent compilers cause a seg fault, and one in Perl's Configure script itself. Fort this we need two patches. The first patch we will apply directly:
    ( cd feeds/packages/lang/perl && wget https://va1der.ca/~public/perl_fix_memmem_and_segfault.patch -O - 2> /dev/null | patch -p 1 )
    That fixes the first two bugs. The next fixes the Perl bug, and we're just going to download and store it in the OpenWrt build system and let OpenWrt sic it on Perl when it builds it.
    wget https://va1der.ca/~public/997-fix-Configure-gcc-parse.patch -O feeds/packages/lang/perl/patches/997-fix-Configure-gcc-parse.patch
  • Now we test to see if your OpenWrt build is actually affected by one of the above bugs. Some are (mine was), and if it is, we're going to have to build perl right now with the above patch to fix your existing system, because the host system's perl is used when OpenWrt builds openssl. Download and run the script:
    wget https://va1der.ca/~public/test_index.pl -O ~/test_index.pl
    chmod +x ~/test_index.pl
    ~/test_index.pl
    It will tell you if your perl is affected by the index() bug. If it is then...
  • We build perl with our bug patch above and copy libperl.so over top of of the buggy one:
    make -j 1 package/perl/compile
    cp $(find staging_dir -name libperl.so) /usr/lib/perl5/5.28/CORE/
    ~/test_index.pl
    The test should now show the index() bug is fixed
  • Now we can make the rest of the build. This is likely to take a while (not as long as LLVM though). I recommend using nohup and V=sc and redirecting to a log:
    nohup nice -10 make -j 1 V=sc world > world01.out &

Other Issues
A couple other minor issues I ran into in my experiments:

  • Lately some snapshot packages of perlbase-b included a bad /usr/lib/perl5/5.28/auto/B/B.so that referenced a wrong symbol. I found that this was only in the package delivered by opkg - if you build it yourself the library seems fine.
  • There is a bug in LLVM that makes libatomic need to get linked in to libLLVM.so, but then the LLVM build doesn't add it when it builds the rest of the package. You end up getting link failures. This only shows up on armv7l, I believe. I think you can get around it by usinv my LLVM above. I haven't gone as far on armv7l yet, though (just to the end of the toolchain).

On Errors
The above procedure Works For Me™. But this is very much an experimental process. By the time I was done, I was up to world13.out on two different platforms. I tried to document everything I did, but I may have missed packages. And there may be other subtle issues on other architectures. When htop tells you compilation has stopped, look at the log. Jump to the end, then (if you were doing multi-core builds) search backwards for "Error" (capital E) to find what actually went wrong. Most of the time it's a missing tool. Feel free to post your results here.

SUMMARY
There are a number of bugs that were exposed. Bugs in OpenWrt, Perl, LLVM, and other places. But I don't consider the presence of bugs the deciding factor on whether a system is, in general, self-hosting. OpenWrt needs one small utility (rev), and three bandaid libs to cover for GNU extensions to GLIBC. That makes OpenWrt about 99.9% self-hosting ready. This is a great achievement, especially for an embedded OS where self-hosting wasn't even the goal, and more especially considering the enormous amount of complex code and a build system that uses so many different build tools. The OpenWrt devs are to be greatly commended for an excellent embedded OS!

12 Likes

Such great timing! I recently acquired one of those Changwang Intel N5105 4x2.5Gbps boxes as an experimental/dev router. Yesterday I was thinking to myself, "I'm only using like 150M of the 250G drive and this thing has a ton of horsepower, I wonder if I could set up an OpenWrt build system on OpenWrt?" I am definitely going to try out your recipes here.

1 Like

And

That's pretty much mutually exclusive. Sure, it's more powerful than your average consumer MIPS router, but it's still a gutted x86 CPU. Of course, if you want to use it as your router and compiler farm... Have at it, I'd say.

@VA1DER What I'm missing is how long it took to compile everything? :angel:

1 Like

About 18 hours for the toolchain. After that, the actual image "only" takes about 5 hours.

I'm 52. I grew up with Apple ][s and cut my teeth using 6502 assembly to turn the PIO user port of CBMs into serial ports for my first BBS modem. I still remember as a project manager deploying a huge process control server for Canadian Autoparts Toyota. It had a (then) massive 768MB of RAM. I remember saying "three quarters of a gigabyte" over and over to myself. I remember when "DEC Alpha" was spoken with reverence. And now my router, in every aspect, is more capable than those machines.

So yes, even though I have a laptop that could probably walk on water, I still tend to look at routers differently. There is a ton of stuff they can do, and I don't tend to discount them as junk.

And there are still places in the world where cheap consumer grade hardware like a router is the best computing equipment available.

Besides, it's FUN.

3 Likes

I applaud your effort, don't get me wrong. Impressive how you tracked down all the issues and addressed them.

(Second quote you responded to was addressed to someone else BTW.)

Beside fun what's the real advantage here when you can get cloud servers for free (see OCI) that can compile the build in a few tens of minutes?

Doubt the free OCI will compile anything bigger than hello world in tens of minutes... ,)

not the x64 but the arm64 ones - you can assign 4 cores there :wink:

Ah, didn't know they were available, long time since I set up the ones I have.

1 Like

Yup, that would be me. I was gauging the N5105 on the spectrum of devices in my little lab here, a handful of ESP8266 boards, some 6502-based embedded systems, through various ARM and x86 routers, old and new Intel workstations and servers, up to the big 16-core AMD box on which I'm typing this reply...

The N5105 isn't actually doing any work, it's just a test bed for various firewall and DNS experiments, so if I can dedicate about 15 watts of power for a day to building OpenWrt from scratch, that'll broaden its purpose in life at little cost to me.

Like @VA1DER, this is a fun project for me, and if like him, I find some platform-specific bugs in the process, that's a win for everyone.

2 Likes

Do you mean these?

It suggests at the free tier that you get 1/8th of a CPU core. Am I reading that correctly?

I was referring to arm64 ones (Ampere A1) as you can build instances with up to 4 cores https://developer.oracle.com/free.html

A quad core ARM is respecable. About the same as my BPI-R3. Nothing wrong with that at all.

To answer your question, though, to me the advantage is being able to use infrastructure I own, not needing to pony up a credit card (which immediately precludes a lot of the people who would most need this sort of free service), and not being in some way the product.

I will also note they did not endear themselves to me right off the mark as I was creating an account to test it out. They asked for my email address and I went and put it in in the format I always use (name+uniquesuffix@address.ca) so I can have unique email addresses for every different service I have. I do this for safety and tracking. Safety in case my user data is breached, it's then not immediately evident what login/email address I use for every other account I have. And also to track to see what email addresses are being sold to spammers. When I put it in as name+oci@address.ca, I got the message "We trimmed the tag you added to your email. Please use your email address as shown to sign in to your account" and they stripped off "+oci" from my address. Luckily I have my email server set up to use underscores as the separator too, but this doesn't endear them to me. I don't like their "you better use the email address we tell you to use or we don't want you" message. I don't like being the product. And I don't like them putting in code to strip out the safety mechanism people use.

That all being said, I'm sure it's a very useful service for a lot of people and I hope they continue it. My purpose here in this thread isn't to advocate for any particular way of building OpenWrt, but simply to give people options. More options is never a bad thing.

2 Likes

Oh well I'd rather "be the product" if that means giving oracle my disposable email/payment card than wait 24h for one build.

1 Like

The CC I gave to Oracle 3 years ago expired a long time ago, they haven't asked for a new one...

Now on to topic - how about compiling inside an lxc - like ubuntu or debian - in theory you wouldn't need any quirks - right ?

I've never personally used LXCs. I haven't even looked at them since their early days, when it was pretty easy for code inside to break out of jail. I've personally had little use for that kind of containerization. I myself have Linux Mint running in seamless mode inside VirtualBox on my laptop for workstation use, and a full VM from a hosting company for my server needs. But in theory, yes, a Debian or Ubuntu LXC with the right dev tools should have little trouble building OpenWrt. Best thing to do is try!

Still cross compiling, isn't it?

And this way you know that your are compiling using it's own tools. And as show here, there are some bugs that have shown up that need to be addressed. Not that useful for building packages for distribution, but useful to verify the software.

About 24 h compile time, you don't need to babysit the compilation, you know. So what the machine does will not stop you from doing other stuff later. And you only need to build the development tool one time. Then you can use that to compile 5 h sets for the new software.

But yes, build in containers are also a way to build OpenWrt, but still not native compilattion using OpenWrt own tools. A really useful and needed project.

2 Likes

Recent snapshots have updated to using llvm 15.07. This now has a dependency on libzstd.

OpenWrt now supports this library. Unfortunately, OpenWrt is stil by default stripping all shared libs in such a way as you can't actually compile against them. So if you want to build OpenWrt with its internally-supplied llvm, you need to get libzstd:

Switch directories to your devel dir then:

wget https://github.com/facebook/zstd/releases/download/v1.5.2/zstd-1.5.2.tar.gz
tar -xvzf zstd-1.5.2.tar.gz
cd zstd-1.5.2/lib
make
make prefix=/usr install