[NOTE] This is now its own wiki article at: https://openwrt.org/docs/guide-developer/toolchain/building_openwrt_on_openwrt
The wiki page above is now more up-to-date than the instructions below.
It's been noted before that OpenWrt isn't self hosting. Well, with only a very few additions (but with a bunch of bug workarounds) it can be! My BPI-R64 running snapshot just made its own build. This was a learning experience about the OpenWrt build system, and it exposed a LOT of bugs in things like LLVM, musl, perl, and in OpenWrt itself. But it was fun. This is still, of course, experimental. But I'll go into the steps you can use to replicate this:
First: Why Build OpenWrt on OpenWrt?
- Becoming self-hosting is an important milestone for any OS
- It makes building OpenWrt more accessible for those with less resources.
- Continuously building on a mono-culture (x86-based full blown Linux systems) lets a lot of bugs slide under the radar. Building on a different system can (and did) expose a lot of issues.
- For the brownie points of knowing (or bragging) that the OS on your device was BUILT on your device.
Hardware Requirements
- Device with 1GHz+ CPU, preferable dual core or better
- 512MiB RAM per core you intend to use**1
- 2GiB swap space on non-built-in storage (sd card if your playform supports it, or USB stick) **2
- 2GiB of space on whatever your rootfs is
- ~26GiB storage space for the build, **2
Note1: Building on 64-bit systems is more RAM-hungry than building on 32-bit systems. You can get away with a little less ram-per-core on 32-bit systems. You can also get away with a little less ram-per-core if you can use a pre-built LLVM.
Note2: This should be a fairly good quality storage device. Certain steps in the process will be somewhat heavy on the swap. A good quality storage device will handle the wear better. A larger storage device will spread the wear more.
Setting Up The Dev Environment
You need a whole lot of packages:
- Some good-to-have packages to make life easier:
opkg install joe joe-extras bash htop whereis less file findutils findutils-locate chattr lsattr wget-ssl xxd - The dev tools themselves:
opkg install pkg-config make gcc diffutils autoconf automake bzip2 check git git-http patch libtool-bin - Perl:
opkg install perl perlbase-findbin perlbase-pod perlbase-storable perlbase-feature perlbase-b**3 - A bunch of coreutils, since busybox versions aren't good enough in some cases, or aren't there:
opkg install coreutils-nohup coreutils-install coreutils-sort coreutils-ls coreutils-realpath coreutils-stat coreutils-nproc coreutils-od coreutils-mkdir coreutils-date coreutils-comm coreutils-printf coreutils-ln coreutils-cp coreutils-split coreutils-csplit coreutils-cksum coreutils-expr coreutils-tr coreutils-test coreutils-uniq - Other miscellaneous tools:
opkg install grep rsync tar python3 getopt procps-ng-ps gawk sed xz unzip gzip flock - A few libs:
opkg install libncurses-dev zlib-dev musl-fts - We need a softlink for libncurses:
ln -s libncursesw.a /usr/lib/libncurses.a
Note3: My system had a lot of perl installed already, so there may be more perl packages necessary that aren't listed here.
Before we can go further, we have a few Initial bugs to fix:
- OpenWrt Bug: OpenWrt doesn't put execute permissions on the scripts for the automake, autoconf, and libtool packages. This prevents a lot of build systems from working correctly. To fix:
chmod +x /usr/share/automake-1.16;chmod -x /usr/share/automake-1.16/COPYING /usr/share/automake-1.16/INSTALL
chmod +x /usr/share/autoconf/Autom4te/*
chmod +x /usr/share/libtool/build-aux/*
Now we can install the three dev libraries that OpenWrt doesn't support. One of them is included in OpenWrt as a non-dev library, but it's stripped in such a way as you can't compile against it. These three libs are all to support GNU LIBC extensions that musl doesn't have and are required mostly for elfutils. Switch to your ~/devel folder (or wherever you're going to use to build things) and do the following:
-
libargp:
git clone https://github.com/xhebox/libuargp.git
cd libuargp
make
make prefix=/usr install
ln -s libargp.so /usr/lib/libargp.so.0
cd .. -
libfts: We installed the non-dev version above already, so this will overwrite it with one that can be compiled against. We installed the non-dev version first so that it doesn't get installed later and overwrite your dev version with a lobotomized one:
git clone https://github.com/void-linux/musl-fts.git
cd musl-fts
./bootstrap.sh
./configure --prefix=/usr
make
make install
cd .. -
libobstack:
git clone https://github.com/void-linux/musl-obstack.git
cd musl-obstack
./bootstrap.sh
./configure --prefix=/usr
make
make install
cd ..
Now a few more bug fixes and issue workarounds:
- OpenWrt Workaround: Not a bug per se, though one might call it a bug by omission. The bzip2 binary also acts as bunzip2 and bzcat, but it can only do so if the proper softlinks are made when it's installed, which the OpenWrt package doesn't do. So:
ln -s bzip2 /usr/bin/bunzip2
ln -s bzip2 /usr/bin/bzcat - Elements of the build system don't like to be built as root. OpenWrt is essentially single-user, so to get around this:
export FORCE_UNSAFE_CONFIGURE=1
You might want to add the above to your ~/.profile - Speaking of .profile, I use bash as my shell. I suspect that the busybox Ash isn't good enough for some of the scripts in the build system, so I recommend installing bash and setting it as your /bin/sh
opkg install bash(already included above)
ln -sf bash /bin/sh
The rest of the bug workarounds, and the installation of the one tool we're still missing are all done in the OpenWrt source tree itself. So let's get it:
Get and Build OpenWrt
This is based on the wiki instructions, with a bunch of deviations to fix various bugs.
Make sure you're in your devel directory.
- First the preliminaries:
git clone https://git.openwrt.org/openwrt/openwrt.git
cd openwrt
git checkout master
git pull
./scripts/feeds update -a
./scripts/feeds install -a - Get the config file for your architecture and save it as .config. For me (BPI-R64) it is this:
wget https://downloads.openwrt.org/snapshots/targets/mediatek/mt7622/config.buildinfo -O .config - Configure your build:
make defconfig
make menuconfig
Things to change in the config: set it up to build for only your platform, read the next step below about LLVM and configure whether or not you're going to use a pre-built one, and then make any other tweaks you normally do. - EITHER get a pre-made LLVM (preferable)...
Compiling LLVM is the single most time and resouce-intensive step in the whole process. It can easily take two days by itself. I have so far built it on two platforms: aarch64 and armv7l. If you have one of those platforms, you can grab my pre-made LLVM and untar it into the openwrt build folder. If not, you will have to... - ...OR Build LLVM
BE CAREFUL HERE! Early in my experiments I burned out a Sandisk Extreme 64GB microSD card because I did a multi-core build and the memory requirements to build LLVM pushed the system so far into swapping that it did about 18 hours of non-stop writes. A single-core build will take a long time, but is safer. This is going to take a long time, so best use nohup in case you get disconnected - you don't want compilation to stop. And start getting into the habit of usingV=scand logging everything, because this whole process is still experimental:
nohup nice -10 make -j 1 V=sc tools/llvm-bpf/compile > ~/llvmcomp01.out &
Periodically check htop to make sure it is going well, and check the log when it stops. Hopefully there's no error, but if there is, welcome to the experiment. Debug, rinse and repeat and report it here for the rest of us. - Now we build the rest of the toolchain. Now that LLVM is out of the way, if you have multiple cores and enough RAM per core, then you should be ok to use multi-core compiling. Keep an eye on it, though and htop is your friend here. I am still going to put
-j 1in all the copy/paste commands for safety - but feel free to up the core count at your discretion.
nohup nice -10 make -j 1 V=sc toolchain/install > ~/toolchain01.out & - Now that the toolchain is done we can do a make download. A bug in qosify's makefile prevents make download from working before the toolchain is complete in some cases.
make download - We will now build the last tool we're missing:
rev, which is a little-known and littler used unix tool found inutil-linux. OpenWrt doesn't make a package out of it, even though it's built, so we're going to build util-linux then copy it out. This is quick, prolly no need to log it:
make -j 1 package/util-linux/compile
cp $(find . -wholename \*ipkg-install/usr/bin/rev) /usr/bin/rev - Now we're going to fix some bugs. One which causes a problem with perl on musl, one where recent compilers cause a seg fault, and one in Perl's Configure script itself. Fort this we need two patches. The first patch we will apply directly:
( cd feeds/packages/lang/perl && wget https://va1der.ca/~public/perl_fix_memmem_and_segfault.patch -O - 2> /dev/null | patch -p 1 )
That fixes the first two bugs. The next fixes the Perl bug, and we're just going to download and store it in the OpenWrt build system and let OpenWrt sic it on Perl when it builds it.
wget https://va1der.ca/~public/997-fix-Configure-gcc-parse.patch -O feeds/packages/lang/perl/patches/997-fix-Configure-gcc-parse.patch - Now we test to see if your OpenWrt build is actually affected by one of the above bugs. Some are (mine was), and if it is, we're going to have to build perl right now with the above patch to fix your existing system, because the host system's perl is used when OpenWrt builds openssl. Download and run the script:
wget https://va1der.ca/~public/test_index.perl -O ~/test_index.pl
chmod +x ~/test_index.pl
~/test_index.pl
It will tell you if your perl is affected by the index() bug. If it is then... - We build perl with our bug patch above and copy libperl.so over top of of the buggy one:
make -j 1 package/perl/compile
cp $(find staging_dir -name libperl.so) /usr/lib/perl5/5.28/CORE/
~/test_index.pl
The test should now show the index() bug is fixed - Now we can make the rest of the build. This is likely to take a while (not as long as LLVM though). I recommend using
nohupandV=scand redirecting to a log:
nohup nice -10 make -j 1 V=sc world > world01.out &
Other Issues
A couple other minor issues I ran into in my experiments:
- Lately some snapshot packages of perlbase-b included a bad /usr/lib/perl5/5.28/auto/B/B.so that referenced a wrong symbol. I found that this was only in the package delivered by opkg - if you build it yourself the library seems fine.
- There is a bug in LLVM that makes libatomic need to get linked in to libLLVM.so, but then the LLVM build doesn't add it when it builds the rest of the package. You end up getting link failures. This only shows up on armv7l, I believe. I think you can get around it by usinv my LLVM above. I haven't gone as far on armv7l yet, though (just to the end of the toolchain).
On Errors
The above procedure Works For Me™. But this is very much an experimental process. By the time I was done, I was up to world13.out on two different platforms. I tried to document everything I did, but I may have missed packages. And there may be other subtle issues on other architectures. When htop tells you compilation has stopped, look at the log. Jump to the end, then (if you were doing multi-core builds) search backwards for "Error" (capital E) to find what actually went wrong. Most of the time it's a missing tool. Feel free to post your results here.
SUMMARY
There are a number of bugs that were exposed. Bugs in OpenWrt, Perl, LLVM, and other places. But I don't consider the presence of bugs the deciding factor on whether a system is, in general, self-hosting. OpenWrt needs one small utility (rev), and three bandaid libs to cover for GNU extensions to GLIBC. That makes OpenWrt about 99.9% self-hosting ready. This is a great achievement, especially for an embedded OS where self-hosting wasn't even the goal, and more especially considering the enormous amount of complex code and a build system that uses so many different build tools. The OpenWrt devs are to be greatly commended for an excellent embedded OS!
