Building / Compiling OpenWrt ON OpenWrt - A Howto

For Developers
21

@VA1DER

perl test test_index.pl cannot be downloaded, attempt results in 403 unauthorized error.

Is there a way to create your own package for llvm? You have it available for very limited set, so I was wondering, since I'm attempting for x86-64; maybe I could share a pre-built llvm?

I also had a problem right from the start when trying to build llvm; libdl and librt were missing; on musl they are empty, ackording to one doc I found, but from openwrt, they are missing completely. This was not a problem for me, as staticly linkable archives were available from my previous built that was cross-built, on staging_dir/target/root-x86/lib so I copied them from there.. These were necessary for cmake - propably a small patch that disables linkage with these libraries would also had solved the trouble. But then again, who knows where building stops next time because they were missing, so I've got that covered, but this solution might work just for me.....

22

That's a problem with my server - it's trying to treat a .pl as an executable and run it, but doesn't have permissions. You'll need to right-click the link and select "Save link as..." or the equivalent choice in your browser.

In principal, absolutely. I hoped that people would share their llvm's from platforms I don't have. That said...

...I didn't post that one since it's already available in the normal download chain. I should have put links to them on the wiki page, though, and I will remedy that shortly.

Oh! I should have known to address that. I had already run into that issue long before I started building OpenWrt on OpenWrt. I didn't think to include it in the howto.

So yes, the functions normally in libdl and librt (and, incidentally, libpthread) are all incorporated directly into MUSL's libc. However some (a lot) of tools that use those functions try and link the libraries directly, which fails not because the functions aren't there, but simply because the linker can't find the file when you give -ldl to gcc. All you need to do to fix is this:

cat <<EOF > /usr/lib/libdl.a
!<arch>
EOF
cp -a /usr/lib/libdl.a /usr/lib/librt.a
cp -a /usr/lib/libdl.a /usr/lib/libpthread.a

Basically this creates an empty static library, just so the linker doesn't give a "file not found" for the lib when it tries to link against it.

For OpenWrt not to include those stubs is one of the dumbest policies I've ever seen and was already a fight I had with OpenWrt devs. I will also add the above instructions to the wiki page.

24

Hey, great tips - thanks, though, I already have librt and dl.. But I'll add libpthread archive with your guidance, though I could had just copied either of file librt or dl as they contain same content..

I'm already building llvm, but surely if it's available as pre-built, next guy likely will choose that option instead. You also could provide some guidance how to extract it, or actually- where to extract it, so one wouldn't mess their build trees, is it supposed to be extracted to base dir, or to staging/toolchain, or...? Ofcourse, one could first check what it contains to figure it out, but you know it as well as I do, that it won't happen about 90% of times :wink:

I built depencies you listed, though in my case propably it was not necessary for all as my build is unstripped. I'll be sure to follow your wiki in the future as well.

I took a different approach from nohup- I chose to use tmux instead. Build process keep running even if connection fails, and I can resume it, similarly one could use screen or any other alternative solution.

Yes, I've been missing those stubs also some time ago... Since files are so small, it sounds silly to me aswell, that such files should be missing. Or poorly documented if one is looking for them, I never though it before and just started looking for intel about it and accidentally found out that they are just empty archives.

Yes, right-click won't work- otherwise it would had worked with wget. It attempts to save a file with output of .pl script instead, without having sufficient permissions though..

You could rename the file, as your guide instructs it to be downloaded to destination, so why not add .pl to the file at that moment; though it ofcourse would be very nice to work it out with trickery..

25

maybe gzip it..? Some servers though tend to unpack if something has .gz and we are back to square one if that's the case..

26

Yes, that's what the guide does now. It should be working as given in the guide. :slight_smile:
The problem was my server was treating it as cgi and trying to execute it and send its output.

wget https://va1der.ca/~public/openwrt/patches/perl/test_index.perl -O ~/test_index.pl
chmod +x ~/test_index.pl
~/test_index.pl

The x86_64 has always been available as a pre-built on https://downloads.openwrt.org/ at the bottom of every platform, under supplementary files. I have put links to x86_64 in the wiki guide as well.

The openwrt base dir, yes.

I give an example of what to do in step 4. The example has to be modified depending on your platform, so you get the right llvm.

Yes, there are many solutions. Using tmux is great! I love its session handing. Harder to write the guide around it, though. I expect advanced users will use whichever methods they are comfortable with.

27

By extraction guide I meant the location where it's supposed to be extracted :wink:

Yes, tmux indeed is one great solution, but nohub ain't bad either. Topic of the wiki might be forgotten if it would be just listing alternative ideas of retaining build process in the background...

It would propably be wise to download pre-built... But now when llvm has already been compiling for few hours, I kinda don't want to stop it... That time would be wasted. But doesn't matter, if I won't- time remaining will be wasted. Well, I'll let it build, it's just a bit more homebrew that way..

28

From the wiki:

That is unclear? I will perhaps change to:
"If you are using a pre-built [llvm], then get it now and untar it into the root of your openwrt working folder"

29

Sorry, I propably missed that line.... I checked it again, and yes there it reads. That just gets lost in there easily..

llvm-bpf is the the Berkeley Packet Filter bytecode compiler for Linux. It is a tool that is needed in order to build OpenWrt, but also can be built along with the rest of the OpenWrt toolchain below. Compiling llvm is easily the single most time and resource intensive step in building OpenWrt. If there is a pre-made one for your platform of the correct version for the version of OpenWrt you are building, then it is recommended to use it. Check the resources below. If you are using a pre-build one, then get it now and untar it into your openwrt working folder, so it will be there when you configure OpenWrt.

How I would had written it:

llvm-bpf is the the Berkeley Packet Filter bytecode compiler for Linux. It is a tool that is needed in order to build OpenWrt, but also can be built along with the rest of the OpenWrt toolchain below. As it is definetly most heavy single task in build process, I must strongly recommend to download a pre-built llvm if available for your platform and extract it to root of your openwrt working directory.

My version is not better in any way. Already first version of instruction has necessary details, but there's just something in the text that made me un-knowingly skip part of it and unnoticing the important part. This happens with my attempt as well, you do not need to change it any way, but if you have small perfectionist in you, you could make that single instruction better by either or both: reducing it's length and some how pointing out the important part better out so even morons like myself won't skip it :smiley: But not necessary, problem was in the one who was reading it.......

EDIT: oh, I took a short nap, and llvm has built during that.. Continuing then..
I previously used a container running alpine which I used for building openwrt inside my truenas server, I had problems with perl as well- most likely due to same reasons and had to do some trickery to get it working with that too, I had way messier hack for that..

@VA1DER
btw. in my experience, most time consuming build on openwrt buildroot, is definetly node. In time consumed it easily beats llvm + full toolchain building + kernel all combined.

30

Maybe there could be a libc-stub package, if not accepted in actual tree; as a pre-built package, it would be pretty easy, as it's all-arch compatible if it just has those 3 stubs...

One more thing... As openwrt doesn't come with users and even though nearly every distro discourages usage of root account, most of the time we still do it... So.. There could be somewhere a mention of environment value pair of FORCE_UNSAFE_CONFIGURE=1 which allows building as root... Information though is available on other pages with instructions on building openwrt, but I think, in this case, it's even more approached way than with other ways to build openwrt..

If someone finds quilt handy for patching where necessary, my private repository also holds it- but bash is required for it to work.

31

My perl was reported by test that it won't need fix. Nice.
Toolchain installation though has a problem:

make[4]: Entering directory '/usr/src/openwrt/build_dir/toolchain-x86_64_gcc-12.2.0_musl/gcc-12.2.0-initial/gcc'
g++ -O2 -I/usr/src/openwrt/staging_dir/host/include -pipe -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -DHAVE_CONFIG_H -DGENERATOR_FILE -static-libstdc++ -static-libgcc -o build/genpreds \
build/genpreds.o build/rtl.o build/read-rtl.o build/ggc-none.o build/vec.o build/min-insn-modes.o build/gensupport.o build/print-rtl.o build/hash-table.o build/sort.o build/read-md.o build/errors.o ../build-x86_64-pc-linux-musl/libiberty/libiberty.a
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_personality.o): in function `base_of_encoded_value(unsigned char, _Unwind_Context*)':
eh_personality.cc:(.text._ZL21base_of_encoded_valuehP15_Unwind_Context+0x2e): undefined reference to `_Unwind_GetTextRelBase'
/usr/bin/ld: eh_personality.cc:(.text._ZL21base_of_encoded_valuehP15_Unwind_Context+0x37): undefined reference to `_Unwind_GetDataRelBase'
/usr/bin/ld: eh_personality.cc:(.text._ZL21base_of_encoded_valuehP15_Unwind_Context+0x40): undefined reference to `_Unwind_GetRegionStart'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_personality.o): in function `parse_lsda_header(_Unwind_Context*, unsigned char const*, lsda_header_info*)':
eh_personality.cc:(.text._ZL17parse_lsda_headerP15_Unwind_ContextPKhP16lsda_header_info+0x1a): undefined reference to `_Unwind_GetRegionStart'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_personality.o): in function `__gxx_personality_v0':
eh_personality.cc:(.text.__gxx_personality_v0+0x8c): undefined reference to `_Unwind_GetLanguageSpecificData'
/usr/bin/ld: eh_personality.cc:(.text.__gxx_personality_v0+0xd9): undefined reference to `_Unwind_GetIPInfo'
/usr/bin/ld: eh_personality.cc:(.text.__gxx_personality_v0+0x39f): undefined reference to `_Unwind_SetGR'
/usr/bin/ld: eh_personality.cc:(.text.__gxx_personality_v0+0x3b2): undefined reference to `_Unwind_SetGR'
/usr/bin/ld: eh_personality.cc:(.text.__gxx_personality_v0+0x3c0): undefined reference to `_Unwind_SetIP'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_personality.o): in function `__cxa_call_unexpected':
eh_personality.cc:(.text.__cxa_call_unexpected+0xf2): undefined reference to `_Unwind_Resume'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_throw.o): in function `__cxa_throw':
eh_throw.cc:(.text.__cxa_throw+0x38): undefined reference to `_Unwind_RaiseException'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_throw.o): in function `__cxa_rethrow':
eh_throw.cc:(.text.__cxa_rethrow+0x3d): undefined reference to `_Unwind_Resume_or_Rethrow'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(vterminate.o): in function `__gnu_cxx::__verbose_terminate_handler()':
vterminate.cc:(.text._ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x13b): undefined reference to `_Unwind_Resume'
/usr/bin/ld: /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.a(eh_catch.o): in function `__cxa_end_catch':
eh_catch.cc:(.text.__cxa_end_catch+0x54): undefined reference to `_Unwind_DeleteException'
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:3011: build/genpreds] Error 1
make[4]: Leaving directory '/usr/src/openwrt/build_dir/toolchain-x86_64_gcc-12.2.0_musl/gcc-12.2.0-initial/gcc'
make[3]: *** [Makefile:4597: all-gcc] Error 2
make[3]: Leaving directory '/usr/src/openwrt/build_dir/toolchain-x86_64_gcc-12.2.0_musl/gcc-12.2.0-initial'
make[2]: *** [Makefile:29: /usr/src/openwrt/build_dir/toolchain-x86_64_gcc-12.2.0_musl/gcc-12.2.0-initial/.built] Error 2
make[2]: Leaving directory '/usr/src/openwrt/toolchain/gcc/initial'
time: toolchain/gcc/initial/compile#0.37#0.02#0.00
ERROR: toolchain/gcc/initial failed to build.
make[1]: *** [toolchain/Makefile:97: toolchain/gcc/initial/compile] Error 1
make[1]: Leaving directory '/usr/src/openwrt'
make: *** [/usr/src/openwrt/include/toplevel.mk:231: toolchain/install] Error 2

Know anything about this?

EDIT:
I fixed it... Although it happened again shortly after this..

# cd /usr/src/openwrt/build_dir/toolchain-x86_64_gcc-12.2.0_musl/gcc-12.2.0-initial/gcc
# g++   -O2 -I/usr/src/openwrt/staging_dir/host/include  -pipe   -DIN_GCC  -DCROSS_DIRECTORY_STRUCTURE   -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings   -DHAVE_CONFIG_H  -DGENERATOR_FILE -static-libstdc++ -static-libgcc  -o build/genpreds \
    build/genpreds.o build/rtl.o build/read-rtl.o build/ggc-none.o build/vec.o build/min-insn-modes.o build/gensupport.o build/print-rtl.o build/hash-table.o build/sort.o build/read-md.o build/errors.o ../build-x86_64-pc-linux-musl/libiberty/libiberty.a -lstdc++

By linking against libstdc++ the problem got away, is this helpful for others? Hard to say, gcc comes with libstdc++ - but is it linked against gcc provided, or openwrt provided, as both can be true since I disabled stripping when building my "build os" - I need to check gcc's Makefile, maybe I can add -lstdc++ to ldflags..

EDIT2:
I did not find a suitable location for adding -lstdc++ to ldflags, so I went to build_dir/toolchain-x86_64_gcc-12.2.0_musl/gcc-12.2.0-initial/gcc and edited Makefile.
On row 1039, I found:
INTERNAL_CFLAGS = -DIN_GCC $(PICFLAGS) -DCROSS_DIRECTORY_STRUCTURE

I added -lstdc++ there and build continues. Not ideal solution.

32

When you use g++ on the command line (as opposed to gcc), then -lstdc++ is supposed to automatically be linked against. I don't know why it's not in your case. Can you gzip the whole build log and send it to me?

33

Unfortunately I was using tmux, so no log exists except backlog of terminal window.

Despite that I chose unstripped version.... Still gcc shared libraries are stripped. Automaticly when using g++ it links against /usr/lib/gcc/x86_64-openwrt-linux-musl/12.2.0/libstdc++.so(.6.0.30) - and adding -lstdc++ it links to /usr/lib/libstdc++.so - which contained those missing references to unwind functions. Gcc's own libstdc++ was around 1.7mb when libstdc++ on /usr/lib was 2.4mb - I checked with nm, to find out that gcc's libunwind.so is stripped :frowning:

I checked my original buildroot on alpine, in gcc build_dir (target), ipkg-install contains libstdc++.so.0.6.30 that is huge 12mb.. I replaced shared libraries with their unstripped versions, but this wasn't helpful- still without -lstdc++ I get same error.

EDIT: eventually manually adding -lstdc++ initial-gcc was succesfully compiled. Very strange. On alpine I had similar problems when I tried to integrate rust-lang, propably adding -lstdc++ there too, would had solved the issue. But this is very strange. This should not had happened, especially if you didn't have same issue as I did...

EDIT2: same problem re-occurred with gcc-final, same solution seems to continue building..

EDIT3: Had to add it to 3 Makefiles, and link one file manually and to complete the build, it had to be removed from one of Makefiles

EDIT4: All requirements to begin building are full-filled. Tomorrow morning there's either error.. or success :slight_smile:

34

Perl test script results:

root@media:~# ./test_index.pl
index=0 - your system is NOT affected - your perl is fine!

But yet, attempt to build openssl fails:

...
Applying ./patches/150-openssl.cnf-add-engines-conf.patch using plaintext:
patching file apps/openssl.cnf
touch /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/.prepared_cb94ce14a0ab458034d1241b1ee541c9_6664517399ebbbc92a37c5bb081b5c53
rm -f /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/.configured_*
rm -f /usr/src/openwrt/staging_dir/target-x86_64_musl/stamp/.openssl_installed
(cd /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8; ./Configure linux-x86_64-openwrt --prefix=/usr --libdir=lib --openssldir=/etc/ssl --cross-compile-prefix="x86_64-openwrt-linux-musl-" -I/usr/src/openwrt/staging_dir/toolchain-x86_64_gcc-12.2.0_musl/usr/include -I/usr/src/openwrt/staging_dir/toolchain-x86_64_gcc-12.2.0_musl/include/fortify -I/usr/src/openwrt/staging_dir/toolchain-x86_64_gcc-12.2.0_musl/include -L/usr/src/openwrt/staging_dir/toolchain-x86_64_gcc-12.2.0_musl/usr/lib -L/usr/src/openwrt/staging_dir/toolchain-x86_64_gcc-12.2.0_musl/lib -znow -zrelro -Wl,--gc-sections shared no-tests -DOPENSSL_PREFER_CHACHA_OVER_GCM no-async disable-dynamic-engine enable-devcryptoeng no-padlockeng zlib-dynamic && { [ -f /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/.configured_51a2158dbd47627a03c798ac4e99cc8a_ade4165991d90c58d806dc229f39678c ] || make clean; } )
Can't locate IPC/Cmd.pm in @INC (you may need to install the IPC::Cmd module) (@INC contains: /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/util/perl /usr/lib/perl5/5.28 /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/external/perl/Text-Template-1.56/lib) at /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/util/perl/OpenSSL/config.pm line 19.
BEGIN failed--compilation aborted at /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/util/perl/OpenSSL/config.pm line 19.
Compilation failed in require at ./Configure line 23.
BEGIN failed--compilation aborted at ./Configure line 23.
make[2]: *** [Makefile:402: /usr/src/openwrt/build_dir/target-x86_64_musl/openssl-3.0.8/.configured_51a2158dbd47627a03c798ac4e99cc8a_ade4165991d90c58d806dc229f39678c] Error 2
make[2]: Leaving directory '/usr/src/openwrt/package/libs/openssl'
time: package/libs/openssl/compile#0.23#0.24#0.00
ERROR: package/libs/openssl failed to build.
make[1]: *** [package/Makefile:116: package/libs/openssl/compile] Error 1
make[1]: Leaving directory '/usr/src/openwrt'
make: *** [/usr/src/openwrt/include/toplevel.mk:231: package/openssl/compile] Error 2

But that's just because necessary perlbase packages weren't present.
Similar errors came up while installing depencies, but one by one, I got them all installed and decided to document list of missing depencies to here:
perlbase-ipc, perlbase-module, perlbase-extutils

These also installed following as depencies to those 3 packages:
perlbase-version, perlbase-dirhandle, perlbase-autosplit, perlbase-ostype

35

If host is powerful enough, building openwrt on openwrt is really fast, unless i5 beats my xeon (not going to do comparisons, just noticed that build speed for kernel could be considered as quick)

It's a bit of work to get it going, but I think it's also rewarding at the end even though there aren't many real advantages. But I always did like gentoo's approach and it seems that I can play gentoo on openwrt :wink:

Some time ago I already managed to build openwrt on openwrt, but that was heavily modified version with lots of stuff compiled natively, not just few libraries; and eventually that reached a break point at some point, so this seems well documented with explanations/reasons why to do this and that, list of caveats why things fail, how to work-around it, etc- whilst still being honest and letting to know, that while someone might benefit from this, it's not propably worth time for everyone who wants to bake their own...

Very good work; I'll keep you posted on my discoveries; no plans to give up any time soon- there is advantage on this. If you are developing something of your own, and are running with stripped system; this will save a lot of time when rebuilding your work as depending on what it is and how many files it has, but often it's single executable; so you can execute it directly on your device from build_dir- another solution is native building... But sometimes that's a lot of work until you have set all up if there's depencies to shared libraries and need to replace stripped versions with unstripped- so it can also be very useful instead of just being a toy project.

On a cross-building host... I would update makefile/files ackordingly, build a package, go to build dir/ipkg-x86_64 ... scp binary to device's /tmp, ssh there and try and fail and repeat this in worst scase, about thousand times. With this kind of setup, there are few steps less. Less time for tinkering, more time for coding.

@VA1DER thumbs up :slight_smile:

1 Like
36

Thank-you. I always knew there might be more perl dependencies than I listed (because my system includes a lot of perl already). I will add these to the list.

39

Various build problems on openwrt buildroot

Similar problems that I had with alpine as host os for building, are usually related to building on musl host. Also some strange issues have existed, on Arch I had trouble with multiple kernel modules to build, and on alpine I had one kernel module that did not build: kmod-ipt-coova

There are some strange problems on openwrt buildroot that happen on some build hosts, and others then do not have same issues on a same os at all, likely these issues are related on your build config and your host system config in symbiosis. This does and does not have anything to do with host using either musl or glibc, or whether it was ArchLinux, Alpine Linux - or even openwrt.

One of being that building of git required some extra love to succeed.. It is not related per-se to building openwrt-on-openwrt; it's something else, I had problem to build git-http-fetch on ArchLinux at one time too and it used glibc, not musl.
My fixes for git might not be perfect, but if someone needs it- it's available here: https://github.com/oskarirauta/local-overrides/tree/main/net/git

Problems and solutions during build

libc-stubs: package that I previously provided, needed one more stub for libresolv.a, I updated package.
libc-stubs package source is available at https://github.com/oskarirauta/packages2/tree/main/devel/libc-stubs
Edit: I also added libcrypt.a, libutil.a, libxnet.a and libm.a, that should cover all that were missing.
After these, both refpolicy and policycoreutils compile without issues.

buildroot-env: I provided another package that selects necessary depends to build openwrt on openwrt and adds required header files as well. Package uses also 2 other provided packages, libuargp and musl-obstack from same repository; needed missing packages. But for these to work properly, package is available only when CONFIG_NO_STRIP is enabled. One could also make a post-install script to it to build 4 packages in /tmp if CONFIG_NO_STRIP is not enabled, but I did not do that.
Package is available at here.

fail2ban:
fail2ban failed with 2to3, first because of it was missing from built hostpkgs, I then attempted to build it manually but failed for some reason, every time. So I removed fail2ban from my packages for now, and focused on building other stuff- since I don't include it as default to my system anyway; eventually 2to3 did build, but still, command 2to3 was missing; instead I had 2to3-3.10 where 3.10 stands for used python version. So I made a symlink from 2to3-3.10 to 2to3 just to find out that I failed again due to some missing python stuff. Then I installed 2to3 on building host, it does not include 2to3 command, but that didn't matter, it included everything required to run 2to3 symlink, I restarted fail2ban build and it succeeded.

libxcrypt, needs more perl modules on host: perlbase-open, perlbase-if
backuppc needs perl module: perlbase-digest

mosh requires perl module perlbase-diagnostics - but it isn't enough, as it wants also file
/usr/lib/perl5/5.28/pod/perldiag.pod which isn't included in package; but if you have built perl,
you can find it in your build directory and copy from there.. After this mosh build succeeds.

privoxy required user privoxy(uid 8118) and group privoxy(gid 8118) adding to system to be able to build.

boinc failed to build, configure thought that libcurl is unusable, fix it by adding this to CONFIGURE_ARGS: _libcurl_try_link=no and libcurl_cv_lib_curl_usable=yes

libv4l failed to build, it wanted to link with libargp, but it wasn't available as dynamic library, so I statically linked it. Check this Makefile if necessary. libv4l must be compiled before mjpg-streamer.

pdns once again is one of packages that have a issue with curl-config; configure thinks curl is un-usable. Adding libcurl_cv_lib_curl_usable=yes to configure args, fixes this. Seems to be pretty common problem with many applications linking to libcurl. https://github.com/oskarirauta/local-overrides/blob/main/net/pdns/Makefile

shorewall needs more perl modules: perlbase-autouse, perlbase-sys and perlbase-sort

yubiko-pam needed additional configure argument ac_cv_libykclient=yes

Some packages needed updating and minor fixes, I made PRs on those packages and they were merged to mainstream, list of updated packages is here: nfs-kernel-server, libtirc, open-vm-tools.

won't-fix - or what I didn't/couldn't fix.

  • cjdns: failed, first to missing cflag and then to error of missing .a on it's own files. I didn't fix it.
  • ruby: won't build, configure had issue with cflags validity, I made a patch that overrided it, then build failed. Hopefully someone else will take a look at this.
  • avahi-compat + shairplay: avahi-compat is needed by shairplay. avahi-compat does not want to build. Not related to musl/openwrt-on-openwrt - issue exists on ArchLinux and Alpine as well.
  • frr failed to build. Though it propably has nothing to do with building on openwrt. It does not link vty.o to libfrr.so which leaves undefined reference, build system is that complicated that I skipped it for the moment. Should propably check if there's update available that would fix it..
  • lxc refused to build with complex errors
  • vallumd, look next chapter
  • schroot: linking error, possibly not related to building on openwrt-on-openwrt, same issue was with alpine linux, possibly musl related?
  • sing-box failed to build, errors in build process

I recommend all perl modules that I get listed as missing, will be added as requirements to avoid surprises mid build.

ATTENTION - WARNING

While building, I constantly arrived point where my host lost it's network connection and became unresponsive, I first thought that this is because of cpu overheating, or filesystem error or something, but finally I found out the cause.
This occurred every time when build system attempted build vallumd - for unknown reason, as it begun perfectly normally, but soon init restarted and I got choice to press F for safemode etc and only thing system responded was using keyboard and pressing ctrl+alt+del.

I tested this several times, even by trying to compile vallumd as a single package and yet it happened again every time without process having enough time to print to screen what the issue could be.

Finally I gave up on vallumd, after all, I haven't ever even tested it or really know what it is and what it can be archieved with it- I disabled it in my config and used ./scripts/feeds uninstall vallumd to avoid it's build in future.

If your system hangs during build, check if you have tried to build vallumd.

Some details on my build configuration

My build is near full feature build, meaning that most optional features are enabled, if SSL is used, always choose openssl; and with full language support(iconv). My build is also unstripped, which grows binary size substantially while retaining possibility to link to shared libraries when building natively.
My local-overrides repository mostly contains fixes regarding iconv issues + other stuff, like changes enabling musl-host compilation. It's private while publicly available, so I am resposible over it's being up-to-date for only myself.

Build progress

My attempt to build openwrt-on-openwrt has finished with about 99% of packages in default repositories. Including succesful build of all kernel modules.

Build statistics

root@media:/usr/src/openwrt# pkgcount
3.3G total
6114 packages / 6153 files

root@media:/usr/src# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               977.3M    471.4M    438.5M  52% /
devtmpfs                512.0K         0    512.0K   0% /dev
tmpfs                     7.8G    712.0K      7.8G   0% /tmp
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda2                64.9M     25.9M     39.0M  40% /boot
/dev/sda9               198.6G    120.4G     68.1G  64% /usr/src

Build tree's total size is around 120gb, I have about 500mb of stuff like npm cache, etc that is on /usr/src but not in /usr/src/openwrt. I have completely excluded repositories telephony and video.

Openwrt can very well be built on openwrt. To build all this, including setting up environment; it took 3 days. If someone else attempts it with similar hardware effieciency, and uses tips I gave, skips vallumd attempts after failing the first time and accepts my patches, propably it will be complete way faster.

I also built rust toolchain and few packages for it succesfully; repository is here - main rust Makefile has some commented out parts which need to be un-commented in case your build host and target are not the same.

Note
Since I use unstripped in my config to be able to use shared libraries when compiling natively, my build grew fast, so, I decided to strip some packages, so on most of my packages, binary stripping is used if no stripping is selected, for executables. It is ignored, if stripping in general is enabled. Stripping is most effective on huge go binaries.

Final thoughts
I cannot say, do I recommend this or not- but I can verify, that it isn't the foolest choice either.
Personally I have no regrets on walking through this, and very likely, I am going to use my current build tree for some time, after fixing some issues, I am quite happy on openwrt that was built on openwrt.

40

@Borromini

That's pretty much mutually exclusive. Sure, it's more powerful than your average consumer MIPS router, but it's still a gutted x86 CPU. Of course, if you want to use it as your router and compiler farm... Have at it, I'd say.

I have 7 routers in my house, and only my outdoor router is mips; and getting that was a huge mistake. My all indoor routers run with x86-64 and have atleast 6 rj-45 ports. My main router has 8 ports, wi-fi, 4g/lte card inside, 32gb ram and i7. My development/living room router has only 16gb ram and i5, but rebooting it doesn't affect my internet connection.

For sure, they were a bit pricy.. And need a bit more from power outlet, even though they all run on 12V. All of them have efi, and boot openwrt with refind and have 3 versions of openwrt for testing purposes (recovery, upgrade and sys-root) that I periodically update. I am very pleased on building openwrt on openwrt, previously I built in on a alpine container running in Truenas server, but I have to admit that it's better as a openwrt-on-openwrt as my Xeon on truenas server runs a ton of other things at same time.
This kind of setup also gives benefit of rapid development when you can use your built code on same device immediately.

That rapid development is a real game changer when developing strictly for openwrt. Huge advantage that has already saved so many hours from my life.

What I'm missing is how long it took to compile everything? :angel:

I just re-built on my i5; all packages that build (read below, there's a handful that doesn't build), with natively built llvm completed in less than 2 days. Most long build time was with node.
All packages also include my custom packages and a bit experimental rust-lang support.

But ofcourse, it all depends on how much you have to give for the process. For example, disk space required for all to build shows currently 134gb- though I used my previous dl folder, so it's likely that there are more than one sources for kernels and atleast 2 sources for node along with other small sources as well. On my routers, there also was a risk of hardware malfunction, i5 reported constantly temperature of ~85 for cpu out of max 100 - and it's passively cooled without any fans, so you have to be confident on devices like these to really cool efficiently enough.

41

@Borromini

I started from fresh. Almost (kept downloaded files and my .config).
I have re-built now an updated build daily, or even more often if lot of updates were pushed; until recently updated libtool update broke it somehow. Currently building it with hopes that this works out with lib tool 2.4.7 - so far, so good.. But there's still 80% of packages to compile until that reveals.

This host was already built with openwrt-on-openwrt, and this time I had no problems while building gcc. Meaning that libstdc++ problem- magically it was gone. Also 2to3 was installed as it should. So, 2nd generation build worked out even better than first.

On my repository, I have constantly tried to improve my buildroot-env package, which mostly is a meta package installing necessities to build openwrt-on-openwrt, adding every time something that I see build process wants, latest being coreutils-fmt that was looked for while building openzwave; I also added package for rev, that installs it from util-linux, keeping util-linux as build depency. There's also some of that other missing software/libraries, though it's useful only if building without stripping. But if there's anything useful to you, please take any part.

Pre-built llvm from wifi (x86_64/amd64) is broken

  • /lib64/ld-linux-x86-64.so.2 is missing. Fix: I linked it to libc.so
  • llvm-bpf/lib/libLLVM-15.so is missing symbol mallinfo. Fix: not fixable
1 Like
42

Interesting, I will try to build OpenJDK by building it with OpenWRT. took openjdk from alpine as boot_jdk

43

If you are building on a relatively new host, using musl-1.2.4 or newer, llvm-bpf doesn't build because it attempts to lseek64 and other functions ending with 64; which were renamed since 1.2.4 came out to exclude 64; so lseek64 becomes lseek.

There is a workaround for this issue for almost everything, building with define _LARGEFILE64_SOURCE, so you need to add
HOST_CFLAGS+= -D_LARGEFILE64_SOURCE
to your tools/llvm-bpf/Makefile under
include $(INCLUDE_DIR)/cmake.mk
so that in the end it will look like this:

...
include $(INCLUDE_DIR)/host-build.mk
include $(INCLUDE_DIR)/cmake.mk

HOST_CFLAGS+= -D_LARGEFILE64_SOURCE

LLVM_BPF_PREFIX = llvm-bpf-$(PKG_VERSION).$(HOST_OS)-$(HOST_ARCH)
...

this will instruct compiler to accept functions ending with 64 and translating them properly. I have also added this information to wiki. It might be worth a while to update wiki, since I've given quite a lot here to take one building openwrt on openwrt a bit further. Ever since I built my first image of openwrt on openwrt, I've been doing it eventually even getting changes even into mainstream to get it more compatible with this build method and I still like this approach very much.

I also attempted to build a newer llvm; version 17.0.6 - which built nicely without patches and tinkering Makefile, unfortunately it just didn't work, there's just something fundamentally different with that newer version and as I am not an expert with llvm, I had to revert to older version of llvm and add that host CFLAG to be able to build it.