How to reduce the compile time

For Developers
1

Normally , the total compile time of the openwrt is about 3-4 hours, it cost a lot time , how to reduce the compile time

2
  • get a faster build host
  • dabble with ccache
  • get a faster build host
  • don't clean between builds
  • get a faster build host
  • reduce the build config to what you really need
  • get a faster build host
6 Likes
3

thanks
i mean to reduce the compile time just limit to software bot hardware, the envionment is fixed

4

I achieved the first bullet by "allowing" my teenage child to purchase and assemble a gaming system (ensuring it had 32 gig ram and that virtualization was enabled in the bios), then set about installing ssh (so I could log on and build projects with them knowing it), and finally, as it is running win10, installing wsl2 (there are some tricks here to keep a screen session running on wsl even if I log out from ssh).

Assuming one does not have a teenage child interested enough in gaming to unwittingly build you your own build box or a teenage child is beyond one's current budget, the second bullet point is likely a better option.

EDIT teenage children also typically take 12-14 years from conception to get to the point of desiring a gaming pc.

Note sure exactly about ccache with openwrt. For building android under wsl/ubuntu I do something like:

sudo apt install ccache
export USE_CCACHE=1
export CCACHE_EXEC=/usr/bin/ccache
ccache -M 50G # android is a beast, openwrt is not.  Reduce this to 20G?
# optionally use
ccache -o compression=true # and reduce the 50G to even less

You'll have to do some hunting around to see exactly how to get ccache to function with openwrt.

At this very moment, I have an android build going with ccache. Without it, the build would take 2-3 hours. With it, it should be done in under an hour.

Good luck.

One other thing, building the toolchain takes a lot of time. When possible avoid make distclean and just do make clean. That said, it's hard to know when you need to rebuild the toolchain, not doing so can lead to chasing silly errors, and if your building different kernel versions, I think you should do it every time you change kernels versions.

4 Likes
5

Generally speaking, if you have a multi-core system, you can specify the "jobs" in the make command (-j) to increase the number of simultaneous jobs/threads that will be spawned during the compile process.

That said, I don't how well this works with OpenWrt compiles as it depends on how many of the compile-time routines can be parallelized. I've used arguments like -j4, but I've never actually timed the execution so I don't know how effective the argument is.

1 Like
6

-j n definitely helps. I follow the recommendation to use 1+ the number of threads/cores for n. Why 1+? The theory is that there is always one job waiting to go when another job is finished.

i.e. my wsl setup has six "cores" available to it. I use -j 7.

1 Like
7

Get a AMD threadripper? So many cores.

8

my CPU is 2 cores / 4 threads, should I use -j3 or -j5? thanks.
I always use -j4 before.

9

See Advanced Configuration Option For Developers in nconfig (menuconfig, . . .)

As per @anon98444528 let the buildroot figure it out

date && make -j $(($(nproc)+1)) && date

1 Like
10

if your using older hardware, you might try turning off hyper threading in the bios. In other words, make your 2 core 4 thread system a 2 core 2 thread system (in this case use -j 3 or better yet, @RuralRoots suggestion).

Surprisingly, your build might actually complete sooner this way.

Even before specter/meldown (and the subsequent fixes that can result in multi-threading actually slowing down your system under high loads), HPC users, particularly for fluid dynamics problems, found that using hyperthreading resulted in slower computation times. This is problem dependent and not always true so do your own evaluation.

A few other suggestions, customize your .config to only build what you need/want. The openwrt build bot config seeds are for building several common targets. (Already suggested above by slh)

Once you've done a "full" build, you can rebuild only specific packages as needed. The instructions here (actually this is better) are a good place to start.

Lastly, take a look at imagebuilder. Images with all the packages you want and none that you don't, assembled in minutes even on slow hardware. Unless I absolutely have to compile something from source, I use imagebuilder.

1 Like
11

Well... -j16 FTW I'd say. Except for a few fringe cases where the parallelism breaks stuff occasionally (race conditions in some packages etc) it works just fine.

The buildroot is designed with this parallelism in mind, you can set misbehaving packages to use only a single thread etc.

12

Thanks @anon98444528 @RuralRoots .

yes, I'm using an old PC + ubuntu 20.04, and hyper-threading enabled.

Then, -j 5 should be better than -j 3, am I right?

Screenshot from 2022-07-15 21-09-05
Screenshot from 2022-07-15 21-09-051032×869 73.7 KB

13

With 4 threads yes use -j 5.

But time it. Then try turning off multi threading, using 2 threads and -j 3. You will have to be careful to start from the same point between these two cases (make distclean is probably best before each build - painful I know). I recall doing this once on an intel i5-4300U CPU and found the build completed in ~15-20 minutes sooner with fewer threads. YMMV. Have fun.

14

Btw, I always enable ccache, add all wanted packages to (=y), and no more (=m).

on -j4, the compiling time >1 hour for fresh, and ~15min rebuild. Next time, I'll try -j5.

Thanks.

15

oops， my environment just can available for -j1, if i use -j2 , the compile will fail

16

While I've done amount of cores +1 since I can remember, at some point someone explained (might be on the forum here somewhere even) that there is little point to keeping an extra thread in the wait queue. Exactly the amount of threads you have works better, apparently.

1 Like
17

I’m not so sure about that. At least for my environment.

I do recall a conversation about this when I first started rolling my own. All things considered, $(($(nproc)+1)) provides an optimal build time for me over $(($(nproc)))

18

That makes me wonder what the openwrt build system (for the downloadable images) uses. I'm interested enough to try something different but not motivated enough to set up a test to get data (i.e. timing builds of the same openwrt build 10 times with cores+1 vs 10 times just with the number of cores).

It won't surprise me to find out there is a fair amount of noise in the build times and that it depends on what is being compiled.

EDIT graph ipq806x build bot times since Jan this year - about 1-2 hours and looks like its trending higher.

19

openwrt building images wiki
make download will pre-download all source code for all dependencies, this will enable multi core compilation to succeed, without it is is very likely to fail. make -j N will speed up compilation by using up to N cores or hardware threads to speed up compilation, make -j9 fully uses 8 cores or hardware threads.

HTH