When building all firmwares and packages for a target using multiple cores

For Developers
1

hello!
i know that when i use multiple cores ( make -j8) it fails more often than i just use serial 1 build.
has this changed? is it more stable by now with eg v18 to use multiple cores for building?

or to be on the safe side i should keep using one core for build?

how the buildbots working , do they use multiple cores?
I want to be 100% safe.

just it is so long (15-17 hours) to run my builds.

for example when i use 8 cores for MongoDB, it is 3-4 hours instead of 1 day and it is perfectly building. is it possible with OpenWrt by now?

2

I usually compile with multiple cores, and usually it works ok. If there are problems, I revert to single core to see more clearly the problem. the debugging is more difficult from multicore log file...

1 Like
3

do you know, that with using multiple core building, the multiple cores are built by one by one every package/firmware. so the packages are serial and only the one package building is using multiple cores, so it will not building package 1 and package 2 at once is it how it works? because that is what i am afraid of....

4

I think that usually one core builds one package at a time, and packages get distributed to cores. Three cores are building three packages simutaneously.

Upstream gcc documentation should give more info.

5 
make -j12 clean download world

with

CONFIG_DEVEL=y
CONFIG_BUILD_LOG=y
CONFIG_CCACHE=y

has been pretty robust for me. The logs capture errors without having to resort to re-running everything single-threaded in many cases (though you have to guess which package went wrong). Use of ccache is optional, but I've never had problems that I've associated with it and it speeds my development builds significantly.

6

what does the clean download world mean, does it mean before i build all via IGNORE_ERRORS=1 make V=s -j$CORES, i should pre-download the repos first?

eg:

make download V=s -j$CORES
 IGNORE_ERRORS=1 make V=s -j$CORES

???

CONFIG_DEVEL=y
CONFIG_BUILD_LOG=y
CONFIG_CCACHE=y

what does CONFIG_DEVEL and CONFIG_CCACHE means? where is it in the make menuconfig?

of course i use CONFIG_BUILD_LOG=y

7

Sequences things, does clean, then download, then world (what you would build with just make).

Often it is a "late" download that causes multi-threaded builds to fail.

/ in menuconfig will search

man ccache on your build machine -- compiler cache.

I never run with IGNORE_ERRORS=1 -- an error is an error. For that matter, so are a lot of "warnings" if you're writing code.

Builds take me 15-20 minutes from scratch, 5-7 minutes on rebuild. I have no idea why it would take you hours.

8

but does it know if a package has a preqrequiremet and wait until that is built?

9

for you when you build all packages is 20 minutes? that looks so good.
i usually build everything, so of course there are a few errors, that is why ignore, i usually have about 10 packages are errored. so it still works.
the buildbots have no wrong packages at all?

10

I only build packages I need for the ROM. “Loose” packages are of very little value to me, build a new ROM, flash it, done.

If you’re trying to build a repo, you’re much better off with the two-pass approach of the buildbots.

1 Like
11

packages define dependencies...
They need to be correct.

Ps. Building a single image with -j3 after "make clean" takes some 25 minutes for me with 2 cores on virtualbox. But I only build for one device at a time, and do not build all packages

12

so the buildbot when it builds all packages is it still running with multiple jobs?
i want to use multiple jobs for both the firmwares and the all packages build...

13

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.