ImageBuilder make image segfaulting

For Developers
10

segfault is also @libc ( install exception ) and was introduced ( at least the recent major triggering of it ) around the time of the wireguard commits... ( ~3-Nov )

pkg_hash_fetch_best_installation_candidate: libc arch=aarch64_cortex-a72 arch_priority=200 version=1.1.24.
pkg_hash_fetch_best_installation_candidate: Candidate: libc 1.1.24.
pkg_hash_fetch_unsatisfied_dependencies: satisfying_pkg=0x7f9aff5f6a00
pkg_hash_fetch_unsatisfied_dependencies: satisfying_pkg=(nil)
pkg_hash_fetch_unsatisfied_dependencies: satisfying_pkg=(nil)
Makefile:157: recipe for target 'package_install' failed
make[2]: *** [package_install] Segmentation fault
Makefile:112: recipe for target '_call_image' failed
make[1]: *** [_call_image] Error 2
Makefile:210: recipe for target 'image' failed
make: *** [image] Error 2
1 Like
11

Not sure what happened in the last few days, first the OpenWrt downloads server went down, and after it went back up this happened (the ImageBuilder broke).

As of today it seems the kmods (the packages folder) are still not present in the ImageBuilder archives... guess I won't be able to make any new builds this way for the time being.

Will keep an eye on any updates regarding this.

1 Like
12

this is by design, it is no longer created... so that's 'normal' ( now )

2 Likes
13

I wonder if there is nobody looking into it? Still building useless output. It is not that worse like last time where the built images were bricking the whole device (because it is not finishing). But if a lot of things are going wrong/failing (on a server) I would assume that there is one getting an email and at least stopping this. Is there no snapshotting on the server itself available like to go 3 days back and provide this state online to fix the most recent state in order to bring it back later? Comming over a feeling that using buildbot is useless for snapshots and me. :smiley:

1 Like
14

So with the packages folder gone, how should I put additional packages (that aren't part of any repository) I wanted to include for ImageBuilder from now on?

On the other hand, guess I'll have to wait for the issues resolved before I could build updated images again...

Not sure what happened but I'm suspecting the last outage of the downloads server might be related, as the ImageBuilder was said to be fine until Nov 13, which was just before the OpenWrt downloads server went down (that would be Nov 14 in the place I live, when I noticed).

2 Likes
15

good question...

hopefully someone will document all the new changes once they are complete? like you ( everyone else? )... i'm just trying to work things out via usage... which does get tricky when things are broken...

so long as nobody's router is getting bricked... i'm ok with these issues... so long as the end product is one that is better ( or at least equivalent )... they'll probably be fixed in a day and we'll never see them discussed again?

1 Like
16

my use case for community builds is that I use the imagebuilder once... from this point on... i 're-use' a self-contained imagebuilder which for the 'new' format, seems to work well with...

repositories.conf

src imagebuilder file:FOLDERNAME #(copy of ./dl)

I can then rebuild exactly the same image with the same or less packages without having to rely on the upstream servers...

commstats
commstats1375×594 49.8 KB

peaks in the graph above are representative of buildsystem / imagebuilder issue dates over the last month... the community build seems to serve to offset official issues... and provide a fallback when peoples devices are not supported via official release...

1 Like
17

Just putting my 2 cents worth in here. Imagebuilder building was broken for me when I tried it for building from the raspberry pi 4 "bcm2711" snapshot yesterday. I opened a github issue, not sure if that was the right place. Just hope a fix is found soon :slight_smile:. The Imagebuilder process ended with the following error and exited:
Configuring kernel.
make[2]: *** [Makefile:161: package_install] Segmentation fault (core dumped)
https://github.com/openwrt/packages/issues/13936

1 Like
18

????
AFAIK, design has not changed.

There is a bug related to the package kernel symvers handling, like I diagnosed above.

There is now a proposed patch for the issue, but it has not yet been merged:

http://lists.openwrt.org/pipermail/openwrt-devel/2020-November/032181.html

2 Likes
19

@aparcar could probably explain it better as he made the change... his reasoning was to save space? or something to do with a more dynamic... 'consistent' package set via kmods over the internet...?

20

Not quite sure what you mean, as kmod checksums per build prevent any "dynamic" content.

(But at kmods are currently stored per build to the download server, it might be possible to drop kmods from the imagebuild file and download them on the fly during the imagebuilder usage. Might be that kind change that you refer to.)

But the current problem is related at least to the symvers file hadnling, which has been much more recent change.

That has been broken for a week, but now there is a proposed patch for it.

1 Like
21

I saw that the patch got merged yesterday evening. It still produces the same faillogs:

I don't know buildbot internals. If I look into:

https://buildbot.openwrt.org/master/images/builders/ipq806x%2Fgeneric/builds/722

it is referencing the latest changes including PKG_SYMVERS_DIR patch.

It might be that there has to be updated sth. other before (where buildbot is relying on; what is counting for all devices)?

22

It requires

E.g. for ipq806x devices we need arm_cortex-a15_neon-vfpv4 packages that are built with armvirt32 SDK, meaning that there needs to be

  • ipq806x image build (for ipq806x imagebuilder)
  • armvirt32 image build (for SDK to be used for packages)
  • arm_cortex-a15_neon-vfpv4 packages build using that fixed armvirt32 SDK

I think that all targets / SDKs will be built in phase1 in the next few hours (2127accd441b... or later)

But building all packages may take 1-2 days, depending on the luck with packages build round starting with or without the fixed SDK.

Sadly it takes time to fully recover from this kind of problem in core of the build system logic.

Ps. note that this "symvers" problem may have hidden some other problem caused by the commits on last Friday, but hopefully that fixed bug was the only bug...

1 Like
23

Looks like things got fixed:

lookign at the newest completed phase2 build (happens to be mipsel_24kc_24kf), the failurelogs look again "normal".

Just the usual failures...
https://downloads.openwrt.org/snapshots/faillogs/mipsel_24kc_24kf/

That makes me to think that other archs will recover, too. :wink:

1 Like
24

Ah, that looks good. I didn't know exactly how it is processed. Thx.

25

Based on faillogs it looks like package builds for almost all archs have now been completed, and the two last ones (powerpc_464fp and arm_cortex-a9) are being compiled right now.

Hopefully no other bugs surfaced for imagebuilder (and this symvers thing leading to missing packages was the bug causing the segment fault).

2 Likes
26

thankyou for representing the user base on the mailing list @hnyman your efforts were the missing link and are valued.

1 Like
27

I couldn't agree with that @anon50098793 has written more. For a user without deeper knowledge about the processes and code it is very difficult to make a diagnose if sth. is failing. Because most likely the error is in front of the screen. :smiley: Personally I'm scary to file bugs or write into mailing lists because of making possible false claims. So I'm thankfull if others with more expertise are doing this. :smiley:

1 Like
28

Looks like ImageBuilder is working normally now. I just built an image, flashed, and everything looks fine.

1 Like
29

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.