I am experimenting with the OpenWrt build system. I am building Git tag v19.07.2 at the moment (the latest stable release).
In this step:
./scripts/feeds install -a
I get many warnings like these:
WARNING: Makefile 'package/utils/busybox/Makefile' has a dependency on 'libpam', which does not exist
WARNING: Makefile 'package/utils/busybox/Makefile' has a build dependency on 'libpam', which does not exist
WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libgnutls', which does not exist
and later on warnings like this:
Installing package 'dbus' from packages
Installing package 'libical' from packages
WARNING: No feed for package 'jansson' found
Installing package 'apcupsd' from packages
Installing package 'libgd' from packages
If I build the Git master (the development head) instead of a particular tag, the warnings are for different packages, but otherwise similar.
I would have expected the build system to fail if a package dependency is not satisfied.
But I wonder how the package system handles versioning. I guess that different versions of OpenWrt and/or packages will have different dependencies.
Say I am building an older OpenWrt version, and a package has been dropped in the mean time. How will that work? I had a quick look and could not make out clear OpenWrt version branches in the package repositories.
There are separate branches for master, 19.07 and 18.06 for packages and LuCI, not only for OpenWrt itself.
You can't mix packages from master to 19.07 builds, and vice versa.
If you switch branches in the same buildroot, you need to clear the semi-hidden package data in tmp directory. (several .dot starting files there). delete the whole tmp from the buildroot, in addition to the normal make clean.
The decision to ignore (warn) missing packages on feed installation is a deliberate one. Busybox for example only depends on libpam when a specific feature is enabled (CONFIG_BUSYBOX_CONFIG_PAM).
Failing hard here would mean OpenWrt would become unbuildable unless the optional libpam feed package is installed despite it not actually being needed.
Fixing it would require adding full conditional dependency resolution support to scripts/feeds and it would introduce a chicken-egg problem since scripts/feeds usually is called before any .config exists, so it cannot determine if a potential conditional dependency is actually required or not since the conditionals are not resolveable due to a lack of .config symbols.
Another problem is that one source package may declare multiple binary packages with different sets of dependencies. The nftables source package for example provides both nftables-json and nftables-nojson of which only the former depends on jansson.
There is a kind of soft rule for core packages to not depend on feed packages and to avoid inter-feed package dependencies but this is not 100% enforceable due to various technical and organizational reasons.
Furthermore, jansson was recently moved from the packages feed to core in master, so switching between master and openwrt-19.07 branches within the buildroot without also switching the feed branches and installing / removing the jansson feed package every time will lead to the warning you observe.
I think I know what mistake I made. I have scripted the OpenWrt build from the beginning, and in order to avoid permanent downloading, I cloned all feed repositories locally. I saw a way to specify a different remote server for them, but that looked more complicated, because I would have to set up a local server.
I then generated a feeds.conf with local file paths for the Git repositories, but I forgot that they need a particular commit ID. Thanks for all the infos.
I wanted to clarify a couple of aspects. Hopefully this will help other people in the future.
I get no warnings on master or 19.07, which I build.
This is not quite correct. You do get such warnings for the x86/64 architectures. It is important to note this, because, otherwise, you may think that you have made a mistake in your own build scripts.
OpenWrt's own build bot does generate such warnings:
./scripts/feeds install -a
in dir /builder/x86_64/build (timeout 1200 secs)
watching logfiles {}
argv: ['./scripts/feeds', 'install', '-a']
environment:
CC=/builder/x86_64/ccache_cc.sh
CCACHE=/usr/bin/ccache
CCC=/usr/bin/gcc-6
CCXX=/usr/bin/g++-6
CXX=/builder/x86_64/ccache_cxx.sh
HOME=/builder
HOSTNAME=0be3eebd6f6b
LANG=en_US.utf8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/builder/x86_64/build
SHLVL=1
using PTY: False
WARNING: Makefile 'package/utils/busybox/Makefile' has a dependency on 'libpam', which does not exist
WARNING: Makefile 'package/utils/busybox/Makefile' has a build dependency on 'libpam', which does not exist
WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libgnutls', which does not exist
WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libopenldap', which does not exist
WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libidn2', which does not exist
WARNING: Makefile 'package/network/utils/curl/Makefile' has a dependency on 'libssh2', which does not exist
WARNING: Makefile 'package/boot/kexec-tools/Makefile' has a dependency on 'liblzma', which does not exist
WARNING: Makefile 'package/network/services/lldpd/Makefile' has a dependency on 'libnetsnmp', which does not exist
Installing package 'acl' from packages
Installing all packages from feed packages
...
I have been fighting with Git lately. Like I mentioned, in order to avoid downloading the repositories over the Internet every time, I cloned all feed repositories locally to a kind of master clones.
That has other advantages too. For example, the package servers are down at the moment. OpenWrt's own build bot has noticed it:
Collecting target info: merging...
Collecting target info: done
Updating feed 'luci' from 'https://git.openwrt.org/project/luci.git' ...
Cloning into './feeds/luci'...
fatal: unable to access 'https://git.openwrt.org/project/luci.git/': Failed to connect to git.openwrt.org port 443: Connection timed out
failed.
But because I have local clones, I am not impacted by this service outage.
My build script generates a feeds.conf with local file paths for the Git repositories. The trouble is, I did not know that a clone of a clone is not actually perfect clone. All branches are lost (!).
That Git idiosyncrasy is explained here:
I wonder whether there is an official way to cache the Git repositories. I would have thought it is not in the OpenWrt project's interest that everybody writing automated build scrits is hitting the official Git servers every time.