`/usr/bin/env: Argument list too long`-error persisting -- OpenWRT 21.02.0-rc1 package installation creates a huge command line although far far from most packages selected

After two failing builds, I here re-run it to produce output for my report:

make -j1 V=s > make.stdout 2>make.stderr. The standart output and standard error files:

  • make.stdout: → Here,
  • make.stderr: → Here.
    (long files.)
  • Combined stdout and stderr in order from a subsequent run (make -j1 V=s > make.out 2>&1): → Here.

From there, the interesting part:

make.stderr:

[...]
make[2]: /usr/bin/env: Argument list too long
make[2]: *** [package/Makefile:69: package/install] Error 127
make[1]: *** [package/Makefile:109: /home/felics/tmp/owrt/staging_dir/target-i386_pentium-mmx_musl/stamp/.package_install] Error 2
[...]

The previous make-call was a very long one, a 210800(!) characters long command. From make.stdout (I cannot copy the whole command line here because it is too long for the forum software. The whole very long command line is → here):

[...]
IPKG_NO_SCRIPT=1 IPKG_INSTROOT=/home/felics/tmp/owrt/build_dir/target-i386_pentium-mmx_musl/root-x86 TMPDIR=/home/felics/tmp/owrt/build_dir/target-i386_pentium-mmx_musl/root-x86/tmp /home/felics/tmp/owrt/staging_dir/host/bin/opkg --offline-root /home/felics/tmp/owrt/build_dir/target-i386_pentium-mmx_musl/root-x86 --force-postinstall --add-dest root:/ --add-arch all:100 --add-arch i386_pentium-mmx:200 install \
	/home/felics/tmp/owrt/bin/targets/x86/geode/packages/base-files_1-r16046-59980f7aaf_i386_pentium-mmx.ipk [...] /home/felics/tmp/owrt/bin/packages/i386_pentium-mmx/telephony/sngrep_1.4.8-1_i386_pentium-mmx.ipk
[...]

I have already found this post, where there is a link to this patch. I extracted the patch from the email (→ here the plain patch file),

but it fails to apply (*click* to show):

patch -N --verbose -p1 --dry-run fix_argument_list_too_long.patch:

Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/include/package-ipkg.mk b/include/package-ipkg.mk
|index 5f7f2583a2..f2c31d1d3c 100644
|--- a/include/package-ipkg.mk
|+++ b/include/package-ipkg.mk
--------------------------
checking file include/package-ipkg.mk
Using Plan A...
Hunk #1 FAILED at 18.
1 out of 1 hunk FAILED
done

My build path, under ${HOME}/tmp/owrt, is not something extraordingly long.

Should I report it as a bug to bugs.openwrt.org? Seems that "Argument list too long"-type errors are around for some time already.

Can setting a insanely huge ulimit -s really be the only viable solution, as layed out in this post and this post and this post? And if that is the case, then I suggest to put into the build-documentation an information about that problem and that some quite high ulimit -s values might be needed.

Maybe the underlying build system might need a proper rework to not generate and need insanely long argument lists?

My configuration:

  • OpenWRT version: 21.02.0-rc1,
  • build target: x86, AMD geode (CONFIG_TARGET_x86_geode=y), with hand-tuned gcc-optimisations (CONFIG_TARGET_OPTIMIZATION="-O3 -pipe -mtune=geode -march=geode -mmmx -m3dnow -fomit-frame-pointer"), enabled some but far far away from most packages,
  • git describe: v21.02.0-rc1,
  • Latest commit 2021-04-19, commit hasg 2ce89a3,
    'git log -n1 --pretty=medium --abbrev-commit --date=iso' (*click* to show):
    commit 2ce89a3 (grafted, HEAD, tag: v21.02.0-rc1)
    Author: [...]
    Date:   2021-04-19 21:10:14 +0200
    
        OpenWrt v21.02.0-rc1: adjust config defaults
        
        Signed-off-by: [...]
    

    (Author details removed by myself for potential privacy reasons.)

  • 'feeds.conf' (*click* to show):
    src-git packages https://git.openwrt.org/feed/packages.git^4ceeb8fc90ed2c2e650ddddc855e7ed1df071c22
    src-git luci https://git.openwrt.org/project/luci.git^7d913b997601d533cca187cfc1b3057c3c98effc
    src-git routing https://git.openwrt.org/feed/routing.git^5b4d4c7fb6a97cac68c7d8b156fd0ab27bab4dcc
    src-git telephony https://git.openwrt.org/feed/telephony.git^178822957123b821407e1216e9e7314161512ac6
    
  • OpenWRT .config: See → here. (Long file.)
  • ulimit -n: 1024, ulimit -s: 8192,
    'ulimit -a' (*click* for details):
    real-time non-blocking time  (microseconds, -R) unlimited
    core file size              (blocks, -c) 0
    data seg size               (kbytes, -d) unlimited
    scheduling priority                 (-e) 0
    file size                   (blocks, -f) unlimited
    pending signals                     (-i) 30804
    max locked memory           (kbytes, -l) unlimited
    max memory size             (kbytes, -m) unlimited
    open files                          (-n) 1024
    pipe size                (512 bytes, -p) 8
    POSIX message queues         (bytes, -q) 819200
    real-time priority                  (-r) 98
    stack size                  (kbytes, -s) 8192
    cpu time                   (seconds, -t) unlimited
    max user processes                  (-u) 30804
    virtual memory              (kbytes, -v) unlimited
    file locks                          (-x) unlimited
    

Further things tried out:

    • Retried after issuing ulimit -s unlimited in the logged-in shell.
    • ulimit -Ss and ulimit -Hs show both unlimited after issuing that.
    • Rebuild.
    • Same problem.
    • Did set for my build user in /etc/security/limits.conf the soft and the hard limit for stack and nofile both to 65535.
    • Logout and login (on text console; re-login needed because the PAM-infrastructure does handle settings from that file),
    • checked with ulimit if those settings had effects. They had.
    • Rebuild.
    • Same problem.
    • Did set for my build user in /etc/security/limits.conf the soft and the hard limit for stack to unlimited and for nofile to 65535.
    • Reboot.
    • Checked with ulimit if those settings had effects. They had.
    • Rebuild.
    • Problem persists.

Any idea what may be wrong?

cat .config | grep PACKAGE_ | grep '\=y' | wc -l

2008.

(With the kernel modules I went to "enable as many as possible", rest specifically selected.)

1 Like

Now tested this also with stack unlimited and nofile 65536 and reboot.

Any idea what may be wrong?

what is wrong is pretty obvious...

you are exceeding the maximum length allowed for arguments...

you can;

  • patch your OS
  • patch the makefiles
  • select less packages (built-in)

package counts over 1000 are excessive... (arguably 500-700) even then that is likely due to copius perl/php/asterisk includes et. al...

no point bogging the whole OS down with 1000 kmods... if you can compile them as 'm' and install them later (if needed)...

built-in max...

# getconf ARG_MAX
2097152
2 Likes

Thanks for the answer.

I treat it as that my use case is kind of a "not supported use-case" and there is no will from the present developers to make that work, i.e. either someone else (e.g. me) has to provide code to make that work or it won't work. Am I correct?


I think that would be the "ideal" way to go to make the build system scale if it is used to build "fuller-blown" systems for machines with a lot of ressources; unfortunately I do not know how the OpenWRT build system works internally.

Well, it should support almost any device which could be thrown at it "out of the box".

That would be a bit hassle, because the image would be written to the CF-card which holds the root filesystem of the device and then it will be "stored away". Yes, would be possible in theory. Maybe that would be the workaround to go.

For me: getconf ARG_MAX is currently 4611686018427387903 with ulimit -Ss -Hs both unlimited, but still failing.


I will not mark your previous post as solution because for me this are only workarounds for a seldom-encountered use case but not a solution to make that use case work.

1 Like

how did you arrive at this conclusion?

including many SOC's that are often only hardwired?

writing the extra ipks to cfcard takes a few more seconds and saves you lots of runtime bloat...

silly not to....

I would like to patch the build process.

If I am right, I only would need to convert the one long command line that installs all packages in one round into a loop installing package by package individually.

I am not familiar with the OpenWRT build system's internals.

Am I right that the long install command line is generated in package/Makefile in the lines

        $(call opkg,$(TARGET_DIR)) install \
                $(call opkg_package_files,$(foreach pkg,$(shell cat $(PACKAGE_INSTALL_FILES) 2>/dev/null),$(pkg)$(call GetABISuffix,$(pkg))))

?

I was recursively-grepping fort parts of the long command line directly but that way I could not find that way any file generating it (grepping for a package ipk filename, grepping for --force-postinstall which appears in the long command line)


side-topic "how I arrived at my assumption about no-will of present developers" (*click* to see):

this:

reads for me like "I do something the build system is not designed for" and "to circumvent the issues I described, I have to do stuff (patch my OS, patch makefiles, select less packages)".


1 Like

The relevant lines are listed in the 'patch that fails' from your OP...

  1. i'm not a developer
  2. I am a single entity

my e-pinion is not representative of any stance from official developers...

creating a bug report on flyspray or emailing the devel mailing list for an official response/fix/opinion... would be your way forward for any official fix...

1 Like

So you mean the file include/package-ipkg.mk? -- As a feedback: Would be helpful if you make it more explicit what you mean.

It was "too high" for me to understand that file's working, and so I directly did work with packages/Makefile.

I replaced[1] the command

	$(call opkg,$(TARGET_DIR)) install \
		$(call opkg_package_files,$(foreach pkg,$(shell cat $(PACKAGE_INSTALL_FILES) 2>/dev/null),$(pkg)$(call GetABISuffix,$(pkg))))

with

	$(eval OPKG_INSTALL_PACKAGES := $(foreach pkg,$(shell cat $(PACKAGE_INSTALL_FILES) 2>/dev/null),$(pkg)$(call GetABISuffix,$(pkg))))
	$(foreach pkg,$(OPKG_INSTALL_PACKAGES),$(call opkg,$(TARGET_DIR)) install $(call opkg_package_files,$(pkg))$(newline))

(and defined the variable newline outside the recipe to contain just a newline).

This runs opkg install commands one by one for each package file to be installed.

This did not throw the "Argument list too long"-issue, but already failed to install the very first package "base-files" with
pkg_hash_fetch_best_installation_candidate: Packages for base-files found, but incompatible with the architectures configured:

IPKG_NO_SCRIPT=1 IPKG_INSTROOT=/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86 TMPDIR=/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86/tmp /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/staging_dir/host/bin/opkg --offline-root /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/build_dir/target-i386_pentium-mmx_musl/root-x86 --force-postinstall --add-dest root:/ --add-arch all:100 --add-arch i386_pentium-mmx:200 install /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/bin/targets/x86/geode/packages/base-files_1-r16046-59980f7aaf_i386_pentium-mmx.ipk
Unknown package 'base-files'.
Collected errors:
 * pkg_hash_fetch_best_installation_candidate: Packages for base-files found, but incompatible with the architectures configured
 * opkg_install_cmd: Cannot install package base-files.
make[2]: *** [package/Makefile:79: package/install] Error 255
make[2]: Leaving directory '/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1'
make[1]: *** [package/Makefile:137: /home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/staging_dir/target-i386_pentium-mmx_musl/stamp/.package_install] Error 2
make[1]: Leaving directory '/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1'
make: *** [/home/felics/download/Router-OS/OpenWRT/OpenWRT/21.02/source/openwrt-21.02.0-rc1/include/toplevel.mk:230: world] Error 2

(Later came to me the idea that calling opkg on every package individually without adding --force-depends would probably not work because it cannot resolve dependencies (and with addig --force-depends I am also not sure if this would be "clean" -- I am not sure if some essential checks are going on during the opkg-call in the Makefile which might lead to legitimate failures). So I think this cannot easily be solved that a lot number of packages to be pre-installed in the image at build time as long as opkg cannot read by itself the list of packages to operate on from a file.)

Can it be that the error above is because opkg needs some other packages as well to proceed -- even though the error message indicates just "wrong architecture" to me? If so, why the "incompatible with the architectures configured"? Or is that some other problem?

Regards!


[1]Thereby having learnt some stuff about make :slightly_smiling_face:.

1 Like

good points... your familiarity with this task equals or exceeds mine... good luck!

This may be helpful if it's the same issue

1 Like

thanks; I already mentioned it in my first post. This mention of yours made me reading it once again more thoroughly.

That issue is for a remove operation and the patch does not apply anymore anyway.

If this is analogous to the case here, there would be a need for a similar tweak for the install operation which uses xargs on installation conditionally.

Can anyone say something if this would work or might lead to dependency checking errors if opkg does not get all packages which might interdepend on each other within one call?:

Has anyome more insight on ↑this?

Regards!

1 Like

As a note:

I now filed a task to bugs.openwrt.org for a feature request to enhance opkg such that it can read the list of packages to operate on from a file or standard input: → Here.

1 Like