"opkg list" with 10MB free memory gives "out of memory"

I got a MikroTik RouterBOARD RB941-2nD (16M flash, 32M RAM).

  • With 19.07, I got "out of memory" trying to flash it (via sysupgrade -n).
  • With 18.06, I managed to flash, but then got "out of memory" running an "opkg list" (and LuCI shows "no packages available" even after successful "opkg update").
  • Upgraded to 19.07, "opkg list" still does "out of memory", but now I can install packages with LuCI! I thought LuCI used "opkg list" as a backend?..

However, "opkg list" works fine on a TP-Link TL-WR940N running the same version (18.06.8). And I never had an "out of memory" with any other 32M devices before (at least not with such basic things). And LuCI seemed to otherwise run well even with the whole image in the memory! (I didn't try to install packages though.)

Is 32M RAM really not enough to even run basic stuff like "opkg list"? I would understand being unable to pxeboot a whole image into RAM, but that actually worked! Unlike a simple "opkg list" or "sysupgrade"!

Are there any tips on building a low-memory image? There was a FAQ page, but it was removed. If it works on other devices with the same amount of RAM, then I guess it should be at least possible to make it work with this one?

It is not enough :frowning:
opkg package list fetching may cause memory consumption to spike.

4 Likes

It's one of the instances where https://openwrt.org/supported_devices/432_warning becomes glaringly obvious, but by far not the only one.

4 Likes

https://openwrt.org/faq/build_image_for_devices_with_only_16_32mb_ram

Logging in by SSH loads another instance of dropbear into RAM, which takes about 1.3 MB. (There will be two running, one handling your session and another waiting for another login)

If you're not using LuCI, stop uhttpd. odhcpd runs by default but it is unnecessary in an IPv4 only network.

How much RAM is used varies by device as different hardware drivers will be loaded. That's why one device works and the other goes out of memory, assuming you have the same packages/services enabled on both.

32MB is really close to the precipice for many devices, and many others simply have no more RAM left to run at all and were dropped (their image is not built anymore). There is a thread about reporting such devices in the forum

pxeboot is a thing that is done by bootloader while OpenWrt is NOT running, the bootloader is using only a few kb of RAM so it can store whatever it wants in the rest of the RAM.

sysupgrade is done by the OpenWrt system, and this means that the RAM needed by the system won't be available to store a new image for flashing.

2 Likes

Thank you for your answers!

I was aware of the 4M flash limitation, but I have initially overlooked the 32M RAM one. I did not expect something like opkg to need almost 10M. Indeed, that seems to be the case.

On 19.07, while "opkg list" does not work, listing the packages via LuCI works for some reason (even when I'm logged in via SSH at the same time). Also, "opkg list" works if you have not done "opkg update", because there is not many packages to list. Also, the first "opkg update" works fine, but consequent runs do not - apparently because too much memory is already occupied by previously downloaded lists (even though they don't seem to take much space).

The package list is about 1.5 MB stored in the RAM disk, which of course takes away available RAM.

If you have lots of flash but limited RAM you can edit /etc/opkg.conf to store the list in a directory on flash, which normally isn't a good idea.

Where are you reading the 10M of free space from? It sounds quite a lot of free RAM for a 32MB RAM device running a modern OpenWrt system.

Can you post the result of writing "free" in SSH terminal? (should print a table to show how the RAM is used at the moment)

        total     used     free   shared  buff/cache   available
Mem:    26932    12652     5300       92        8980       12124
Swap:       0        0        0

I know "free memory" is a complex subject, but this does mean "12124k available if we flush 8980k caches", right?..

kind of, some caches can't be removed (they are created again if you try to flush them).

You can write


echo 3 > /proc/sys/vm/drop_caches ; free

to clear caches immediately and immediately see how much "free space" is reported. Some "buffer/cache" will still be in use.

But still, even now it's showing 5300k of "free space" and that's quite a bit, I don't think it should run out of memory.

Can you do a opkg update and then check with free again, to see how much RAM space is occupied by the package lists?

What is the error message exactly?

root@MIKROTIK:~# echo 3 > /proc/sys/vm/drop_caches ; free
       total     used     free   shared  buff/cache   available
Mem:   26932    13572    10556      312        2804       10644
Swap:      0        0        0

That's what I'm talking about. I don't see any problems with my RAM. :slight_smile:

This is an almost-configured router with everything I need installed and a pptpd running with one connection. So I was really surprised that this is not enough for someone, and certainly not for opkg.

After opkg update:

root@MIKROTIK:~# echo 3 > /proc/sys/vm/drop_caches ; free
       total     used     free   shared  buff/cache   available
Mem:   26932    13468     9808     1164        3656        9892
Swap:      0        0        0
root@MIKROTIK:~# opkg list
Killed
root@MIKROTIK:~#

Apparently, lists take less than a megabyte.

hmm, it seems there is already a bug report about opkg taking "too much RAM" even when there is ample free RAM https://bugs.openwrt.org/index.php?do=details&task_id=2734&string=opkg&type[0]=&sev[0]=&pri[0]=&due[0]=&reported[0]=&cat[0]=&status[0]=open&percent[0]=&opened=&dev=&closed=&duedatefrom=&duedateto=&changedfrom=&changedto=&openedfrom=&openedto=&closedfrom=&closedto=

it seems the guy that usually works on opkg (jo-philipp) does not want/have time to look into it though.

I took the liberty to edit the title of this thread to better suit the actual issue, this is not a "classic" 32MB device running out or RAM.

You could try disabling the "packages" repository (and keep other repos) as that's by far the biggest one.

1 Like

While this is correct[0], it isn't the real problem either. For opkg to do anything, it does need to decompress the package lists in RAM and generate dependency graphs for all available packages. This is a complex operation[1] and the size of these dependency graphs rise with the number or packages and their interdependencies.

Contrary to popular belief, packages managers are one of the most complex tools to run[2] - and 32 MB RAM are completely insufficient for this (with kernel and basic userland needing more than a decade ago - and with lots of new packages having become available for installing), even 64 MB RAM devices currently can't provide enough free RAM for opkg to function properly. By this time https://openwrt.org/supported_devices/432_warning has become reality and one should act accordingly, by scheduling a replacement for these devices.

--
[0] I'd still strongly recommend against storing the package lists on flash, es doing so would wear out the flash prematurely - and most of the effected devices don't have a lot of free flash to begin with.
[1] graph theory is one of the most complex topics in discrete mathmatics and theoretical information technology, and usually not a lot of fun to work on.
[2] and opkg can only reduce these RAM/ CPU requirements slightly by being a bad one, without support for features you'd find in apt/ yum/ dnf/ <insert package manager of your desktop distribution here> - proper support for in-place upgrades and all that entails (sonames tracking, symbol versioning, much more complex package dependencies, ….

1 Like

interesting. What I find weird of this explanation is why it has to make this for all packages and not just for the packages I want to install/remove.

I can literally unzip manually the package lists and use grep to find any dependencies in very little RAM. Might be (a lot) slower, but it works, which is sure better than failing like that.

Thank you!

@slh The funny thing is, opkg install, which actually needs to build graphs, works just fine. What does not work is:

  • opkg update, which only downloads the lists, and not even look inside. Interestingly enough, IT EVEN WORKS if you are downloading the lists for the first time (or just remove the existing folder with downloaded lists). And even if you create a 8M dummy file to take all memory! It actually works just fine with only 2M free RAM! But not the second time, when downloaded lists already exist.

  • opkg list, which simply lists ALL names from those lists. It does not even need to check any conditions or dependencies.

As mentioned in the bugreport you linked, that is already happening where possible (at a considerable expense of performance and probably also the quality of the the dependency resolver), but it still leaves rather large partial graphs to work with. Keep in mind that these days, dependencies aren't really binary, but there are also conflicts, provides and alternative dependencies (in full grown package managers also weaker dependencies, like recommends/ suggests - and build time dependencies like Build-Depends and Build-Conflicts, or multi-stage Build-Depends for bootstrapping), in order to generate a valid dependency resolution the package manager needs to be able to keep track of these. This is not too unlike from the travelling salesman problem, given the available resources it's very well possible that a dependency resolver can't find a solution, even if there is one.

Each new package has a cost, each package that can replace and stand-in a busybox applet significantly raises the complexity the dependency resolver has to cope with - each package that introduces alternative dependencies (e.g. ustream | wget | curl) raises the bar.

I'm still not convinced. I mean, I'm sure what you say is correct for the average Linux distro that can fully self-host, but OpenWrt is not exactly like that.

I've seen quite a few OpenWrt package manifest files, which is what opkg is working with (I've written the script that reads them and creates the package table and index in the wiki) and I'm pretty sure most OpenWrt packages have only a few dependencies, very rarely a conflict, there is no such thing as "build depends" or "build conflicts" (as OpenWrt can't compile itself, these things are specified in the package makefile, in the build system source).

Then OK there are a few packages that provide the same command/alternative, but afaik there is not more than 5-6 alternatives for each thing.

The overwhelming majority of packages are just dumb things with one or two libraries as dependencies.

See this file for example, that is a "full-fat" package manifest containing all possible information about packages in base repo of x86_64 target https://downloads.openwrt.org/releases/19.07.2/packages/x86_64/base/Packages.manifest
The manifest in this "full fat" form is used by the wiki script, but the actual package manifests in the package lists use the same format, they just dropped the less-important fields (source, maintainer, license and whatnot).

But it does need to consider the short/ long description of the the packages, an entry that doesn't matter for the dependency resolution itself and can largely be skipped there.

Hm, that's a good point. I think I found a way it can use less RAM. At least in concept, because I don't think I can actually fix the code. I can write a shell script that replaces this opkg list functionality though (as that's more or less what the script I wrote for the wiki also does when parsing package manifests).

Each repo gives a package manifest for Opkg (which is a text file), and this means it has to parse them all. But it can't just read and write instantly to console because it wants to print an alphabetical list of packages.
This means it has to read all files, sort stuff and then print it. It seems this requires too much RAM.

But if it does not care anymore about making a single alphabetical order, and is just reading and printing immediately it's not using much RAM anymore. This means it can only print an alphabetical list of packages for each repository (the manifest is already sorted alphabetically), but I think it's still OK for what most people use opkg list for.

@dark-penguin since this is an issue only when doing opkg list, it's not the same as the bug I linked. You might want to open a bug report where you explain the issue and maybe how I proposed to "fix" it, or just make the bug report, link it here and I'll post the proposal there myself.