I got a MikroTik RouterBOARD RB941-2nD (16M flash, 32M RAM).
With 19.07, I got "out of memory" trying to flash it (via sysupgrade -n).
With 18.06, I managed to flash, but then got "out of memory" running an "opkg list" (and LuCI shows "no packages available" even after successful "opkg update").
Upgraded to 19.07, "opkg list" still does "out of memory", but now I can install packages with LuCI! I thought LuCI used "opkg list" as a backend?..
However, "opkg list" works fine on a TP-Link TL-WR940N running the same version (18.06.8). And I never had an "out of memory" with any other 32M devices before (at least not with such basic things). And LuCI seemed to otherwise run well even with the whole image in the memory! (I didn't try to install packages though.)
Is 32M RAM really not enough to even run basic stuff like "opkg list"? I would understand being unable to pxeboot a whole image into RAM, but that actually worked! Unlike a simple "opkg list" or "sysupgrade"!
Are there any tips on building a low-memory image? There was a FAQ page, but it was removed. If it works on other devices with the same amount of RAM, then I guess it should be at least possible to make it work with this one?
Logging in by SSH loads another instance of dropbear into RAM, which takes about 1.3 MB. (There will be two running, one handling your session and another waiting for another login)
If you're not using LuCI, stop uhttpd. odhcpd runs by default but it is unnecessary in an IPv4 only network.
How much RAM is used varies by device as different hardware drivers will be loaded. That's why one device works and the other goes out of memory, assuming you have the same packages/services enabled on both.
32MB is really close to the precipice for many devices, and many others simply have no more RAM left to run at all and were dropped (their image is not built anymore). There is a thread about reporting such devices in the forum
pxeboot is a thing that is done by bootloader while OpenWrt is NOT running, the bootloader is using only a few kb of RAM so it can store whatever it wants in the rest of the RAM.
sysupgrade is done by the OpenWrt system, and this means that the RAM needed by the system won't be available to store a new image for flashing.
I was aware of the 4M flash limitation, but I have initially overlooked the 32M RAM one. I did not expect something like opkg to need almost 10M. Indeed, that seems to be the case.
On 19.07, while "opkg list" does not work, listing the packages via LuCI works for some reason (even when I'm logged in via SSH at the same time). Also, "opkg list" works if you have not done "opkg update", because there is not many packages to list. Also, the first "opkg update" works fine, but consequent runs do not - apparently because too much memory is already occupied by previously downloaded lists (even though they don't seem to take much space).
root@MIKROTIK:~# echo 3 > /proc/sys/vm/drop_caches ; free
total used free shared buff/cache available
Mem: 26932 13572 10556 312 2804 10644
Swap: 0 0 0
That's what I'm talking about. I don't see any problems with my RAM.
This is an almost-configured router with everything I need installed and a pptpd running with one connection. So I was really surprised that this is not enough for someone, and certainly not for opkg.
After opkg update:
root@MIKROTIK:~# echo 3 > /proc/sys/vm/drop_caches ; free
total used free shared buff/cache available
Mem: 26932 13468 9808 1164 3656 9892
Swap: 0 0 0
root@MIKROTIK:~# opkg list
Killed
root@MIKROTIK:~#
While this is correct[0], it isn't the real problem either. For opkg to do anything, it does need to decompress the package lists in RAM and generate dependency graphs for all available packages. This is a complex operation[1] and the size of these dependency graphs rise with the number or packages and their interdependencies.
Contrary to popular belief, packages managers are one of the most complex tools to run[2] - and 32 MB RAM are completely insufficient for this (with kernel and basic userland needing more than a decade ago - and with lots of new packages having become available for installing), even 64 MB RAM devices currently can't provide enough free RAM for opkg to function properly. By this time https://openwrt.org/supported_devices/432_warning has become reality and one should act accordingly, by scheduling a replacement for these devices.
--
[0] I'd still strongly recommend against storing the package lists on flash, es doing so would wear out the flash prematurely - and most of the effected devices don't have a lot of free flash to begin with.
[1] graph theory is one of the most complex topics in discrete mathmatics and theoretical information technology, and usually not a lot of fun to work on.
[2] and opkg can only reduce these RAM/ CPU requirements slightly by being a bad one, without support for features you'd find in apt/ yum/ dnf/ <insert package manager of your desktop distribution here> - proper support for in-place upgrades and all that entails (sonames tracking, symbol versioning, much more complex package dependencies, ….
interesting. What I find weird of this explanation is why it has to make this for all packages and not just for the packages I want to install/remove.
I can literally unzip manually the package lists and use grep to find any dependencies in very little RAM. Might be (a lot) slower, but it works, which is sure better than failing like that.
@slh The funny thing is, opkg install, which actually needs to build graphs, works just fine. What does not work is:
opkg update, which only downloads the lists, and not even look inside. Interestingly enough, IT EVEN WORKS if you are downloading the lists for the first time (or just remove the existing folder with downloaded lists). And even if you create a 8M dummy file to take all memory! It actually works just fine with only 2M free RAM! But not the second time, when downloaded lists already exist.
opkg list, which simply lists ALL names from those lists. It does not even need to check any conditions or dependencies.
As mentioned in the bugreport you linked, that is already happening where possible (at a considerable expense of performance and probably also the quality of the the dependency resolver), but it still leaves rather large partial graphs to work with. Keep in mind that these days, dependencies aren't really binary, but there are also conflicts, provides and alternative dependencies (in full grown package managers also weaker dependencies, like recommends/ suggests - and build time dependencies like Build-Depends and Build-Conflicts, or multi-stage Build-Depends for bootstrapping), in order to generate a valid dependency resolution the package manager needs to be able to keep track of these. This is not too unlike from the travelling salesman problem, given the available resources it's very well possible that a dependency resolver can't find a solution, even if there is one.
Each new package has a cost, each package that can replace and stand-in a busybox applet significantly raises the complexity the dependency resolver has to cope with - each package that introduces alternative dependencies (e.g. ustream | wget | curl) raises the bar.
I'm still not convinced. I mean, I'm sure what you say is correct for the average Linux distro that can fully self-host, but OpenWrt is not exactly like that.
I've seen quite a few OpenWrt package manifest files, which is what opkg is working with (I've written the script that reads them and creates the package table and index in the wiki) and I'm pretty sure most OpenWrt packages have only a few dependencies, very rarely a conflict, there is no such thing as "build depends" or "build conflicts" (as OpenWrt can't compile itself, these things are specified in the package makefile, in the build system source).
Then OK there are a few packages that provide the same command/alternative, but afaik there is not more than 5-6 alternatives for each thing.
The overwhelming majority of packages are just dumb things with one or two libraries as dependencies.
See this file for example, that is a "full-fat" package manifest containing all possible information about packages in base repo of x86_64 target https://downloads.openwrt.org/releases/19.07.2/packages/x86_64/base/Packages.manifest
The manifest in this "full fat" form is used by the wiki script, but the actual package manifests in the package lists use the same format, they just dropped the less-important fields (source, maintainer, license and whatnot).
But it does need to consider the short/ long description of the the packages, an entry that doesn't matter for the dependency resolution itself and can largely be skipped there.
Hm, that's a good point. I think I found a way it can use less RAM. At least in concept, because I don't think I can actually fix the code. I can write a shell script that replaces this opkg list functionality though (as that's more or less what the script I wrote for the wiki also does when parsing package manifests).
Each repo gives a package manifest for Opkg (which is a text file), and this means it has to parse them all. But it can't just read and write instantly to console because it wants to print an alphabetical list of packages.
This means it has to read all files, sort stuff and then print it. It seems this requires too much RAM.
But if it does not care anymore about making a single alphabetical order, and is just reading and printing immediately it's not using much RAM anymore. This means it can only print an alphabetical list of packages for each repository (the manifest is already sorted alphabetically), but I think it's still OK for what most people use opkg list for.
@dark-penguin since this is an issue only when doing opkg list, it's not the same as the bug I linked. You might want to open a bug report where you explain the issue and maybe how I proposed to "fix" it, or just make the bug report, link it here and I'll post the proposal there myself.
