Should OpenWrt/LEDE support devices with only 4MB Flash?

Should we support 4MB? Thats the original question. As long as the same target has both 4MB and 8+MB flash there shouldn't be any objection. Development time in terms of getting the linux kernel working / up-to-date / secure for that target will be spend anyway. Giving users the option to create their own (custom) build for that target with less flash should remain. Same goes for 32MB RAM vs 64+MB even if that device has 8MB flash. Let the user (repurpose) an old(er) device, make it a managed switch, range extender etc..

So the question is: "what means support". Support would mean: help the novice end-user to install and basic configure the device. This type of support is not done by most of the developers, but is mostly done by other end-users. As long as there are still people willing to make "small / tiny / custom" builds I don't see the problem.

Lets take the "EdgeRouter Lite" (https://openwrt.org/toh/ubiquiti/edgerouter.lite), dual core MIPS64 512 MB RAM..but 4MB FLASH. It should be able to run circles around most populair TP-LINK / NETGEAR / B-brand lower-end devices...if it wasn't for the small flash.

Sure something like this: https://www.aliexpress.com/item/Fanless-Mini-PC-pFsense-Celeron-J1800-J1900-Quad-Core-4-Gigabit-LAN-Firewall-Router-Windows-10/32884221255.html?spm=2114.10010108.1000023.13.4ce63a6fm8eL1Y would be nice...but even that I wouldn't buy: the J1900 doesn't have AES-NI, the device only has VGA (I don't even have any monitor with that connection anymore).

Long story short: as long as the same target has 8/64 or better it should not make a difference in terms of development time (which is what counts at the end of the day), those 4/32+ devices only take up a little space to accommodate their DTS and some make (type) of files.

Disclaimer: I'm not an OpenWrt developer and can't speak for them.

The question isn't about "should", but "can" - if you look at the ar71xx-tiny target as an example, you see the sysupgrade images currently (18.06.2) have a size of around 3584 KB (and that's not really representative, as 18.06.x is still on kernel 4.9 for ar71xx, for kernel 4.14 you have to add just shy of 192 KB growths - and that's the kernel alone, userspace is growing as well), even on devices that use a very economic partitioning, you lose at least 192 KB to bootloader/ bootloader environment and calibration data, another 192 KB (= three erase blocks) are the bare minimum for the jffs2 overlay to work (and in practice you need at the very least 4 erase blocks (256 KB) for a most basic configuration).

   128 KB (u-boot/ u-bootenv)
+ 3584 KB (average 18.06.2 ar71xx-tiny sysupgrade image size)
+  192 KB (needed on top for kernel 4.14)
+  192 KB (bare 3 erase blocks required for jffs2)
+   64 KB calibration data
=========
  4160 KB

Oops, even assuming zero growths for the rootfs (userland), this is not going to fit into 4096 KB anymore. Yes, this calculation isn't 100% correct, as compression efficiency might sway the numbers slightly, but the tendency is spot on - and many of the affected devices with 4 MB flash are more wasteful in terms of their partitioning. So unless anyone steps up (very soon, basically it's already to late for that now), providing actual patches, 19.03.0 (as we know it, with opkg, luci, ppp support) won't fit on any 4 MB flash device anymore.

Keep in mind, this is not a fault of the 'evil' OpenWrt developers, on the contrary - they've spent considerable efforts to create the tiny target in the first place, to allow saving as much space as humanly possible (otherwise 18.06.0 wouldn't have existed for just about any of the 4 MB devices). And the estimates above already assume 0 byte growths for the rootfs (which is unlikely). A considerable amount of effort has also been spent to push support for these affected devices to use a dynamic partitioning split, in order to make the growing kernel fit at all (most had a fixed size limit between 1.3 MB and 1.4 MB for the kernel). So please don't assume bad faith or an active unwillingness to keep these devices working, really hard work has gone into supporting these even recently.

Considering this, what are your suggestions here?
The only place where one could (relatively easily) save a few more bytes is not including opkg and luci in the images, will that help in the medium term - no, because kernel 4.19 will grow again (that's not a guess, but a fact based on the already existing 4.19 support for some targets, assume at least a bit over 100 KB there). So all you can get with this alone is maybe another major release.

Just to re-iterate this explicitly, no one has suggested to remove source-level support for the affected 4/32 MB devices so far (on the contrary, you will still find source-level support for 16 MB RAM devices, even though they have no chance to even succeed in booting a release- or snapshot image). This means you will still be able to build firmware images for these devices for quite a while longer, but you will inevitably have to make hard choices about which (basic) features you'll have to cull from your custom images (and this will get increasingly harder over time).

Do you really think supporting novice users to configure their devices using ssh alone (because the webinterface simply isn't going to fit into (remotely) release-style image anymore, pretty much regardless of what you do) would be a raging success? Oh, and opkg or pppd aren't going to fit either, so you'd need to teach them about (at least) imagebuilder and/ or building form source at the same time…

Continued support for these devices really needs very active help, as you can't expect the existing developer base to 'suddenly' find a miracle they haven't been able to find within the last >5 years. It's not as if this would be a surprise, kernel- and userspace components have been slowly growing since the dawn of time, for at least 4½ years (15.05.0) it is becoming glaringly obvious that 4 MB flash has come close to its end (given that most affected devices had 5 erase blocks (320 KB) left at most), with it just being a matter of time until it won't fit anymore. Both 17.01.x and 18.06.x (18.06.x wouldn't have fit at all, without the introduction of the tiny sub-target) showed serious problems with the firmware images fitting already - and unless you do drastic last minute changes to the configuration (removing opkg, luci, etc. from the release images) 19.03.x most likely won't fit on any 4 MB flash device anymore (again, we're talking about release images here, that won't prevent you from building stripped down images yourself!). If you consider that 19.03.0 is pretty much just around the corner (weeks, not months left), it's effectively already too late for them.

Just please don't let your only suggestion be "then don't update the kernel, libc, $userspace_in_general", that's almost the only option that isn't on the table - if you don't care about security issues, bugs or support for newer devices (yes, there has even been support for new 4 MB flash devices added to OpenWrt recently, even though the developers are starting to push back against new additions for these underspec'ed devices), others do very much. The current set of developers simply can't keep track of even more concurrent kernel versions in parallel (even for still supported LTS kernel lines this is a considerable amount of work, with basically zero on-device testing happening).
…and if you don't care about glaring security issues anyways, no one will stop you from using older versions and to become recruited by multiple nefarious botnets - just be aware that this is strongly recommended against.

As you'll notice from the figures above, very little of the image growths stems from active decisions of the OpenWrt developers - it's mostly down to growing space (and RAM!) requirements of the various upstream projects (who generally consider smartphones to be 'embedded' and low-spec, the lowest common denominator they care about), if you want to change anything, that's where you need to start working (with the relevant upstreams). OpenWrt can't diverge from upstream much more, without losing track of current upstream work and the option to support new devices users actually care about.

9 Likes

Hear hear! Thanks for the hard numbers and explanations.

@slh, first of I'm not a developer either. Second, I am not disagreeing with you or your logic. The numbers as you show were already valid a year ago from the 17.x to 18.x and other topics here.

I was "happy" when the Lede-project actually took off. It was splitting the OpenWRT user base from the development part. The Linux Embedded Development Environment would have active development with a more power-user / amateur developer based forum. Then the fruits would benefit the OpenWRT community where the much larger (novice) user base would be able to continue supporting the lower end devices. OpenWRT would be kind-off like the fork from LEDE (and no longer the other way around). Fast-forward to today, we are faced with the remerged OpenWRT/Lede.

Lede development should (my opinion of course) continue development keeping up with the latest Linux development (a few steps behind). The developers should concentrate on development (improve features, security, stability).

So should "LEDE" support those devices. Then I guess if we need a Yes / No answer, I will vote "No". Maybe create a fork for those devices and find some users willing to maintain a 4.4 or 4.9 LTS kernel for another few years.

Thanks all for keeping the conversation respectful, and for the good analysis... Here are my thoughts on the several ideas that came up in the discussion:

  • As OP, I took the liberty of changing the subject to "Should OpenWrt/LEDE support devices with only 4MB Flash?" When I posed the question two years ago, I was asking about the then-current build (LEDE, pre-merger). The question is now relevant for OpenWrt.

  • @drbrains asked the critical question:

    So the question is: "what means support". Support would mean: help the novice end-user to install and basic configure the device.

    I agree that "our base of support" should be to help novice users get OpenWrt running on their router. Having spent a lot of time writing "get started" documents, it's clear to me that this implies a base system that includes a GUI. (It's already scary enough for newcomers to take their currently-working router with manufacturer firmware, and download and install something new. Making them also use a terminal to get it working again it is a step too far.)

  • I support the effort of anyone that wants to create a community build that addresses specific, popular routers that cannot fit the full 18.06 (or 19.03) release.

  • I also support preserving the full source-level build capability for marginal and unsupported devices, even if that configuration never gets automatically built (or installed or tested.)

  • Finally, the OpenWrt project needs to conserve developer time. As several have pointed out, time spent on fitting firmware into limited resources (4MBytes Flash) takes attention away from security fixes, new packages, features, etc. @slh's calculations of Flash requirements for current/newer kernels make it clear that 19.03 cannot be "supported" (e.g., with GUI) on 4/32 devices.

1 Like

Shows that it is already quite questionable if it was a good idea to provide prebuilt firmware images for 18.06.x already.

3 Likes

Fully agreed.

I'm already thinking about what to do with devices affected by this and how to mark them in the dataentries.

What about marking them "18.06.2 - BYO" (Build Your Own), and removing the download links?

As looking at things from a product-focused perspective, I can get behind a reasonable objective such as that proposed by

With that focusing the objectives of the project, I'm of the opinion, that any devices that require custom or community builds, shouldn't be put forth on the wiki (OpenWrt's "product brochure") as anything even closely related to the word "Supported".

"Novice users" would likely interpret "Supported" to mean that they could get the full OpenWrt experience as they picture it, which almost certainly includes GUI and potentially includes bufferbloat management and various "security" features, such as VPN and various flavors of DNS control/security/obscurity.

I've lost track of the number of times and the number of general functions that new users repeatedly post here something like

But I can do [Repeater, VPN, Guest network, ... ] so easily in the [OEM] browser,
Why can't I do that in Openwrt?
OpenWrt is supposed to be better than [OEM]'s software!

1 Like

It's all a question of how you inform the users.

Is that clear enough?

6 Likes

The problem, as with all documentation / wikis / warnings... people don't really read those. They will first flash (even the wrong image), then panic and seek help.

Not providing pre-build small/tiny (4/32) images would help a lot, but expect a lot of questions why not, where to find and a lot of custom builds of questionable quality. There was already a long discussion about not trusting "any" custom build.

As an alternative, the Kernel 4.4 is LTS until (projected Feb 2022) so at least should be "secure". People could use a Lede 17.x version. After all, there are still a lot of devices running OpenWRT 14.x or 15.x. which are not secure at all anymore.

DD-WRT and Padavan are still on a 3.x kernel (Tomato I didn't check). So keeping the 4.4 (or maybe 4.9) alive with only security update (no features etc) would be a big plus over all those.

2 Likes

Perhaps the ToH could distinguish between the following:

  • GUI Support - Full release build, with LuCI, IPv6, and the rest of the bells and whistles, as we do now
  • SSH Support - Something like the "tiny" builds today - no GUI, but modern kernel, some ability to support additional packages
  • BYO Support - You're on your own with the current build

This is complicated by the fact that GUI/SSH/BYO is a reflection of the ability to jam the firmware into the available Flash memory. Whether there's enough RAM for the device actually to function is an entirely separate question.

1 Like

Might it be simply something like "Only limited support since 18.06.x".

Trying to differentiate between limited supports classes (SSH, BYO etc.) will get cumbersome in the long run.

Yes, quite clear. And would you like a little attitude with that? :slight_smile:

If we decide to adopt a BYO category, we might soften the article to say something like this:

Why does OpenWrt no longer support 4MByte Flash devices?
Technology marches along, and OpenWrt developers have already spent significant effort to create a build that will fit in these devices.

In view of the wide availability of ~US$20 devices that fully support OpenWrt, it's not fair to ask the developers to spend further time creating pre-built binaries.

Then the other questions...

Another thought: Who's testing those pre-built binaries? My guess - no one. I offer this semi-facetious suggestion:

  • If you wish to have new firmware for a low-capacity device, please post a note to the forum, offering to send one of those devices to a developer who can custom-build a binary for you.
  • For priority service, offer to include US$100. :slight_smile:
1 Like

Yes, but it leaves unsaid what "limited support" means. I was hoping/casting about for short identifiers that could convey that device's status with regard to current pre-built binaries.

There's also the maintenance question - when we ship 19.03 (or 20.04), how does the ToH get updated to reflect new limitations for specific devices? I know @tmomas has a (semi-automated?) process - would these different categories create undue heartache?

I don't think differentiations between SSH / BYO etc is a good idea. It will create a lot of problems, generate a lot of questions and possibly maintenance headaches.

As suggested consider keeping Lede 17.x with a 4.4 LTS kernel and patch big security problems. Don't build those images anymore with the "bots", don't even spend those resources. However keep the sources available so (power)users can built their own. And actually depending how you look at it, build your own should always be done. Every user has a different use case and that has always been the power of OpenWRT in comparison with let's say DD-WRT where the user gets the "full" experience what those developers envision with limited options to add or remove features.

At one point in time every main distribution had to make similar choices: Be it Windows 98 -> 2000 -> XP -> 7 -> 10 all those steps left "hardware" behind. The some (possibly worse) for Apple Mac OSX. My perfectly running iMac (late 2013 with 8Gb, SSD intel i5) is NOT able to upgrade to OSX Mojave (at least not via the official way). Of course Linux always supported old hardware much longer and was still able to have a useful modern machine but even old hardware has to start looking for "alternative" distros, not lets say a full featured Ubuntu / Mint / ..

I think what should be done is distinct from what should be communicated to end users. We need to be nice to end users and this means two things

  1. give them the information they need to make good purchase decisions and to not waste time on fools errands.
  2. Make it hard for them to misunderstand, waste time, or get out of their depth.

Part 1 means avoid giving the misimpression that 4/32 devices should ever be purchased. Table of hardware should list them as unsupportable for modern versions of OpenWrt. Sure if you have one and have skills to build custom stuff you can maybe get it to do some limited stuff, but you should never ever spend any money to get a new one. Advanced users will dig into details, but new users need to see this will not work

Part 2 means don't have downloadable images or anything people could waste time on. A good user will try to do some stuff on their own, they will download the image and flash it and not connect to LuCI because it's not there and then think they have a network problem and debug that and etc. They'll waste a whole afternoon. Don't be mean to your users by trying to be nice to them. Creating an image without a GUI that can barely save configs is not a service, it's an invitation to waste time. Better to point the users at an online build your own instruction manual and a list of inexpensive modern hardware as alternatives.

Here's the current warning

Devices with ≤4MB flash and/or ≤32MB ram suffer from limitations in usability, extensibility and stability of operation. Consider this when choosing a device to buy, or when deciding to flash OpenWrt on your device because it is listed as supported. See 432_warning for details.

I'd recommend rewording this more strongly. Something like:

Devices with < 4MB flash and/or < 32MB of RAM have barely enough space just for the Linux Kernel and should never be purchased for use with OpenWrt. If you have this kind of legacy device already you may possibly be able to build your own image to do limited functions such as acting as a simple managed switch. OpenWrt does not provide pre-built images for these devices

3 Likes

In your own words: Hear Hear !

I'd also recommend to make it hard to get a Table of Hardware that includes these devices. Basically by default they're filtered out and then you can dig in somewhere and eliminate that filter if you're paying enough attention.

And, I think we should set "ideal for OpenWrt" to mean greater than or equal to 16 MB of flash and greater than or equal to 128 MB of RAM. Today you can buy devices that support those specs for $18 https://www.amazon.com/GL-iNet-GL-AR300M-Lite-Pre-Installed-Performance-Compatible/dp/B073TT7KB2

if someone buys a lesser thing and spends a half an afternoon futzing with it, they've lost irretrievable hours, and many of the devices with less actually cost MORE.

How about making v17 an OpenWrt LTS release with only vital security updates? You could stop spending time on v18 and release v19 with one or two maintenance releases. After that add vital security fixes for v17 while the kernel is being maintained, and only add new features and general fixes in master for v20. Make a new OpenWrt LTS release when 4.4 will no longer be maintained. The next LTS release would probably require 8/64, but then those routers might also get a longer life span.

I can't see the problem. Currently even with the 4.14 kernels default config fits into 4/32 devices. It is true that it fits only "just" and installing additional software is almost impossible.

So that could be mentioned in a warning. I ported tp-link mr3040 to ath79 and it builds well with Luci.
For my personal use I had to remove some default packages to be able to add some that I need. But for an ordinary user it's quite OK to use a prebuilt image.