Should OpenWrt/LEDE support devices with only 4MB Flash?

For Developers
133

No. My workhorse, a ZBT-WE826, is about $25 plus VAT, so it is about $30 in summary.https://www.aliexpress.com/item/802-11N-300Mbps-MT7620A-openwrt-wireless-router-300Mbps-WiFi-Router/32361575074.html?spm=2114.search0104.3.112.6ac537cb177LSO&ws_ab_test=searchweb0_0,searchweb201602_4_10065_10068_10547_319_317_10548_10696_10084_453_10083_454_10618_10304_10307_10820_10821_537_10302_536_10902_10843_10059_10884_10887_321_322_10103,searchweb201603_56,ppcSwitch_0&algo_expid=70e20e40-2140-468a-af7f-e01acabf236b-16&algo_pvid=70e20e40-2140-468a-af7f-e01acabf236b&transAbTest=ae803_3
Various variants of this one available; make shure 128MB RAM. For volume orders, you can even get a 256MB version.

134

Anyway, a stable, frozen minimal up-to-date openwrt for popular 4MB flash devices (i.e. TP-Link 841ND) appreciated. Still usable as dumb AP, for example.
Although I am not really "Grean": Recycle, instead of dump.

1 Like
135

Thanks for your note. I understand the incentive to avoid bloat, and the urge to keep OpenWrt relevant to lower-price routers, but I don't necessarily agree with your argument.

  • Don't get me wrong: In 2014, OpenWrt was a godsend - it turned the "steaming pile" of vendor firmware into a functioning router that ran reliably without security problems.

  • But, a "$20 router" in those days was a pretty minimal affair. 2.4GHz wifi, wimpy processor, small RAM/Flash. They worked well with the (minimal) internet speeds that were predominant at the time. Nor did they provide a satisfactory internet experience (bufferbloat).

  • Today, "we all want" 2.4GHz and 5GHz connectivity. Many people have 100-500 mbps ISP connections. Your $20 router wouldn't hack it at all, but low-end (now, $40 routers, especially if you shop on eBay or Craigslist) routers are "the standard" and can support these speeds. SQM with Codel/Cake decreases latency for you traffic, but requires a beefier processor than the lowest end.

  • There's also the question of balancing developer focus: maintaining backward compatibility vs. working on new features.

  • It's also true that Ubuquiti equipment is very compelling (as are other OpenWrt-out-of-the-box routers such as IQrouter) since they "just work". They cost a little more than an equivalent OpenWrt install would but your time and experience count for something, and you don't have to be the tech support person for other people's network.

That said, I do think that a Community Build that's focused on specific popular models (such as TP-Link 841ND as @reinerotto suggests) could be valuable.

But I also want continuing features/security fixes, both for myself and for the project. It provides a good experience for people using them, a platform for experimentation (Codel/Cake were developed on a branch of OpenWrt), and a secure router/gateway for your home/business.

Back when I made the original post of this topic (January 2017), we didn't have a clear way of knowing what fraction of our population uses <4/32 devices. I'm not sure we do today either. I'm checking the Table of Hardware to see the hardware specs of newer devices and will report back.

Thanks for your thoughts.

3 Likes
136

Rarely saw a post on a forum about open source software with that much ignorance.

I want to see Ubiquiti hardware starting at ~35€.

:roll_eyes:

"everything is instable" for millions of devices outthere? Im using at home and on several places network hardware with OpenWrt or Gluon (a fork) since 2012. I provide internet for (mostly in summer) about 150 people thank to a project called Freifunk. Never had big issues the last 7 years. The only fuck ups i saw came from the manufacturers themself (proprietary drivers/firmware/blobs).

About OpenWrt getting bloated... I call myself a bloody beginner in matter of programming, but im following closely what is happening in the Linux world for more than a decade. One is really obvious: software get with the time bigger. You cant stop that. The biggest Problem for 4/32 devices is the linux kernel. It will continue to grow. Not because the devs are lazy, but because of features and security updates. And specially for the kernel: is your code not written perfectly, it will never be accepted. By the way the kernel itself is not made here. Check kernel.org.

So do yourself a favor: better inform yourself for the next time and stop pissing on the head of those working mostly in their freetime to get you access to open source software. Or in better words: freed software.

5 Likes
137

Update: I have rummaged through the OpenWrt Table of Hardware and done a quick (half-assed, I guess) analysis of the Flash/RAM distribution of recent devices to get a sense of "routers that we're talking about" in the discussion of what are popular/relevant routers.

I separated the full list of devices into "Devices available in 201X", where they were listed either as "Available" or "Discontinued" in that year. This is a sloppy analysis:

  • The ToH data is imperfect - there are many devices where the Availability is unknown/not set. (There are 256 routers whose availability is listed as "Unknown 2018", which means that they are simply not counted in this analysis.)
  • For simplicity, I also counted as "Other" devices listed as 128NAND or where RAM was listed as "¿". This also leaves out many, many routers.
  • Finally, the analysis does not consider any particular device's popularity: those devices will be under-represented in these stats.

Observations: Even in 2016, the most common Flash was 8MBytes, the most common RAM was 64MBytes. This has remained steady through the present. In 2019, there were no 4MByte Flash routers, and only 7 with 32MBytes RAM.

The original question posed above is whether 4/32 devices have sufficient popularity to warrant their continued attention from developers. My sense, in 2019, is No.

The underlying data is available from a Google spreadsheet (below). Enjoy!

2 Likes
138

PS I can see that this topic may generate some emotion. Let's all remember to be respectful and Follow Rule #12 - Be Nice to Each Other. Thanks.

2 Likes
139

Well sorry if i went to far. I cant just stand this kind of post. Without your massive efforts, we would all pay thousands in a router with that much features, just to be laid off 2 years later because of "wE hAvE A neWeR PrOduCt".

Back to the topic. I think it wouldnt be that bad to freeze those devices into the 4.4 LTS kernel since it will be supported until at least 2022. Some kind of LTS support for 4/32 devices (security) if you want. Those devices will probably disappear like wrt54gl once did very soon. Maybe its more useful at this point to fork OpenWrt into a 4/32 project for those willing to do so, Like Arch linux did with 32 bit cpus as a community project.

140

The thing is time spent supporting 4/32 devices is time not spent supporting newer devices or adding new devices or etc.

There are plenty of decent routers with more flash and ram at very low prices, like GL-inet devices for $20. It makes no sense to spend effort on these (older 4/32) devices unless someone has a convincing reason. A good convincing reason would be if an organization has say 10 million of these and wants to spend $0.10 each to get them working... $1M will get you a bunch of development time.

Short of that, mostly people are going to find that it makes more sense to get a lot more functionality for $20 by buying newer hardware.

5 Likes
141

Well with all Freifunk communitys combined in germany, we have about 42000 nodes. The problem we have is, that a good part from them are TL-WR841N/D. Mostly Freifunk firmware are based on Gluon, a fork from Lede/OpenWrt.They put already a lot of effort, like less ram usage, to keep those devices running. Some problem exist for the communitys and the Freifunk isps providing the vpn infrastructure because of those low-end devices.

Throwing money in wont help much, its already happening for low-end device related bugs.

And im not counting similar projects in the entire world.

Thats the idea i had behind a fork freezing those device on a certain LTS Kernel and focusing on keeping them running for the next few years. And later the same with 8/64,16/128 devices and so on, once they become obsolete too.

By the way maybe somebody can check gluon sources, they have done some nice things lately about ram usage...

142

So imagine you can do two things, one is buy new $40 devices for each of these, and get improved results due to more power. the other is spend $2 each to reserve one talented developer to maintain this for a year on the existing hardware. It's cheaper to do the maintenance sure, but the functionality is very limited. It seems straightforward to set up the patreon account if a talented developer wants the money. Now, how many people will actually do the donation?

143

Only the core community itself, mostly not the local restaurant providing his clients a cheap internet access. Rare are those making donations toward us. We get financial aids in many ways too, like donations and sometime from towns/city or state parliament. Law changes might come the next years in germany in regard of volunteering work in matter of Freifunk itself. Advantages like taxes cut help a lot too.

We spend some of the donations toward gluon too, sometime money, sometime buying devices to hack around. Or paying bountys to random devs to fix bugs.

But money should not be the driving force here. Pay the running bills but no more. Money can kill a project like that, see what happened to Cyanogenmod (that thankfully survived as a fork: LineageOS).

1 Like
144

I agree that chasing ever more money through "monetization schemes" isn't a good plan, but paying for specific development tasks is a good way to focus energy where it otherwise would have little reason to be focused

3 Likes
145

Time, however, is a driving force. OpenWrt is effectively a volunteer effort as well. I consider it a horrible waste of that effort to try to keep alive every single piece-of-crap device that doesn't have sufficient resources to run a secure kernel and secure software. The bottom of the barrel is now flooded by no-name clones and manufacturers' offerings under the same name that have even less hardware inside than the previously crippled version.

Perhaps a reasonable compromise is to, for a brief period of time, keep a very select list of devices that either don't have at least 8 MB of flash or don't have at least 64 MB of RAM supported. Yes, they'd be EOLed for support too. You can't say you haven't been warned, this comes up every year. So this would be "fair warning" to go out and, for the price of a couple coffees, beers, what have you, buy something decent.

Yes, you can buy a 2.4 GHz router with 16 MB of flash and 128 MB of RAM from a reputable manufacturer for under US$20, delivered, with an unlocked, full-featured U-Boot (including HTTP access) and factory-supported OpenWrt that is current and close to the tree.

I think one way or another, the OpenWrt wiki is doing everyone a disservice by being so wishy-washy on 4/32 devices.

The public information indicates far too clearly that

Sorry, they're hardly "Ideal for OpenWrt" as they're on the borderline for robust operation on modest speed lines and 8/64 devices are about to follow in the path of 4/32 devices.

I wouldn't buy anything with less than 128 MB of RAM these days, even used. Time to move that up and

  • 4/32 warning -- "Not supported" (though firmware may be available, we ain't fixing the bugs or listening to you whine any more)
  • 8/64 warning -- "Expect limited support and lack of functionality for any more than basic functions. Expect that device will become obsolete quickly due to limited resources."
  • Supported devices only lists 16/128 or higher
  • Ideal for OpenWrt devices omit MIPS-based devices and other low-end SoCs. Omit any that do not support 802.11ac. This is "ideal", not "supported" after all
6 Likes
146

I agree with Jeff about what information we give on the wiki and table of hardware. People should get more easy access to up to date recommendations not just technically it sort of works.

1 Like
147

I don't see this as a common consensus among the developers.

That will not happen.

3 Likes
148

In any rational enterprise, there are several different things:

  • What expectations are set with customers and prospects
  • What the product can do
  • What the organization will warrant the product will do
  • What the organization will help customers with on a reasonable basis
  • What's just not going to happen

The wiki reflects the first -- positioning and marketing

I also see that, at least as I interpret it, multiple supported branches, multiple kernels, or build-time hacks are out, from the next bullet in the quoted post

149

I think it would be useful to have ideal, well supported, and minimally supported, a binary "supported" vs "ideal" is not enough granularity.

1 Like
150

Should we support 4MB? Thats the original question. As long as the same target has both 4MB and 8+MB flash there shouldn't be any objection. Development time in terms of getting the linux kernel working / up-to-date / secure for that target will be spend anyway. Giving users the option to create their own (custom) build for that target with less flash should remain. Same goes for 32MB RAM vs 64+MB even if that device has 8MB flash. Let the user (repurpose) an old(er) device, make it a managed switch, range extender etc..

So the question is: "what means support". Support would mean: help the novice end-user to install and basic configure the device. This type of support is not done by most of the developers, but is mostly done by other end-users. As long as there are still people willing to make "small / tiny / custom" builds I don't see the problem.

Lets take the "EdgeRouter Lite" (https://openwrt.org/toh/ubiquiti/edgerouter.lite), dual core MIPS64 512 MB RAM..but 4MB FLASH. It should be able to run circles around most populair TP-LINK / NETGEAR / B-brand lower-end devices...if it wasn't for the small flash.

Sure something like this: https://www.aliexpress.com/item/Fanless-Mini-PC-pFsense-Celeron-J1800-J1900-Quad-Core-4-Gigabit-LAN-Firewall-Router-Windows-10/32884221255.html?spm=2114.10010108.1000023.13.4ce63a6fm8eL1Y would be nice...but even that I wouldn't buy: the J1900 doesn't have AES-NI, the device only has VGA (I don't even have any monitor with that connection anymore).

Long story short: as long as the same target has 8/64 or better it should not make a difference in terms of development time (which is what counts at the end of the day), those 4/32+ devices only take up a little space to accommodate their DTS and some make (type) of files.

151

Disclaimer: I'm not an OpenWrt developer and can't speak for them.

The question isn't about "should", but "can" - if you look at the ar71xx-tiny target as an example, you see the sysupgrade images currently (18.06.2) have a size of around 3584 KB (and that's not really representative, as 18.06.x is still on kernel 4.9 for ar71xx, for kernel 4.14 you have to add just shy of 192 KB growths - and that's the kernel alone, userspace is growing as well), even on devices that use a very economic partitioning, you lose at least 192 KB to bootloader/ bootloader environment and calibration data, another 192 KB (= three erase blocks) are the bare minimum for the jffs2 overlay to work (and in practice you need at the very least 4 erase blocks (256 KB) for a most basic configuration).

   128 KB (u-boot/ u-bootenv)
+ 3584 KB (average 18.06.2 ar71xx-tiny sysupgrade image size)
+  192 KB (needed on top for kernel 4.14)
+  192 KB (bare 3 erase blocks required for jffs2)
+   64 KB calibration data
=========
  4160 KB

Oops, even assuming zero growths for the rootfs (userland), this is not going to fit into 4096 KB anymore. Yes, this calculation isn't 100% correct, as compression efficiency might sway the numbers slightly, but the tendency is spot on - and many of the affected devices with 4 MB flash are more wasteful in terms of their partitioning. So unless anyone steps up (very soon, basically it's already to late for that now), providing actual patches, 19.03.0 (as we know it, with opkg, luci, ppp support) won't fit on any 4 MB flash device anymore.

Keep in mind, this is not a fault of the 'evil' OpenWrt developers, on the contrary - they've spent considerable efforts to create the tiny target in the first place, to allow saving as much space as humanly possible (otherwise 18.06.0 wouldn't have existed for just about any of the 4 MB devices). And the estimates above already assume 0 byte growths for the rootfs (which is unlikely). A considerable amount of effort has also been spent to push support for these affected devices to use a dynamic partitioning split, in order to make the growing kernel fit at all (most had a fixed size limit between 1.3 MB and 1.4 MB for the kernel). So please don't assume bad faith or an active unwillingness to keep these devices working, really hard work has gone into supporting these even recently.

Considering this, what are your suggestions here?
The only place where one could (relatively easily) save a few more bytes is not including opkg and luci in the images, will that help in the medium term - no, because kernel 4.19 will grow again (that's not a guess, but a fact based on the already existing 4.19 support for some targets, assume at least a bit over 100 KB there). So all you can get with this alone is maybe another major release.

Just to re-iterate this explicitly, no one has suggested to remove source-level support for the affected 4/32 MB devices so far (on the contrary, you will still find source-level support for 16 MB RAM devices, even though they have no chance to even succeed in booting a release- or snapshot image). This means you will still be able to build firmware images for these devices for quite a while longer, but you will inevitably have to make hard choices about which (basic) features you'll have to cull from your custom images (and this will get increasingly harder over time).

Do you really think supporting novice users to configure their devices using ssh alone (because the webinterface simply isn't going to fit into (remotely) release-style image anymore, pretty much regardless of what you do) would be a raging success? Oh, and opkg or pppd aren't going to fit either, so you'd need to teach them about (at least) imagebuilder and/ or building form source at the same time…

Continued support for these devices really needs very active help, as you can't expect the existing developer base to 'suddenly' find a miracle they haven't been able to find within the last >5 years. It's not as if this would be a surprise, kernel- and userspace components have been slowly growing since the dawn of time, for at least 4½ years (15.05.0) it is becoming glaringly obvious that 4 MB flash has come close to its end (given that most affected devices had 5 erase blocks (320 KB) left at most), with it just being a matter of time until it won't fit anymore. Both 17.01.x and 18.06.x (18.06.x wouldn't have fit at all, without the introduction of the tiny sub-target) showed serious problems with the firmware images fitting already - and unless you do drastic last minute changes to the configuration (removing opkg, luci, etc. from the release images) 19.03.x most likely won't fit on any 4 MB flash device anymore (again, we're talking about release images here, that won't prevent you from building stripped down images yourself!). If you consider that 19.03.0 is pretty much just around the corner (weeks, not months left), it's effectively already too late for them.

Just please don't let your only suggestion be "then don't update the kernel, libc, $userspace_in_general", that's almost the only option that isn't on the table - if you don't care about security issues, bugs or support for newer devices (yes, there has even been support for new 4 MB flash devices added to OpenWrt recently, even though the developers are starting to push back against new additions for these underspec'ed devices), others do very much. The current set of developers simply can't keep track of even more concurrent kernel versions in parallel (even for still supported LTS kernel lines this is a considerable amount of work, with basically zero on-device testing happening).
…and if you don't care about glaring security issues anyways, no one will stop you from using older versions and to become recruited by multiple nefarious botnets - just be aware that this is strongly recommended against.

As you'll notice from the figures above, very little of the image growths stems from active decisions of the OpenWrt developers - it's mostly down to growing space (and RAM!) requirements of the various upstream projects (who generally consider smartphones to be 'embedded' and low-spec, the lowest common denominator they care about), if you want to change anything, that's where you need to start working (with the relevant upstreams). OpenWrt can't diverge from upstream much more, without losing track of current upstream work and the option to support new devices users actually care about.

9 Likes
152

Hear hear! Thanks for the hard numbers and explanations.