OpenWrt has been struggling with the flash space on small devices for a long time. Hell, it struggles with what to do with ample storage space on large devices, but that's another topic.
The vast majority of those same small-flash-space, though, have USB capability. It wouldn't actually be all that difficult to make the internal flash space a secondary bootloader that booted a device from USB. While this would do nothing for the RAM in small devices, it would eliminate virtually all storage space limitations across the board. The cost is that everyone would need a USB stick of some sort to use OpenWrt.
So, I'm curious, what is the consensus on requiring a USB stick in order to use OpenWrt if it means eliminating storage space limitations across the board.
YES - I support requiring a USB stick for all devices to elimitate storage space limitations
Keep in mind that quite a few of the affected devices don't even have USB ports to begin with (tl-wr841n, most of the xiaomi devices, etc.). Low flash usually goes hand in hand with low RAM as well, so not much to be gained from 'huge' USB storage.
Yes, thank-you. I did mention that in the OP. That said...
... virtually all devices are affected. I would assert that virtually all devices are more hampered by their limitation on storage than they by their RAM.
With more storage space you can add swap. With more storage space the average user can use opkg to swap out features they don't need for ones they want. Something you can't otherwise do without a custom build or a new custom image. And, it eliminates wear on irreplaceable internal flash, something that OpenWrt is hard on.
In any case, this isn't about convincing people. I'm honestly curious about the perception.
4 MB devices are definitively hampered by their flash size, no doubts about that.
Starting around 8 MB (even considering the announcement to deprecate 8/64) that holds less truth for the time being. On the one hand going from wolfssl to mbedtls will recover a little space, on the other hand there are some things to keep 8 MB flash devices going for a few more years (opkg, mbedtls (and with that losing https access and wpa3, neither of which would be a major problem for now) - the problem just is that neither of these do enough to help out on 4 MB flash.
Pretty much all (very few exceptions exist) devices that fall into the 4 MB flash category will however also fall into the 32 MB RAM category, and that is a real problem right now already. Depending on the target, its RAM hunger (xDSL modems allocating some RAM for device firmware), this may range from failing to boot a 'normal' image to failing under duress (WLAN on, multiple connected STAtions). The problem of rising image sizes is the upgrade process needing to master them - e.g. I'm already in a situation where I have to aggressively kill processes and prevent kernel modules from loading to be able to sysupgrade successfully (8/32 devices).
swap on flash (regardless of internal flash (even worse, due to flash wear) or USB (aside from very highend USB stick or SSDs, but you aren't very likely to use to on a 4/32 device) is something that's never a good idea (as it slows down the device massively and damages the swap location).
I do actually think that chainloading OpenWrt from USB might be a nice option, but it will fail for many devices that would need it most - and even where it does work, the low RAM sizes keep its effective improvements limited (you may now phyiscally have the storage space for larger images, but increase the RAM pressure even more).
Then I'm quite concerned about the exact terminology you're using here, as I read it as all devices without USB slots will no longer be supported ("Would you support requiring a USB stick for all devices", which seems to be re-affirmed by "Hell, it struggles with what to do with ample storage space on large devices" and gets rather explicit with "The cost is that everyone would need a USB stick of some sort to use OpenWrt").
My answer to this would be a firm, "no, hell no!".
Quite a few mid- to highend devices these days (802.11ac and 802.11ax) come with healthy amounts of internal flash (256 MB NAND is common there, of which often around 80-100 MB can be used (dual-firmware), down to ~40 MB on some Linksys ones, not great, but not a hindrance for using OpenWrt as a router), but don't have any USB ports (the SOCs all have it, but many brandnew devices don't bring them out) or other means of adding removable storage (nor eSATA, nor sdhc). USB3 ports have become a rarity over the last 2-3 years (at best you may get one USB2 port, but even that's no longer common), even for high-end and popular -already fully supported- devices. This idea -especially following your intended phrasing of 'must have USB'- would break many very useful 802.11ax devices users have bought within the last 2-3 years needlessly.
Btw., I've just now looked through the devices which were added this year (since 2023-01-01; I might have missed devices whose commits didn't start with "add support"), so devices someone cared enough about within the last 3 months to port them to OpenWrt, mostly devices that are still sold on the shelves:
no USB:
Keenetic Lite III rev. A
Mercusys MR70X
Netgear WAX218
TP-Link Archer AX23 v1
FortiGate 50E (FG-50E)
MikroTik RouterBOARD 911 Lite2/Lite5
D-Link DWL-8610AP
Netgear WAX206
APRESIA ApresiaLightGS120GT-SS
ASUS RT-AX54
TP-Link Deco M4R v4
Xiaomi Mi Router 4A Gigabit v2
Senao Engenius EWS660AP
D-Link DIR-629 A1
Linksys WHW03 V2
ZTE MF18A
Xiaomi RA75 Range Extender
ELECOM WRC-2533GHBK2-T
Netgear GS750E
D-Link DAP-X1860 A1
Cudy M1800
Keenetic KN-1613
Fortinet FAP-221-B
Edimax CAX1800
Redmi AX6
Xiaomi AX3600
USB 2.0:
OrayBox X1
CJ-Hello HYC-G920
AVM FRITZ!Box 7330
Sercomm H-500s
Nokia Airscale AC400i
Watchguard Firebox T10
Buffalo LinkStation LS220DE
GL.iNet GL-X1200
Huasifei WS1208V2
Senao Engenius EPG600
Senao Engenius ESR1200
Senao Engenius ESR1750
Senao Engenius ESR900
Edgecore EAP102
USB 3.0:
ASUS TUF-AX4200
Wallystech DR40x9
SNR-CPE-ME2-SFP
D-Link Dir-853 A1
Buffalo WXR-5950AX12
D-Link DIR-1935 A1
ZyXEL NBG7815
Dynalink DL-WRX36
Xiaomi AX9000
QNAP 301w
none of the above have alternative means to add removable storage (eSATA, sdhc or similar); apart from one devboard (but that already counts in the USB3 bracket).
--> 23:19 --> 55% of all newly supported devices this year don't come with any USB ports
EDIT: I've manually added the missing ipq807x devices to this list, which mostly used a slightly different naming scheme in the commit messages.
--> 26:24 --> 52% of all newly supported devices this year don't come with any USB ports
This would affect all supported managed switches (the Realtek rtl838x/ rtl839x/ rtl93xx SOCs don't have USB support) and most wall- or ceiling APs.
I would wholeheartedly welcome the general possibility to boot OpenWrt from an attached USB drive, but not primarily as a stopgap for low-flash devices. IMHO it would be really, really useful to be able to hold OpenWrt versions on inexpensive USB sticks for testing purposes, to move a system from one device to the other, or even to keep OpenWrt off the internal storage altogether (I would love to do that with my My Book Live Duo).
TurrisOS offers something nice, but orthogonal to this topic. You can reflas the turris o,nia via a prepared usb stick from within the bootloader... that is more relevant for TOS as the omnia does not use a simple read only rootfs, but uses btrfs for the root partition... I guess the problem for OpenWrt is that bootloader probably need to support that...
I've got a Ventoy USB stick set up to boot OpenWrt. I just tried it on a laptop, and it came up with the boot menu, I selected OpenWrt 22.03.0 and it came right up.
I can copy any .img to it, and it auto-shows in the boot menu, so you could have lots of different versions from which to choose.
Yes, that's X86, one of the few targets with USB storage enabled, and BIOS/EFI is able to boot from USB out of the box.
However, on embedded systems it's a different story. Lots of bootloaders have no concept of USB in the first place, because why would they. And I'm not entirely sure if it is enough to just enable USB storage support on a target to make OpenWrt bootable from USB. I suppose some targets would require additional work.
When I said bootloader, I didn't mean necessarily a formal bootloader. It could be simply leveraging the existing extroot mechanism and having what's on internal flash be just enough to get the USB going. Essentially turning the internal flash into a second-stage bootloader.
But ... that's literally what extroot does. You may have to strip down images to fit the necessary (kernel) packages, but it's still just the existing extroot mechanism. You would not "boot from USB", you would not "leverage the existing extroot mechanism", you would use extroot just as it is and always has been intended.
No, that's not at all what a second-stage bootloader is and does.
So maybe (not that this is realistic, for a number of reasons*) this proposal boils down to always build factory, syssupgrade, and extroot-sysupgrade images?
Personally I like the idea of this proposal, but what I don't agree is: Using more swap which might kill your USB stick easily, the purpose of squashfs is to protect the flash from wearing, however it's generating wearing on USB stick. You can replace it, but you will not know until one day the router suddenly stopped working, so for the extroot one I won't do it on low memory device because of this concern, I don't want one day someone call from home saying router is dead when I am in a travel.