If your problem is solved (or at least reached a logical endpoint/conclusion), please consider marking this topic as [Solved]. See How to mark a topic as [Solved] for a short how-to.
Thanks! 
The challenges for placing root to the net exceeded the time budget for my current requirement. However, there still may be situations where this approach might help to overcome the flash (not RAM) memory limit of gadgets.
So let's look back to collect what I've learned, just for the case that sbd (may be even myself) might pick up the idea of network based extension of gadget storage at some point in the future.
Reasons that might justify the effort required:
- huge pile / reuse case of 4/32 MB devices
- openWRT outgrowing the limit of 8 MB flash devices in a foreseeable future
- central management by keeping configuration on a server
- large number of similiar devices
- special needs
Network File systems:
sshfs
- was my first idea, since it is ubiquitously readily available
- the client gadget must be able do perform ssh client dialout (dropbear can do that) and some extra (lean) sshfs driver
- however, for a proper mapping of file ownership and access rights, root access on the server is required.
There might be some mechanism of UID/GID-mapping available, but I did not manage to configure it properly. - sshd (at least on debian server where I tried it) allows to configure chroot jail for specified clients, thus mitigating the risk of root access by gadgets deployed out there in the wild.
- sshfs does not provide the extended acl properties required for overlay upper - for chroot, however, it might suffice
network block device aka nbd
- Configuration on the server side is quite straightforward
- server provides a mere bunch of bytes where any file system (including ext4) may be installed
- thus all cabalities, including extended acl, of this file system is available
- full fledged file system drivers are required on the client
- partitioning and formating can be done on the server
- I succeeded in setting up an overlay with standard squashfs
/romas lower and 100 MBext4in annbdas upper layer - I experienced a overlaod condition on the server during my tests, where I'm not sure whether they were are to be attributed to nbd.
For futher tests, I'd recommend to setup dedicated hardware.
In normal operation, hardware requirements are quite moderate. - although
nbdappears to be called quite early in theinitsequence, it's not succesfully instantiated.nbd-clienthas to be restarted, sometimes repeatedly, for full operation, afterinit. - Maybe, if this could be solved,
nbdwere available at the end of init-sequence (/etc/init.d/done), whenextrootis called a second time - the "
extroot"-machine currently ist not aware ifnbd.
Thus, either it's effect has to be achived by other means (as I tried), or it had to be extended to benbdaware.
I don't see any prohibitive reason against doing so (provided the net is up, of course)
... beyond the fact that somebody capable has to do it, of course - This and the desire for pre-mount manipulation via ssh led me to try a after-boot root manipulation
nfs and cifs * (aka smb aka samba)
- are other common net file systems I found proposed for root/boot/purposes, but neither is able to provide the extended acl features required for overlay, nor the simplicity of sshfs, so I tried neither of them
Current implementation of root-fs in openWRT as of 23.05.5 seems to be quite non-standard and sturdy against modifications
remountof root, orpivot_root(callable inbusybox) is not possible, just does not do anything or yields instable system- its was not possible to mount (neither
--bindnor full devices) something into the tree of overlay - I did not follow the Idea of relocating parts of the installations, since both from own experience and reports in other places this approach is hard to manage
- I could manage to start some demons in a chrooted environment, but not all of them. Although I mirrored all static files, tmp, dev, proc, sys, I still noticed stability issues
I can't say whether those are just minor challenges or insumountable obstacles.
Ideas for a new start
- may be the problems with stability of
nbdandchrootare just minor ones an can be solved - may be with some better understanding of
pivot_rootor similiar mechanism, it were possible to remount root a second time, late, after a system is up - the idea of "PXE-like" netbooting might be implemented by the
TFPT-boot feature ofubootor similiar bootloaders.
In most cases, at least for early tests, this will require configuration and manipulation of pree-boot environment and thus debrick-style setupt with e.g. serial access.
Chain loading
- chain loading would insert skd of two step boot process:
a first "stepstone linux" might act as a bootloader for the final "real thing" - may be this first linux might be some legacy version, easily ftting into even 4/32 devices
- in contrast to uboot or stripped vendor boot loaders, it nevertheless might be a full fledged linux - inlcuding network access and interactive ssh intervention
- the first system could e.g. load a large kernel / initfs from the network
- the final system would start into a clean stage, without any restriction from the previous start helper
- the memomory occupied by the starter can be recovered after boot of the final system
- it were possible to maintain a network connection and mounted network file
albeit only if both kernels are the same version - otherwise, the stepstone linux just hands over boot gear in memory and the final linux establishes networking afresh
kexecseems to be a proven chain loarder for linux
https://en.wikipedia.org/wiki/Kexec- Obviously
kececis or was available for openWRT sometimes ago, but I could not find it in the current package list
There is an easier option to overcome the flash limit, replacing the actual flash chip with a larger one - while this won't get you official OpenWrt support (but neither will your remote rootfs), it is a rather straight forward process for many spi-nor based devices (depending on the bootloader capabilities). But be aware that larger rootfs == more stuff wanting to be run increases the memory pressure even more…
If you'd spend 5% of the energy that went into this endeavour already into researching your local used markets, you probably would have found pretty decent -supported- devices (with plenty of flash/ RAM) for 5-15 bucks a pop. Once upon a time, neither money nor smooth talking would give you devices with 'decent' amounts of flash/ RAM, so people got inventive, these days good devices (including new ones) don't even need to be particularly expensive to meet system requirements comfortably.
Maybe I have blind spot there?
Yes, I dropped two TL-WE450 (one of them bricked) in favour of DAP-X1860.
However, could not find yet a proper replacement for the CPE210 (directed outdoor)
Regarding upgrading flash -
I keep that in my mind if I'd ever go to unbrick the TL WE-450.
But then I had to learn how to flash uboot, too, right?
Life is perpetual learning…
No one said this would be trivial, nor even worth it for one or two devices.
Just easier than trying to fit your harvester-thresher into a DIN EN 13978-1 precast garage.
A flash chip is cheap, a usb2serial adapter is cheap, even a spi-nor writer and soic8 clamp are cheap, in combination all these things are still worth it (at least if you order from overseas). However what is not cheap, is half-decent soldering equipment for fine pitch soldering, minimum being a soldering iron (e.g. TS101, pinecil or similar), at least one fine- and one wide (knife?) soldering tip (e.g. TS-I && TS-K, maybe also TS-J02 or whatever combination you prefer), leaded 1mm electronic solder (Sn60Pb38Cu2 with internal flux core), extra flux (you'll need it), solder wick, isopropyl alcohol (>>90%) for cleaning off the excess flux; hot air soldering iron optional, but helpful. If you go all-in, this costs more than a single new AP - if you take the budget route, it's still in the vicinity of half of a new (mid-range to higher end) router.
Learning curve, priceless.
…and you will have to rebuild your self-patched OpenWrt from source for all eternity, as the flash partitions are hardcoded in the DTS.
Success not guaranteed, but at least it's a realistic target.
More realistic than the four+ threads you've been entertaining about this topic.
if you could get sshfs to a certain point you could always mount a file over it to work around the permission issue, assuming you could meet the prerequisites
it might be reliable where apparently nbd wasn't
eg. mount routerOS.ext2 /
you mean to create a single large block file on sshfs and then an block file system inside this?
Yes, that was my thought bevor testing nbd.
But that does not change the situation that at the point where I can access the gadget by ssh, to test this things interactively, overlay root already seems to be chiseled in stone. That's the primary conundrum in my view.
To either we manage to overcome this by resembling what late extroot does (in /etc/int.d/done, or we dig deeper into the setup process to prepare whatever net root availible early, so it has to be done without interactive access.
Yes, I see, but I think the forum rules ask as to start a new thread if we digged into this, right?
You are right about separating different topics out, to give them the proper attention (and to help others to find guidance). However, all of these issues here are closely related, as indicated by the repetition of presenting the situation you have to get into each of them. For all of them, it's basically always the constraint of 8 MB flash and a very ambitious set of packages to go in despite that limitation - yes, the networking filesystem approach is kind of a tangent here, if it would have been successful.
At the end of the day, things like these are always a judgement call - shades of gray, rather than clear black and white. But without counting, I remember at least 4 (maybe 5-6) largely similar threads within the last ~week, which is getting a bit tiresome for anyone reading through it. Especially if the only real answer here would be. "D'oh, it's not gonna fit mate." Even if you can make it fit now (and I'm not totally discounting that option), the next major release is likely to be another 250 KB to 300 KB larger than 24.10, which will set you back again (just rough rule of thumb, ~100 KB go into apk, around 150 KB into the next LTS kernel (v6.12+) - there will be some gain with the luaectomy of LuCI, but that will be eaten up by the rest of userspace growing a bit as well).
openwrt-24.10 will be the last major release to formally support 8/64 devices, but quite a few of those will already fall through the cracks due to unfortunate flash partitioning choices made be the vendor, wasting space and leaving (significantly) less than the "8 MB -192 KB" you may get under ideal circumstances. Yes, if you do build your own images with adapted expectations you will be able to extend this deadline for another 1-3 releases, but that involves hard -personal- decisions on what to leave out, there is no one-fits-all solution anymore (until you're down to no webinterface, no package manager, no PPP, no firewall/ routing, no DNS/ DHCP{,v6}, beyond which it gets really insane). But the keyword here is more about 'cutting' stuff, rather than 'adding'.
If you are faced with users wanting/ demanding support for their 4/16 devices (yes, that still happens, very regularly), with all the bells and whistles - and a pony (OpenVPN, tailscale, zerotier - each of which add another 0.8-1.0 MB to the tally on their own), it is hard to keep a straight face and keep sympathy…
Keep in mind that support for 4/16 devices formally ended in 2012, after the fact of this no longer being viable for at least 2-3 years already. Around 2018/ 2019 it really became untenable (after the easy things like luci, opkg, ppp, etc. were all long gone), in 2024/ 2025 one can just laugh into their faces, because it's not going to happen, whatever (sensible) you do.
4/32 and 8/32 had a quite similar fate, so some extent 8/32 is even worse than 4/32 (4/32 forces you to keep the image lean, 8/32 appears to give you some slack - until time comes for the next sysupgrade, which has to fit the whole new image into RAM - while retaining enough RAM to actually manage flashing it…). Both of these have had -formally- the writing on the wall since around ~2018 (while the public focus was more on the 4 MB flash aspect, contributors were always adamant that the RAM limit is just as much of a problem here). At this point we are 6 years beyond those clear statements, and 5 years beyond the formal deprecation (19.07.x).
We're on the cusp with 8/64 devices now, in 24.10.x. Beyond this release, it won't be possible to keep them with full functionality - as what would be considered a release image (with webinterface, all basic routing features, WPA3 support, PPPoE support (which is still needed by very many users)). Obviously you can extend this deadline, by making hard decisions for yourself. But there can't be much the project can do for you beyond this point, it's rather a question what you can still put up with - before the things to leave out become untenable for your target use case.
Keep in mind, the world changes around us, IPv6 is now in use by over 50% of the population, WPA3 is slowly becoming a requirement (it is mandatory for 6 GHz operations), new encryption ciphers are becoming the norm, the kernel and everything else around it is constantly growing, etc. pp.
E.g., personally, I draw a hard line at IPv6, netfilter, DHCP/ DNS support and for the device to remain self-sustainable (so networked filesystems would be beyond my personal acceptance, as it involves a pretty serious bootstrapping issue), being able to reboot on its own, retain its configuration and remain sysupgradeable and keep failsafe methods working.
Disclaimer: I'm not an OpenWrt developer, nor speaking for the project here.