Dear all,
inside my company we are developing an industrial computing platform based on Raspberry Pi Compute Module 4 (CM4). Basically, we are making a motherboard for the CM4 for acquiring signals from certain instruments who requires an embedded computer for sending/receiving commands, performing data acquisition, send data to our web app hosted on a server and so on. In other words, it is an edge computing thing.
As I understand, OpenWrt runs on Raspberry Pi too, but I was wondering if it is worth using an OS which is designed to be used on limited hardware resources (which is usually the case for common routers) but on a high performance hardware.
The reason for adopting OpenWrt against a debian distribution is basically the presence of LUCI. In fact, many aspects of the confiurations of our product would be configuring network (either LAN, Wifi, 4G), using VPNs, reboot on ping, delivering OTA updates, and so on. Also, the integration with OpenWISP would be interesting. The low footprint of OpenWRT to me it is still a PRO, even with good hardware and plenty of RAM and storage.
What concerns me is probably the increased complexity when some software, libraries or components would be needed: on a debian distribution, an "apt install whatever" is usually what you need, while on OpenWrt the packaging system is offering less unless you package yourself what you need.
Sorry for the long post, but I would like to know if someone encountered the same dilemma
howdy... good background and on the point question
fundamentally... you are looking at some deficiencies/complexity when it comes to leveraging modern high level languages and web frameworks. (which on these boards can really narrow down re-use of many pre-existing resources, particularly when it comes to interacting with external components)
the general advice for someone in your position is that a full distro is better... but as you say you have so much on the networking side going on... if you write a clear outline (functional software analysis) of what frameworks/data-sets you intend to leverage/process... then you will know a bit better on which solution may be more effective for you...
another way of looking at it... OpenWrt will likely afford you ultimately a more 'robust' product... where a full distro will give you a more 'agile' one... if your human resources are geared more towards high levels + networking skills lean one way... if your human resources are more adept with lower levels or you NEED robustness, lean the other...
The essential distinction is that it is an embedded system, not continuous-release distro: every upgrade is a full system rewrite. Package updates on trunk are not validated against earlier revisions and are not meant to be updated in place unless you have a serious need, and you should be prepared for incompatibilities if you do.
If that's not a concern, it's otherwise very much worth exploring. You'd be surprised at the breadth of packages available for OpenWRT. It's used for a lot more than routers or even network appliances, and the package repositories reflect that.
I'd suggest as a line of investigation, mapping out your requirements at a high level, doing a few installs (virtual machines are convenient for this), and actually trying out the available packages that you'll need and see if they fit your requirements.
OpenWRT can occasionally be a little bit deficient in areas outside its main focus -- for example, there are extra undocumented steps you have to take enable it to run unprivileged LXC containers-- but overall you may be very surprised at how complete a distro it is.
You're probably better off looking at a "server" distro as you'll run into limitations once you start to import packages that aren't written in C/C++ such as Python, Rust, Go, Perl etc and a lot of time will be spent on packaging whatever your need. You might also want to consider the somewhat "rough" upgrade paths you might experience at times.
Although OpenWRT is rather stripped back that may make it easier rather than harder to handle things as you can get very familiar with what is on there. I think ithas a lot of merit for you if you are mainly creating a device with IO and networking. If you want to put lots of edge "smarts" like trained ML on there it may shift the other way.
Looking at size from the opposite direction, how much of the onboard storage and memory resources would be consumed by a Debian distro? Having plenty "spare" for running debug code, etc on the same platform as production can be very handy sometimes!
A half-way house might be to use a distro like Gentoo where you can still tune exactly what is running, but with ready support for some general features.
Based on what the OP has described, Gentoo would not be the answer - either some derivative of Yocto or OpenWRT would be more appropriate. Both have the advantage that they were designed for embedded systems, including the option of having a read-only rootfs. In general outside of networking-oriented products, Yocto is more common than OpenWRT.
Desktop/server oriented distributions like Debian or even Gentoo have NO place whatsoever on an embedded device, which really needs the ability to have a simple "factory reset" that nukes user data/configurations to revert to a default stored in a rootfs that is write-only except at updates. I work with a product that uses Fedora in an embedded application and it's a gigantic nightmare for our reliability team because it's damn near impossible to factory-reset it, and even though the manufacturer has added additional power-sequencing hardware to try and protect against unclean shutdowns, filesystem corruption that requires sending the damn thing back to manufacturer for a reformat/reinstall happens on a regular basis. Meanwhile we have multiple other systems here that are running Yocto derivatives and we've never had to send a single one of them back to a manufacturer for filesystem corruption, despite the fact that we abuse them by putting them in an application where they will lose power with no warning. (Because this is a design requirement after our experiences with the system that attempted to do power sequencing, which fails when someone powers on your system, drives it across the facility, then turns off the key and unplugs the battery without waiting for shutdown because they forgot their training or didn't realize the vehicle with a Fedora computer running on it needed special training in the first place.)
Note that Android also has the same concept of a write-only /system partition with sane defaults along with a /data partition that allows an easy factory reset by just reformatting /data.
I would not consider the CM4 to be really "high performance hardware". It's higher performance than many routers, but it's still an embedded set-top-box chipset intended for embedded applications that never run a desktop/server-oriented distro like Debian or even Gentoo - usually Yocto unless it's networking-oriented in which case OpenWRT is more suitable.
I find OpenWRT's build system to be a lot easier to work with than Yocto, but Yocto has some benefits for commercial applications including a built-in license manifest functionality.
Note that Pi 4 support on OpenWRT has options for both ext4 and squashfs - ext4 is easier to work with but is potentially more prone to the corruption issues I mention since the rootfs is R/W as opposed to using some sort of overlay scheme to make rootfs appear writeable when it isn't.
synology is another good/interesting vendor who took this approach... they have an initramfs which borders on another full OS to handle disks/recovery et. al. and they still get alota bricks RMA's / support costs due to this approach...
Fedora is supposed to go with Btrfs by default that supports snapshots, so it should be relatively simple to create a snapshot and rollback it on demand.
I would also be a bit vary about using RPi as a platform, I know a lot uses i.MX series for industrial applications simply because they're most higher grade in terms of hardware a there's much better support. I know a few others looking at RK3399 but that's really a different league. Sometimes speed isn't the only thing that matters.
Corruption is however troublesome, often due to the software (setup, choice of fs etc) and drivers which comes down to QA however RPi's seem to be quite notorious at least using microSD cards with this issue ingeneral. If you can go outside the Linux bubble mfsBSD might be a viable alternative or at least a good base although I wouldn't recommend it for RPi hardware...
OpenSUSE has been doing this for years, and they also have MicroOS and other immutable OS images that are updated atomically for server and embeddded use
i.MX is definitely weaker spec-wise but their long-term support is unparalleled
I would definitely NOT ever consider a product using a normal Pi due to the aforementioned MicroSD issue, nor would I trust anything plugged into a standard USB port. CM4 with eMMC is a whole different animal in that regard. (We use Pi 4s for a couple of projects where I work, but those are all internal-only engineering/diagnostic tools, NEVER anything that might get deployed to a customer. Right now everything we have in customer-facing areas from ourselves with the exception of the aforementioned "problem child" is TI AM335x or TI AM57xx, and a third-party i.MX6 product. All of those are running some derivative of Yocto.
worth noting afaik... they introduced a config.txt fallback os option maybe 3-5months ago as I recall... doesn't really help you if partition1 goes south tho... definately would not be placing in harsh or inaccessible locations for mission critical stuff... (most issues can be offset, worked around... but for a frequently rebooted device you could be talking one or more times a year)
In my case, a Pi 4 that I accidentally hit the power switch on at the wrong time during bootup is the ONLY time I've had a SanDisk MicroSD card experience failure (wear leveller went into failsafe mode, aka read-only) - Read-only rootfs isn't a bad thing, but when the entire card goes read-only, say goodbye to settings changes, upgrades, or - well pretty much anything useful for more than a temporary "limp mode".
I've moved nearly all of my Pis to USB boot, but for someone in a harsher environment, this has the problem that even a Cruzer Fit sticks out of a not-really-ruggedized port.
There is less support for OpenWRT than there is for the full fledged Linux versions. OpenWRT is simpler, but still the learning curve is steeper as documentation is missing or outdated and so sending you in the wrong direction. For the quickest development route I'd suggest to go for a 'real' Linux distribution.
Are you planning on future, more powerful, variations of your product? You will benefit a lot from re-using existing code. Then again, use a full fledged distribution.
I think both OpenWRT and the other Linux variants are all susceptible to the same power down problems. We worked around this by having a read-only partition for the OS and application. The log files and configuration data go to a writable partition. In our app the log data isn't critical so we use ext4. You might consider another file system but those will often be slower.
Development time is the most expensive part of your product. Unless you are planning to sell 10k+ units I suggest to choose the platform that gives the least development effort, and that won't be OpenWRT.
This is inherent to any OpenWRT implementation that uses squashfs (which is most device configurations...). It is, by its nature, read-only and thus any "writing" is handled by an overlay filesystem + copy-on-write setup of some form or another.
It's also extremely difficult to achieve with any desktop-oriented distribution because these almost always expect r/w access to the entire filesystem, and removing all of the crud that wants to write somewhere or another will be more time consuming than the learning curve of Yocto, vanilla buildroot, or OpenWRT.
I disagree here. You underestimate the costs of support and your reputation if something winds up unreliable because it was designed poorly. I know of plenty of embedded products that are definitely not selling 10k+ units and are running some variant of Yocto because running a desktop-oriented OS would be a major problem.
