I recently got a very capable N150 based amd64 system that seems to be very underwhelmed by what I'm throwing at it, so I'm thinking I'd rather setup Debian and run the OpenWRT rootfs image in a container with host networking.
Is anyone using OpenWRT like that? Any downsides? What are your experiences?
It only makes sense to run OpenWRT in a VM/container for testing purposes or if there is a specific application you want to run (e.g. a VPN server, proxy or ad-blocking service) and prefer a Luci frontend instead of whatever the native OS has to offer.
If you want to do any serious networking and want to rely on OpenWRT for security, then you should run it on bare-metal, so it has direct access to all network interfaces.
It makes more sense to then run docker containers on OpenWRT instead, which is what I do for industrial applications (Database, Grafana, Node-RED, etc.) but you can run whatever and manage it with luci-app-dockerman.
Yes there are valid use cases for OpenWrt container but as already mentioned.
If we are talking about "edge router", between you and the ISP network, then bare metal or a VM is performance wise, maintenance wise and functional wise the best fit.
If you need a special (service) appliance and you have full control then sure choose the correct Kernel versions and compile everything together and use OpenWrt as a container base but this needs to be configured for your host and is hard to scale for a single user. If it is your pet project yeah sure go.
Personally I don't support the idea of running complex projects like grafana on OpenWrt especially on routers or access points.
IMHO these kind of services should sit on a not so limited base to be practical. Or you have to do a lot of leg work. But if that's fine for you then there no reason why not to do so
The mentioned N150 system is not "just" a router or AP and OP wanted to use the hardware up to its capabilities. With sufficient storage and RAM you can do a lot (which many home routers lack).
For example I am currently running a fleet of >500 industrial systems with OpenWRT as base OS running complex setups in Docker containers, managed by Ansible and OpenWISP running on comparable hardware.
When I started development I had to make a choice for the base OS and chose OpenWRT mainly because of security and simplicity. There is no systemd, a proper SDK and image-builder, no systemd, a very large community and did I mention no systemd?
For instance Teltonika (that builds industrial routers and edge gateways) also base their OS on OpenWRT and have built their own remote management system for it. Of course this is very application specific and these systems usually don't route a lot of traffic, but there security matters a lot and OpenWRT allows you to set up your networking base with ease and run applications on top of it.
With regards to running OpenWRT in a VM, I do actually use it on my laptop when I quickly need more advanced "internet sharing" or a TFTP server for serving boot images for embedded hardware.
Just from my own experience (20+ years of Linux systems) a VM has more overhead compared to a container (Docker/Podman doesn't matter) with host networking (there are multiple solutions with different naming).
As for maintenance, bare metal OpenWRT has been far harder to maintain for me than Linux distributions I'm used to, but I agree, that's very subjective.
Please correct me but you want if anything i.e. virtio and kvm.
For actual routers.
Sure I fully agree that OpenWrt as a base systems for complex software is great if you do that full time. Getting payed for it at best. But I would argue that's overkill 9000 for average joe.
I'm also aware that OpenWrt is massively used in the industry and commercial products.
I think we agree that we may to view at those two use case scenarios differently.
Regarding maintenance of bare matel x86 at home.
Don't blame me but I just use the image builder and dd on sda
I also would like to thanks for these great insights and high quality content <3
I don't disagree but I wanted to describe the function.
Like an access point doing access point thingies needs at best send metrics and running monitoring checks but not a software stack of end user facing services.
The same for a router.
If I for myself need to have a service which is not critical for the network to function like a dynamic routing demon and maybe DHCP and name resolver then it should run on a "server". What ever hw that actually is.
KVM, virtio, balloon, etc, have more overhead because you're running an entire kernel. A container is "just" cgroups and some magic. Containers are also way easier to maintain, especially because you're mounting instead of having to deal with disk images.
If we're talking about separating the router from the hardware, I'd trust a VM much more though.
On the other hand, an immutable distribution with containers might not be too far from "secure", depending on your needs.
If you want to run just an application with the least overhead, then a container is the way to go, but OpenWRT most likely not the right OS unless you really prefer the web-interface and the familiarity of the setup.
Otherwise I wouldn't worry about overhead from a VM, it is much more convenient if you have to forward hardware, for instance a Wi-Fi dongle or USB-ethernet adapter for setting up a hotspot on a virtual router/AP. This is what I use on my laptop because it's convenient and familiar, but not as a permanent setup.
In ESXi or Proxmox for instance you can add virtual network adapters, connect them to virtual switches and then use OpenWRT for advanced routing between VM's, VPN's, LAN, ISP, etc. where it would make more sense to run OpenWRT in this way, but it really depends on the application and which has the most focus.
I have the same conundrum with TrueNAS, which is an excellent base OS for a system with redundant storage, with a focus on data management. It also allows you to run VM's and Docker containers, but not as advanced as Proxmox would.
You can also run TrueNAS instead in a VM on Proxmox, you'll get better snapshot support, VM migration, etc. but you'll loose the advanced disk management or make that part more complicated.
I first experimented with OPNsense (after having last used pfSense about ten years ago) but that entire product seems like it's stuck in that era. I gave up and went back to OpenWRT, because it's so much more focused on "being a device behind your stupid ISP" (especially with DS-Lite). So yes, I really prefer it, also for the familiarity.
Sorry, to be the downer among the fangirls, but running OpenWrt inside of a container (lxc/ lxd, docker, etc.) is not a supported configuration, both functionally broken and insecure (the forum search will provide more detailed explanations).