OpenWRT as Host for virtual machines?

Installing and Using OpenWrt
1

Backround/Situation:

I will get an upgrade to my internet uplink in the comming weeks to a 25GBit connection. For this, I am building my own router based on the Minisforum MS-01, which offers plenty of CPU power and comes with 2xSFP+ and 2x2.5GBE ports, and expand it with an Intel E810 2xSFP28 25Gbit PCI-E-card. I have all the hardware at hand and assembled already, so far everything seems to be o.k.

Now for the actual software setup, I do want to use OpenWRT as the router system (the alternative seems to be OPNsense, but I have no experience with that and I used OpenWRT a lot in the past, up to the point where I build my own u-Boot that would boot a custom compiled OpenWRT image over netboot).
Since the machine is quite powerful with a fast uplink and I will most likely have it idling a lot, I do want to also integrate some other features (FTP/HTTP server, possibly tvheadend, maybe a little storage), but I do not want that to interfere with openWRT. So the obvious solution: Virtualisation. I tried around a bit with Proxmox, but found that to be a bit over the top (+ the constant "please buy a license" nagging is a clear no-go for me).

I could of course run OpenWRT as the sole OS on the hardware and try to integrate everything I want into it, but that does not sound like a smart idea for the risk of messing the whole system up when just wanting to adjust some introduced service setting plus some of the software I would want to use is maybe not readily available for OpenWRT.

The more I think about the whole system architecture, the more I wonder if I cannot use OpenWRT directly as a host system that will then host 2-4 virtual machines.

The advantages would be obvious: OpenWRT would have direct access to the hardware and would be in control of everything. It would not rely on an underlying host system to hand through the network cards properly, system stability would be given by the OpenWRT release quality, and I would not have to maintain another system just for the sake of offering virtualisation.
Also, from what I have read, there seem to be some issues with handed through PCI-interfaces in the Linux 6.x-Kernel (KVM DOES NOT WORK AT ALL due to 6.x kernel bug, affects OpenWRT x86/64 snapshots) - so an OpenWRT as the host system would also circumvent that topic and make sure that OpenWRT could provide the best performance possible.

But I could not find any useful information on the internet on whether or not it is possible and/or a good approach to use OpenWRT as the host. What I could find was someone using qemu on OpenWRT to run a VM: Running QEMU guests on OpenWrt with qemu-bridge-helper

Additionally, software support from OpenWRT for some of my setup seems to be a bit limited at the moment, at least the Intel E810-Card does not have drivers integrated yet, though there seems to be a merge request that should fix that: Please add support for Intel E800 network cards -> https://github.com/openwrt/openwrt/pull/17564

So, based on this:

  • Is it possible to use OpenWRT as a host for virtual machines?
    • If so, is it a good idea? Why[?| not?]
    • If so, what would be the recommended approach? Qemu? And is there possibly a LuCI-package that would allow some management of the VMs?
    • If not, what would be the recommended approach?
2

Possible:yes
Good: no, all runs as root, network buffer sizes are tuned for 100Mbit 32MB eol-d router.

If so: luci-ttyd, you can edit domain xml files with busybox vi

If not: debian(ubuntu,fedora,opensuse,best rh clone) qemu and openwrt between virtual bridges.

1 Like
3

Also for a router id suggest i5 ipo i9, routing is dependent on RAM speed, and aesni runs at base frequency, not turbo.

4

I'd avoid using OpenWRT as a host.

For these setups, I routinely use Alpine Linux as the host OS, then QEMU for OpenWRT with supported LAN cards passed directly to the QEMU virtual machine with IOMMU/DMAR/PCI passtrough, and also QEMU for any other OS I need, a XEN-like environment.

If you don't want to use pci passtrough or your NIC is not supported by OpenWRT, you can create a bridge with two ports: one with the physical interface you want to use and one with a TAP device that you can expose as a virtio_net device to the OpenWRT/QEMU. Less ideal but works.

You can also use a simple LXC container with OpenWRT (use latest snapshot for this), works well but it's still quite experimental since you are sharing the kernel part. OpenWRT Images are provided into the default LXC image server. YMMV.

I have some setups where I use OpenWRT as the main virtual router of the LXC containers virtual LAN, and it has served me extremely well.

All made with Alpine Linux as the main host, all done via shell scripts, no gui.

I've done this also with a vanilla Debian distro (Bookworm) too, but you can use the same approach with any distribution you like.

1 Like
5

In another use-case I managed to passtrough a double lan card to an OpenWRT virtual machine running in Hyper-V, in a Windows 10 environment, then use this as the main Virtual Router for the whole LAN.

Quite adventurous but feasible, and worked pretty well too.

6

Thank you for that input. Did not take those points into consideration. I guess with the tuned buffer sizes, openWRT will also not be as fast as it could be when run in a VM?

But then I could also just cut out OpenWRT all together and just do some iptables|nftables stuff between the uplink and the rest of the system.

I have a system with an i9 13900h, which features faster ram speed, more cores, and higher clocks over the i5 option, so I figured this system should have more than enough power to deal with the 25GBit routing. I do not understand how AES-NI comes into play, though.

Very valuable input. I did not work with Alpine Linux yet, only Debian and Manjaro (mostly Manjaro in the last years), so I think best for me would be a Debian host OS. (Also am more used to systemd, and possibly Debian has more options should I ever need to do firmware updates on the ethernet cards, etc.)
In terms of throughput and I guess also reliability, I think it would make most sense to pass through the NIC to the OpenWRT so that there are no more exploitable steps in the uplink chain than absolutely necessary.

This sounds great, could actually be a lot simpler in setup and maintenance.

That just sounds like... but seriously, I would prefer to stay clear of Windows in this kind of application.

Given all this, I somehow can't help but get to the conclusion that I would be better off by just using a Debian (or maybe Alpine Linux) with nftables, dhcpd, and network bridges for the actual routing part and throw in an OpenWRT VM||LXC for managing the WiFi with the integrated WLAN-card and some more VMs for everything else. That would of course not be as easy to manage/remember the right commands than doing it all through OpenWRT, but probably less points of failure. Plus I would probably have to deal with most of those aspects even when setting up the OpenWRT VM as routing/firewall port.

7

On the first note - fq_codel buffer is 16MB everywhere while on OpenWrt it is 4MB...
It can push gigabit, but by then you have to watch for tail-drops.
AES-NI comes in for IPSEC HTTPS OPENVPN...

8

debian base, pass through (SR-IOV) the network adapter to openwrt docker container

This way you only expose OpenWRT for routing and not your debian host

You need to check if the E-810 specific nic model supports SR-IOV though

9

Well, thats a misconception, host has no active elements on a bridge member interface by default.

10

if you pass through the NIC completely you isolate the NIC and everything going through it from the host. This is something different than a bridge that may end up passing things through by configuration or bug to the host.

11

Thats the most pessimal waste of resources. Ever heard of SDN like vxlan? QinQ?

12

I have a similar setup here, i use Proxmox for virtualization and openwrt 64 bits is one of the virtual machines, you can connect the hardware directly using pci pass trhough if you motherboard allows it, or you could attach to proxmox then connect to the virtual machine.
Seems to me that it is near of what you want to.
The openwrt machine is the first to startup and offers dhcp and internet to the rest of the network and also for the other virtual machines.

13

ever heard of security concerns? Pass through is as close as you get to running openwrt on dedicated hardware, you can still crap out QinQ and vxlan configuration to expose more publicly than you want to. The non public port will go to a 25/100g switch uplink anyways but it's completely isolated behind the OpenWRT docker container from whatever you do on the host.

14

I think that proxmox would be usefull in this cenario of sdn and can controll vlans too.

15

well yeah you can do proxmox or even Hyper-V if you want to (with passthrough) but you can configure vxlan and QinQ just as fine in the openwrt docker build, no need to have the host exposed by having a bridge which can bring configuration errors or bugs as an additional layer

16

i think that docker itself is a good tool for development and i use it a lot, but i dont see them as valuable for a high speed networks that a hypervisor will be much more well suited to do this job, in fact maybe docker has some limitations that we might not be aware to handle such high speed network.