Running QEMU guests on OpenWrt with qemu-bridge-helper

Installing and Using OpenWrt
1

Hey,

for a project I want to host a QEMU guest on a x86 device running OpenWrt (maybe I'll write a more detailed post about what I'm doing later). Previously I often had problems setting up the network between guest and host, but I realized that it's really easy with OpenWrt and qemu-bridge-helper and I wanted to share a quick recipe.
Using that, you can connect the guest to every bridge network of your OpenWrt host (every Interface with option type 'bridge').

You need the following packages on your device: kmod-tun qemu-bridge-helper qemu-x86_64-softmmu. If your hardware supports it, also install kmod-kvm-amd or kmod-kvm-intel for better performance.
If you install these packages via opkg, reboot afterwards.
Now you're ready to run QEMU guests. In my example the guest is a Debian Stretch installation on /dev/sda, but you can use an image as well. Just use a distribution that comes with virtio drivers by default.
The complete command looks like this:

qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 2G \
    -drive file=/dev/sda,cache=none,if=virtio,format=raw \
    -device virtio-net-pci,mac=E2:F2:6A:01:9D:C9,netdev=br0 \
    -netdev bridge,br=br-lan,id=br0

Line 3 creates a virtio net device with a fixed MAC address and is told to use the network backend with id=br0, which gets created in line 4. The network backend is of type bridge and br=br-lan tells it to connect to the bridge network br-lan of the OpenWrt host.
That's it, if your guest uses DHCP it should receive an address by the server running in br-lan.

To automatically start the VM at boot and shutting it down cleanly before a reboot, I created this init script:

#!/bin/sh /etc/rc.common

START=99
STOP=1

start() {
qemu-system-x86_64 -enable-kvm -cpu host -smp 2 -m 2G \
    -drive file=/dev/sda,cache=none,if=virtio,format=raw \
    -device virtio-net-pci,mac=E2:F2:6A:01:9D:C9,netdev=br0 \
    -netdev bridge,br=br-lan,id=br0 \
    -qmp tcp:127.0.0.1:4444,server,nowait \
    -daemonize &> /var/log/qemu.log
}

stop() {
nc localhost 4444 <<QMP 
{ "execute": "qmp_capabilities" }
{ "execute": "system_powerdown" }
QMP

sleep 10s
}

It uses QMP to initiate the VM shutdown.

I didn't run a benchmark yet, but the guest is performing really well so far.

2

iperf3 results between a separate host in my LAN and the VM are fine:

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.10 GBytes   945 Mbits/sec  345             sender
[  4]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec

The host is an Intel Celeron 2955U (2 x 1,4 GHz), so I think native Gbit speed is great for a VM guest. The CPU load is about 70% during the tests.

3

Hi Sebastian, thank you for your work.
Do you think is possible to compile libvirt to remote manage the vms?

4

Well, the OpenWrt build system ist very flexible and in theory, you can build anything with it. Considering that Alpine Linux packages libvirt, it seems to work fine with musl, too (the C library OpenWrt uses).
However, on your average distro, libvirt has quite a lot of runtime dependencies. I don't know if it can be build without them.
That said, I don't know if it's worth the effort to port libvirt to OpenWrt. If you want a system to manage multiple VMs, there are plenty of alternatives like Proxmox VE or oVirt, for example.

5

Sure, i know proxmox, but my dream is to ingrate a lot of new capabilities in my OpenWRT server x86.
I have found this make file about libvirt
http://wiki.clommunity-project.eu/soft:node-software
Already tried to compile with 18.06.2 but it fails...
I think it is a good starting point

1 Like
6

Finally i can compile it correctly.
If someone want to try i can share my makefile

#
# This is free software, licensed under the GNU General Public License v2.
# See /LICENSE for more information.
#

include $(TOPDIR)/rules.mk
 
PKG_NAME:=libvirt
PKG_VERSION:=1.3.5
PKG_RELEASE:=1
 
PKG_BUILD_DIR:=$(BUILD_DIR)/$(PKG_NAME)-$(PKG_VERSION)
PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION).tar.gz
PKG_SOURCE_URL:=https://libvirt.org/sources/
PKG_MD5SUM:=f9dc1e63d559eca50ae0ee798a4c6c6d
PKG_INSTALL:=1
  
include $(INCLUDE_DIR)/package.mk
 
define Package/libvirt
  SECTION:=libs
  CATEGORY:=Libraries
  TITLE:=Libvirt Virtualization API
  URL:=http://libvirt.org
  DEPENDS:= +gnutls-utils +libgnutls +libdevmapper +libxml2 +yajl +libnl +libcurl +libdbus +librpc \
            +libblkid +libavahi +libncursesw +libsasl2 +libavahi-client +libreadline
 
endef
 
define Package/libvirt/description
	Libvirt is an open source API, daemon and management tool for managing platform virtualization.
endef
 
define Build/Configure
	$(call Build/Configure/Default,--with-yajl=yes --with-macvtap=no --with-xen=no --with-xen-inotify=no --with-qemu=yes --with-uml=no --with-openvz=no --with-vmware=no --with-phyp=no --with-xenapi=no --with-libxl=no --with-vbox=no --with-lxc=yes --with-esx=no --with-hyperv=no --with-storage-dir=yes --with-storage-fs=no --disable-nls)
endef
 
define Package/libvirt/install
	$(CP) $(PKG_INSTALL_DIR)/etc $(1)/
	$(CP) $(PKG_INSTALL_DIR)/usr $(1)/
 
	$(INSTALL_DIR) $(1)/etc
	$(INSTALL_DIR) $(1)/etc/init.d
	$(INSTALL_BIN) ./files/etc/init.d/libvirt-init $(1)/etc/init.d/
endef
 
$(eval $(call BuildPackage,libvirt))
2 Likes
7

Does not work for me on 19.07..........

install: cannot stat './files/etc/init.d/libvirt-init': No such file or directory

Keep getting that error message

Thanks

8

Inside the package folder create the folders "files/etc/init.d/"

And create the file libvirt-init

#!/bin/sh /etc/rc.common
 
START=80
 
boot() {
	start
}
 
start() {
	# Create /var/cache/libvirt if not exists
	if [ ! -d  /var/cache/libvirt ]; then
		mkdir -p /var/cache/libvirt/qemu
	fi
 
	# Create /var/lib/libvirt if not exists
	if [ ! -d  /var/lib/libvirt ]; then
		mkdir -p /var/lib/libvirt/boot
		mkdir -p /var/lib/libvirt/dnsmasq
		mkdir -p /var/lib/libvirt/filesystems
		mkdir -p /var/lib/libvirt/images
		mkdir -p /var/lib/libvirt/lockd/files
		mkdir -p /var/lib/libvirt/lxc
		mkdir -p /var/lib/libvirt/network
		mkdir -p /var/lib/libvirt/qemu
		mkdir -p /var/lib/libvirt/uml
	fi
 
	# Create /var/log/libvirt if not exists
	if [ ! -d  /var/log/libvirt ]; then
	        mkdir -p /var/log/libvirt/lxc
        	mkdir -p /var/log/libvirt/qemu
        	mkdir -p /var/log/libvirt/uml
	fi
 
	# Create /var/run/libvirt if not exists
	if [ ! -d  /var/run/libvirt ]; then
	        mkdir -p /var/run/libvirt/lockd
        	mkdir -p /var/run/libvirt/lxc
        	mkdir -p /var/run/libvirt/network
        	mkdir -p /var/run/libvirt/qemu
        	mkdir -p /var/run/libvirt/uml
	fi
 
	# Starting libvirtd daemon
	echo "Starting libvirt daemon..."
	exec /usr/sbin/libvirtd -d
}
 
stop() {
	echo "Stoping libvirt daemon..."
	start-stop-daemon -K -p /var/run/libvirtd.pid
}
9

Trying it now with your suggestions,

Ok, I think I have it working it compiled into my new image. Do we need to upgrade libvirt or can we? I just noticed there are newer versions. Have you already tried them? Thanks for all the great work.

Cheers,

10

Did you have problems connecting virt-manager to libvirt on the router?

Acting like I dont have ssh running