As a maintainer of a few packages I often need to test them across several OpenWrt releases, sometimes repeatedly for the same PR. Testing on the actual hardware becomes challenging and potentially damaging to it (flash wear). Instead, I prefer to test things in VMs using libvirt and QEMU. This comes with its own set of hurdles as OpenWrt images need some massaging to work and have network access, and some targets (malta) need multiple images to work. Considering it's useful to have a blank slate when testing a package, especially with snapshots, I decided to automate this for myself. And over time this has grown into a nice set of tools that I packaged into a single script.
The tool simplifies the process of downloading official OpenWrt images, creating, and configuring VMs for various architectures, and managing their lifecycle. Multiple instances of the same release/target can coexist. Most commands are interactive. Copying files to and from VMs is also supported. Not everything is well-tested so I don't want to oversell it.
Here's a short demo (updated):
The RFC part that I'm looking for is whether anyone else is interested in this, or have some interesting features in mind, so I can polish this up and release on GitHub. Currently this is a set of scripts that has only been tested on Linux, and I'm not even sure of it's possible to port them anywhere else (e.g. macOS).
What are you doing with the virtual networking? More specifically, do the VMs have WAN and LAN interfaces assigned? Does the host have access to the VM through both WAN and LAN?
For some of my test setups, I end up using a physical device on a subnet to get both WAN and LAN working, because networking in qemu has always baffled me (to put it in words fit for use in public).
uci set network.lan=interface
uci set network.lan.device='br-lan'
uci set network.lan.proto='static'
uci set network.lan.ipaddr='$vm_ip'
uci set network.lan.netmask='${netmask:-255.255.255.0}'
uci set network.lan.ip6assign='60'
uci add_list network.lan.dns='$gateway'
uci add network route
uci set network.@route[-1].interface='lan'
uci set network.@route[-1].target='0.0.0.0/0'
uci set network.@route[-1].gateway='$gateway'
The host virtual network is a NAT on a physical interface (the first step in the demo). With this setup the guest is accessible from the host and has working internet access.
Network configuration is optional, so a more advanced or custom setup can be used. Why do you want to separate WAN and LAN? Test some networking tools? I suppose there could be separate WAN and LAN virtual networks and guest interfaces, with LAN not being connected to the outside.
The main goal behind the tool is a quick setup/teardown of VMs with network access to have a clean slate for each package.
The easy/ sensible answer would be host bridges and (virtual) tap interfaces as bridge members. Both systemd-networkd and more traditional networking dæmons like ifupdown allow this easily; qemu itself provides a semi-dynamic variant of this via qemu-bridge-helper, which isn't without its problems (and I would recommend against this). Frontends to kvm like libvirtd tend to provides its own mechanisms beyond that.
I've been using tap bridge slaves for multiple networks for the last >15 years, never caused any troubles (starting from ifupdown, towards networkd now). If necessary, this can also be combined with VLANs, instead of bridging to multiple host network cards.
Yes, technically it's also possible to import complete PCIe devices/ network cards into the VM, but imho that's quite some overkill - at least for lab environments (if you really wanted to use this setup as border gateway, this might make sense).
Is there something something specific I should keep in mind for separate WAN and LAN?
I split the networks on the host side with WAN being forwarded to an interface and LAN being isolated. However, I'm not sure what the expectations are on the guest side. I'm doing this now:
uci set network.wan=interface
uci set network.wan.device='eth0'
uci set network.wan.proto='static'
uci set network.wan.ipaddr='$wan_ip'
uci set network.wan.netmask='${wan_netmask:-255.255.255.0}'
uci add_list network.wan.dns='$wan_gateway'
uci set network.wan6.device='eth0'
uci set network.@device[0].ports='eth1'
uci set network.lan=interface
uci set network.lan.device='br-lan'
uci set network.lan.proto='static'
uci set network.lan.ipaddr='$lan_ip'
uci set network.lan.netmask='${lan_netmask:-255.255.255.0}'
if ! uci show network | grep -q "target='0.0.0.0/0'"; then
uci add network route
uci set network.@route[-1].interface='wan'
uci set network.@route[-1].target='0.0.0.0/0'
uci set network.@route[-1].gateway='$wan_gateway'
fi
uci set dhcp.lan.ignore='1'
I.e. ignoring DHCP for LAN since IP is manually managed. Does that make sense for any use cases you all had in mind?
I guess DHCP on LAN would be nice if it doesn't hurt anything. Also perhaps consider a couple of VLAN's? I'm not sure how to go around connecting actual clients to it or if it's possible at all. If not then yeah, no need in VLAN's. As you probably know, I'm maintaining adblock-lean and that project supports multiple dnsmasq instances, which would normally translate to separate VLANs on the user device. Currently I'm working on an update to enable multiple blocklists which the user could assign to select dnsmasq instances. So if there was an easy way to test that in a virtualized environment, it would be helpful of course.
Yeah, your WrtLab is what I guess I'm looking for in some cases...
Sometimes I want to test setups, say snort or wireguard, where I have "external" hosts/routers talking across an "edge" router to "local" hosts/routers -- the WrtLab target.
Other times, I want to just install a new build to check code changes -- the Vadas solution.
It would be nice to have both in one tool, as my aging brain is always on overload these days.
I haven't tried setting this up, so not sure what that would look like and it's a bit out of scope now. But nothing is preventing any custom network setup either.
How would you envision automating these tests? Particularly from the host side. Since there's a virtual console (read serial) it's possible to automate anything even without a working network connection.
Final touches I feel like it's a bit rough around the edges so I've been procrastinating about making it public for a while.
Looks interesting! And a great name! I didn't know this one existed. I didn't look for any alternatives TBH as this grew organically without any purpose from as set of basic scripts. Reading your use-cases, it looks like my goal is slightly different as I am focusing on quickly testing packages against the official images. I was thinking of adding custom built images, mainly for rootfs size (I need custom VMs to run compiler test-suits), but it's out of scope for now.
That's what I said before my IRL friends pushed me that I do the git push ^^
Thanks, it was the pusher friends idea.
Init release was like one week ago. It's still RC because I'm also not quiet sure what could be missing from a first time users perspective.
I just wanted to see how you do the live cycle management
I'm also still thinking about how to integrate custom image builder builds.
In general I would like to see even more of these tools. Dev, Testing, Learning, Teaching, ... its good when we have different tools with different scopes or functionality, or for different target groups...
I don't fully understand your question but I'll try to answer it anyway. My thinking was that if I had a ready-to-go system with VLANs in a VM and there was a way to access those VLANs then I could configure adblock-lean to use let's say blocklist A on VLAN 1, blocklist B on VLAN2 and simply verify that it's blocking what it's supposed to block. Now thinking about it, being able to access VLANs from the VM is good enough (connecting external clients not required), since each dnsmasq instance binds to a certain network interface and listens on certain IP addresses, so DNS queries to those addresses from the virtual router itself would confirm correct blocking.
As for technicalilies of how those VLANs would be set up, I've never actually set up VLANs, so I honestly simply do not know But I imagine that there is some typical and simple way to set them up and that would be good enough.
I made a POC with letting the VM assign IPs through a LAN interface but haven't taken it much further by defining VLANs (which is not an issue on OpenWrt either way). Seems to be pretty straightforward with libvirt and network manager. So that use-case is very much doable.
I feel like it's a bit rough around the edges so I've been procrastinating about making it public for a while.
But I imagine that there is some typical and simple way to set them up and that would be good enough.
it's