Using Systemd to run Openwrt on Raspberry Pi 4...welcome to join

Actually putting some finishing touches on the setup.
Will be creating a wiki and submitting to openwrt webstie.

Getting closer to the end that was further than I first thought.
link on drivers and binding directories.

For everyone who is interested:
Where this project is at this time.
pull latest compatible version of openwrt ( 32 bit or 64 bit)

credit to sakaka in raspberry forum

You can easily switch your Raspbian host to use the 64-bit (official) kernel; after doing so ,your existing (32-bit) Raspbian userland will continue to work fine, but you should then also be able to chroot/nspawn into aarch64 userlands too

To do so, on your host system edit the file /boot/config.txt, and add to the end:

Code: Select all

[all]
arm_64bit=1

now you can run x86_64 img
If this is done while in host, reboot to take affect.

run apt-get update & apt-get install systemd-container

if working in linux us wget followed by download link.
you need to mount these guys using:

pi@raspberrypi:~ $ gzip openwrt_latest 
Archive:  openwrt_latest
  inflating: latest-version-openwrt.img 
pi@raspberrypi:~ $ sudo losetup --show -P -f {latest-version-openwrt.img}
/dev/loop0 
pi@raspberrypi:~ $ ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p2

```pi@raspberrypi:~ $ sudo mkdir -p /mnt/openwrt
pi@raspberrypi:~ $ sudo mount -v /dev/loop0p2 /mnt/openwrt
mount: /dev/loop0p2 mounted on /mnt/openwrt
pi@raspberrypi:~ $ sudo mkdir /mnt/openwrt/boot
pi@raspberrypi:~ $ sudo mount -v /dev/loop0p1 /mnt/openwrt/boot
mount: /dev/loop0p1 mounted on /mnt/openwrt/boot.

now test your container

pi@raspberrypi:~ $ sudo systemd-nspawn --directory=/mnt/openwrt

Now we need to setup container as a service to start on boot up.

copy all the contents from /mnt/openwrt and place in this directory
cp -r /mnt/openwrt /var/lib/machines/ . ---dont forget the -r ------

For network config you need 2 files in /etc/systemd/network/
./Bridge.netdev

[NetDev]

Name=br0
Kind=bridge

./bridge.network

[Match]
Name=br0

[Network]
DNS=192.168.1.1
Address=192.168.1.2/24
Gateway=192.168.1.1

The ip addresses can be whatever, this is what machinectl will see and show in machinectl -a
Now setup your config file:

/etc/systemd/nspawn/openwrt.nspawn

[Exec]
PrivateUsers=yes

[Files]

[Network]
Private=yes
VirtualEthernet=yes
Bridge=br0

When you are ready to deploy you need to add Interface=eth0 to the above config file
Here I made sure the Bridge was assigned in .nspawn conf of the single container.

Now start it up: machinectl start openwrt

you should be able to SSH into the ip 192.168.1.1 or whatever you put in the .network config file
if you can not, then manual startup the container
first kill the one running
machinectl kill openwrt give it about 5-10 sec to shutdown check with machinectl -a
now run systemd-nspawn --directory=/var/lib/machines/openwrt

Once in the container on openwrt add bridge for interfaces.
vi /etc/config/network
go to line with option ifname then add 'eth0 host0' save [ Esc / : / wq / Enter]
then terminate container session
Hold Ctrl and press ] key rapidly.
If you check ip a you will see that vb-openwrt now has an ip address, 192.168.1.x

Now that is as far as i have gotten so far. Working on trying to bind ports and mount essential "host' directories is still needed. If anyone wants to contribute to testing or knows of some advice, it would be much appreciated.

@dlakelan has been working with me on this and im sure he is wanting to see this fully working.

2 Likes

Testing on debian9 host...( not Pi ) so far pretty easy...

Needed;

sudo brctl addbr brNUM

And spitting ( but working-ish );

Failed to mount n/a on /fs/nspawn/openwrt/sys/fs/selinux (MS_BIND ""): No such file or directory
Failed to mount n/a on /fs/nspawn/openwrt/sys/fs/selinux (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND ""): Invalid argument

But I haven't rebooted ( installed rewrote bootfiles ) and nor did I mount "boot" within the rootfs...

Ive never operated on plain Debian so unsure of a few things.

are you running from command line systemd-nspawn or as service machinectl ?
if you are running from command you need to add --network-veth to create a bridge, you can further insure the bridge ID by adding --network-bridge=br0

shouldnt need to use bridge utils

you should be copying the contents of the mounted image to the /var/lib/machines/openwrt directory.

nor did I mount "boot" within the rootfs...

I dont believe its really necessary but it was what was in a tutorial by another contributor so i stuck with it. I was able to start a container without it before but i didnt get as far into things as i am now with it in. I think it has something to do with issuing reboot command or something.

1 Like

@blee Did you succeed to bring up br-lan on the container (openwrt)? I can't see it. Here is the output of ip link.

root@openwrt64:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether dc:a6:32:75:ac:d9 brd ff:ff:ff:ff:ff:ff
3: host0@if7: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether c2:4e:81:92:26:bc brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:e0:4c:68:00:48 brd ff:ff:ff:ff:ff:ff

When run /etc/init.d/network restart, I got an error:

root@openwrt64:~# /etc/init.d/network restart
/etc/rc.common: /lib/functions/procd.sh: line 54: can't create /var/lock/procd_network.lock: nonexistent directory

After manully make a dir, another error occurred

root@openwrt64:~# /etc/init.d/network restart
Failed to connect to ubus
Failed to connect to ubus
Failed to connect to ubus

I too had issues with messages like "Failed to connect to ubus". I didn't debug it further, but someone familiar with the OpenWrt boot system might be able to improve on this outcome. I think what's happening is that systemd isn't booting the container in a way compatible with the assumptions/mechanisms in place for OpenWrt.

one issue may be that OpenWrt wants to create device files or similar stuff and it's not allowed in the container?

I'm not really sure

What is your /etc/config/network

What hardware device are you running this on?
what target image are you using? Please post link to download page as well.

What kernel is your host machine?

/etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdd1:e403:c127::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

I'm running it on the raspberry pi 4 with the official image 2020-02-13-raspbian-buster-lite.img, and then installed raspbian-nspawn-64. So the kernel is 64bit one supported offcially. The openwrt root file system is downloaded from openwrt link.

First you need to add option ifname 'eth0, host0'
This will create the bridge between host0 and container. This will then create br-lan. eth0 set to bridge by itself cant make a bridge.
I assume you have --network-veth in your setup?
Why did you choose that image?

Yes, I'm using --network-veth. The complete commmand is sudo systemd-nspawn --machine=openwrt64 --network-interface=eth0 --network-interface=eth1 --network-veth.

Which image do you mean? 2020-02-13-raspbian-buster-lite.img or openwrt-19.07.1-armvirt-64-default-rootfs.tar.gz? No special reason, just want to use official images.

I tried use option ifname 'eth0, host0', but still no luck.

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdd1:e403:c127::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0, host0'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

Yes, I'm using --network-veth . The complete commmand is sudo systemd-nspawn --machine=openwrt64 --network-interface=eth0 --network-interface=eth1 --network-veth .

The RPi only has a single eth0 so adding eth1 may or may not cause issues. remove this interface. If you want to create an interface you need, use the virtual network option.

Which image do you mean? 2020-02-13-raspbian-buster-lite.img or openwrt-19.07.1-armvirt-64-default-rootfs.tar.gz ? No special reason, just want to use official images.

Talking about openwrt, you raspbian image is fine. I used the sysupgrade version for RPi2 and had good experience. RPi3 img kept running into issues so I stayed with RPi2. If you so intend on using a 64bit image, make sure you are modifying the config.txt file on your raspbian boot partition to 64bit kernel.
http://downloads.openwrt.org/releases/19.07.2/targets/brcm2708/bcm2709/

for the br-lan, to better help check things. only launch container this way:

sudo systemd-nspawn --machine=openwrt64 network-veth

I would prefer you use --directory but this should work.

This will simply provide a virtual bridge. modify the bridge interface to have br0, host0. you should then be able to ping the static address. if you can, go ahead and install luci then from browser log in. go to interfaces and see if br-lan is visible.

RPi4 ---> {br0,host0} ----->openwrt . this will be how they talk to one another. 

i am running from memory at the moment because i dont have this setup currently.

after checking with the vm (x86) img version. it does not work properly.
Use the RPi image and let me know how it goes.

I have installed raspbian-nspawn-64 so the kernal is 64bit and confirmed in the config.txt.

I'm using a USB NIC so I have eth1 on host but that's not the problem, I still don't have br-lan even with sudo systemd-nspawn --machine=openwrt64 --network-veth, maybe I should try vanilla raspbian 32 image and openwrt32.

I think the issue is that openwrts boot system isn't bringing up some important bit, and so it doesn't initialize the network on the container... hence you don't get br-lan

the key here is getting OpenWrt to boot inside the container properly. then the complaints about ubus would go away and br-lan would get created by openwrt

1 Like

To add, i am not an expert on openwrt but it appears they dont have a 64bit version for rpi.
Yes, i suggest using 32bit as the container relies on the host os for key parts when booting up. Host =32bit, container=32bit. Keeping the kernel at 32bit wont hurt performance as many say. Its just too small of a device to really benefit much.

I wonder if anyone here got WireGuard working inside container with OpenWRT? I did, and even routing to other WireGuard peers works fine. What I can't get working is 0.0.0.0/0 route via one of WireGuard peers that serve as "VPN Server" in this particular setup.

OpenWRT running inside container can let's say "traceroute openwrt.org" via this "server" peer, but OpenWRT "clients" (that use this OpenWRT container as their default route) can't.

A little bit more information about this setup:
Other wg peers (including OpenWRT on hardware and inside virtual machines) can route traffic via "server" peer just fine. So there is no issues with "server" side configuration.
I tried to set this peer as default gateway instead of using 0.0.0.0/0 route created automatically by WireGuard. This didn't fly too.
lan interface, wireguard interface and firewall configuration is identical to what I use on hardware router running OpenWRT, yet for some reason it behave differently in container.
I thought lack of net.ipv4.ip_forward=1 on host system could cause issues - tried to enable it, but it didn't help.

So I asking if anyone done something similar and got working setup? What else to try?

Hi all,

I got a lot of help from the earlier posts in this thread - so thanks to all of you.

I started a separate thread for a specific problem I'm having running openwrt using nspawn on a Pi4 and was asked about if/how I got nspawn/openwrt working on the pi and perhaps contribute to this thread. I'm not actually sure if it is working properly - others have probably gotten further. There are some strange aspects I'll cover later, but it is definitely starting up, including the luci website, and basic router functionality also appears to be working (besides the problem I describe in the other thread) - so I'll share what I've managed so far. Perhaps others can confirm and/or give me appropriate tests to run.

The other thread I created is No IPV4 using Raspberry pi as router - if you have any suggestions for troubleshooting IPv4 / routing / firewall / iptable vs nftable issues please take a look.

In brief, I'd like to use my PI as a gigabit WAN router and if it really doesn't need too much CPU, then I'll also use it for Kodi and maybe office tasks at the same time (hey - 2 HDMIs - why not). I've been really happy with OpenWRT to date in running our hostel network, so I'm happy to keep using it - but my old TP-Link 1043NDs struggle to reach ~170MBps with SQM on my upgraded gigabit fiber.

For initial basic testing, I decided to dedicate a port on a 1043 and try to get the Pi running as a router. I'm using the pi's built-in eth0 with my existing local VLAN11 (eth0.11) acting as the WAN input to the Pi. The openwrt container should route VLAN11 onto VLAN16 and back onto the same physical cable to the same switch. So the basic topology is: Internet <-> ONT <-> OpenWRT TP-Link 1043nd router <-> a 2nd 1043nd running as a switch/AP <-> Raspberry pi <-> back to the 2nd 1043nd and a test WiFi on a different VLAN.

My steps were:

  1. Get the standard RPi4 Rasberry Pi 32-bit Buster image working (a full 64-bit installation should also work if you are willing to go without luci initially and also don't want, say, full Kodi on your pi)
  2. Install systemd / nspawn.
  3. Download the 32-bit ext4 Pi2 image (do not use squashfs images - another mistake I made):
    http://downloads.openwrt.org/releases/19.07.4/targets/brcm2708/bcm2709/openwrt-19.07.4-brcm2708-bcm2709-rpi-2-ext4-factory.img.gz
  4. Unzip and place the img file on the pi - I used: /home/pi/OpenWRT/openwrt-19.07.4-...
  5. Create the relevant outer mount point - I used: mkdir /mnt/openwrt
  6. (I forget if I also had to install losetup.)
  7. The boot folder needs to get mounted inside the openwrt folder - see @blee 's post (17 on this thread) - basically mount the main partition and create the boot folder, then mount the boot partition:
sudo losetup -P -f /home/pi/OpenWRT/openwrt-19.07.4-bcrm2708-bcm2709-rpi-2-ext4-factory.img
sudo mount /dev/loop0p2 /mnt/openwrt 
mkdir /mnt/openwrt/boot
sudo mount /dev/loop0p1 /mnt/openwrt/boot

(note: at this point you can edit the config files in /mnt/openwrt/etc/config)

  1. Assuming systemd-nspawn is installed and using the folders I describe above, the following line will then launch the openwrt container:
    sudo systemd-nspawn --directory=/mnt/openwrt --boot --network-interface=eth0 --network-veth

Be sure to use --boot. Without it, the terminal will display the basic openwrt text splash screen and login prompt - just as if you connected via ssh - but nothing works. With the --boot option, nspawn only displays basic information on how to kill the container, but (almost) everything just worked on the actual ethernet port - it's just a bit harder to test.

  1. I connected a cross-over cable between my laptop and the pi, set my laptop's IP-address as 192.168.1.10, browsed to 192.168.1.1 - and luci was there waiting for me. Note: veth works as well (a bit more on that later) but again you need to configure it in luci (or by editing the various config files) before you can use it.

  2. With luci, I could change the password, do basic configuration, and modify the interface to use DHCP and connect to my existing VLAN. At this point, the connection broke down, I plugged it into my already configured switch, checked the DHCP leases for openwrt, and reconnected to luci on that address - and after that everything gets a bit easier.

  3. For my test, I created a wan and wan6 interface on the pi to connect to the existing vlan 11 subnet which also connects to the internet (eth0.11). I then assigned lan to vlan 16 and added a static IP and DHCP, and added the normal firewall routing rules. On my existing openwrt 1043nd switch, I put VLAN 11 and 16 on the switch port to the Pi, added a test interface and wlan for vlan16 (just static IP with the Pi's IP as the gateway and no DHCP) and then got my laptop onto that wlan for further testing.

So that's where I am now. Basically - everything works except IPv4 beyond the local subnet. So DHCP, IPv6 DNS and the IPv6 internet all work, but I can't ping on IPv4 beyond the local VLAN16 subnet or browse the IPv4 internet - although these all work when I ssh into the pi's openwrt - more on that in the other thread.

A few minor notes:

  1. The pi has no switch device, so that configuration screen isn't available and adding a VLAN is less obvious. You can either directly edit the /etc/config/network file or use luci as follows: wherever luci displays a device connection drop-down, at the very bottom just type in the full vlan address (e.g. "eth0.16"). Afterwards, luci will display that VLAN option among the others.
  2. Inside openwrt, the veth virtual adapter appears as "host". I used luci to bridge it to my vlan16 interface. At this point you have a weird reverse host/guest situation where the guest is providing internet to the host - but it works. The little cable-connect icon will appear in the upper right corner of the raspberry and the network will start working as well as it works elsewhere (so no ipv4 yet).
  3. Almost all configuration changes stick - so they get written back to the image. But I can't change the name of the openwrt instance. I can edit it and it goes into the config file but luci and the ssh login continue to display "openwrt". There is a brief error which is displayed on the website indicating some type of read-only filesystem (but the name is changed in the file - I assume "openwrt" is the default and some error occurs during the initialization process).
  4. I also can't reboot inside the container. If the container tries to reboot, nspawn exits with a TTY error. That TTY error persists and nspawn fails until I reboot the pi host. If I simply kill the container using ctrl+] 3-times, I can restart it.

So basically, I'm not sure if the system is doing a proper boot - obviously the container does a lot of magic, and I'm not well versed in the linux or openwrt boot process.

My next goal is to get ipv4 working and then to install sqm and repeat all the basic configuration I have on my existing router.

I'll post again if I learn anything else I think may be useful or if I can help someone else get this far.

VMWare has announced ESX for ARM. I have no idea if it will run on a RPI4, or if OpenWRT will run under it, but it might be a viable alternative for what you are wanting to do.

It's a bit easier to do it like this in my opinion:

  1. Configure networkd bridge (I tested this on Armbian; on Ubuntu images you will have to get rid of netplan first) and install systemd-container
  2. Create /etc/systemd/nspawn/openwrt.nspawn with following content:
[Exec]
PrivateUsers=0

[Network]
Bridge=br0
  1. Untar OpenWRT rootfs into /var/lib/machines/openwrt (create openwrt subvolume if you are on btrfs, otherwise just "mkdir openwrt" there).
  2. Change pi_preinit_no_failsafe to "y" in /var/lib/machines/openwrt/lib/preinit/00_preinit.conf and launch container once with "machinectl start openwrt" to generate /etc/config/network inside container.
  3. Replace "eth0" with "host0" (or "host0.X where X is VLAN number, if you need VLAN) in /var/lib/machines/openwrt/etc/config/network and now you can create clones ("machinectl clone openwrt router") or enable autostart on boot with "machinectl enable router" without need to write your own systemd unit.

One could ask why I propose to clone openwrt container if you can use single openwrt container? In my opinion openwrt container with untouched default settings could be useful for future deployments on other RaspberryPi's or other ARM boards. To do so export it with "machinectl export-tar openwrt openwrt.tar" (you also could use this to backup whole thing) and import with "machinectl import-tar openwrt.tar" (although you still need to create /etc/systemd/nspawn/openwrt.nspawn manually).

I also have to note that if you would like to use WireGuard inside OpenWRT container - you have to put wireguard into /etc/modules on host, to make it available inside container.

Thanks for those suggestions. I may look into the initialization settings - perhaps it can help with the boot process. In the end - I was able to get OpenWRT running - as far as I can tell including the firewall and ipv4 comes up. I haven't had much time to work on it in the last month but the key point was to run OpenWRT on an iptables OS - I used UbuntuMATE instead of Raspberry OS. The other issue was getting the firewall settings to come up and persist - I'm still not sure on the cause but logging into LuCi and clicking on Status->Firewall would trigger the processing of the firewall configuration after which I could save it using iptables-save and then use iptables-restore during the boot process. So now I have a Pi that boots into MATE with OpenWRT running in a container and the firewall starts up. Pretty happy overall. Now to work on Kodi.
More at:
https://forum.openwrt.org/t/no-ipv4-using-raspberry-pi-as-router/80730/31

Thanks to all forum members.