Using Systemd to run Openwrt on Raspberry Pi 4...welcome to join

Good news is it is working.
I am able to create the container and kill and recreate.
Now just need to figure out some more configurations. I know you mentioned a bridge setup on host but I honestly dont know how to do that yet.
I have been reading through the freedesktop.org pages on systems-nspawn. I wish the documentation was written better. can hardly understand what they mean in description because it is so convoluted and confusing.

I also need to figure out how to get the machine to stay in the system after a reboot.
Thanks again for directing me to this route, Wish there was as much on this as there was on docker when i searched. have to omit docker in all my searches. ha. .

I can probably make some netdev and network files to set up the bridge. getting the container to boot automatically should be relatively easy. unpack the disk images into /var/lib/machines/openwrt you will need to create a .nspawn file somewhere in /etc/systemd you'll need to read the man pages, that file will describe how to setup the network and soforth for the container, and then machinectl enable openwrt to make it boot at system boot.

not too complicated, but yes requires wading into the docs a little.

I do wonder if you'll have issues with the firewall. Debian/raspbian have moved to nftables. Probably your best bet is to use nftables in OpenWrt. but it won't be a thing you can do in Luci.

It appears I have the option to switch back to iptables according to wiki.
https://wiki.debian.org/nftables

What would you saw is the most stable openWRT right now?
Currently the img i am using has 18.06.7 on it. It is the GoldenOrb for Rpi2.


-sh: can't access tty; job control turned off
  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 18.06.7, r7976-ca47026b7d
 -----------------------------------------------------

In the code above, it says cant access tty, is this something to address or will it be alright?
so img goes in /var/lib/machine/{machinename}
boot up config goes in /etc/systemd/nspawn/{machinename}.nspawn
the .nspawn file is what essentially replacing running systemd-nspawn with arguement and options, correct?
So I would just keep modifiying the .nspawn file until the network, exec, files settings are input correctly?
if these 2 are correct, then i believe i am starting to get the hang of it. just need to understand some of the options better. the way they are written can be a little confusing. also jumping back and forth from .nspawn page to -nspawn back can be a pain. wish the (see page x had a shortcut to the place on the page.
This is really cool :slight_smile:

Awesome, I am going to give this a go.

Actually putting some finishing touches on the setup.
Will be creating a wiki and submitting to openwrt webstie.

Getting closer to the end that was further than I first thought.
link on drivers and binding directories.
https://forum.manjaro.org/t/how-to-install-video-drivers-kernels-and-grub-while-being-chrooted-using-systemd-nspawn/6755/22

For everyone who is interested:
Where this project is at this time.
pull latest compatible version of openwrt ( 32 bit or 64 bit)

credit to sakaka in raspberry forum

You can easily switch your Raspbian host to use the 64-bit (official) kernel; after doing so ,your existing (32-bit) Raspbian userland will continue to work fine, but you should then also be able to chroot/nspawn into aarch64 userlands too

To do so, on your host system edit the file /boot/config.txt, and add to the end:

Code: Select all

[all]
arm_64bit=1

now you can run x86_64 img
If this is done while in host, reboot to take affect.

run apt-get update & apt-get install systemd-container

if working in linux us wget followed by download link.
you need to mount these guys using:

pi@raspberrypi:~ $ gzip openwrt_latest 
Archive:  openwrt_latest
  inflating: latest-version-openwrt.img 
pi@raspberrypi:~ $ sudo losetup --show -P -f {latest-version-openwrt.img}
/dev/loop0 
pi@raspberrypi:~ $ ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p2

```pi@raspberrypi:~ $ sudo mkdir -p /mnt/openwrt
pi@raspberrypi:~ $ sudo mount -v /dev/loop0p2 /mnt/openwrt
mount: /dev/loop0p2 mounted on /mnt/openwrt
pi@raspberrypi:~ $ sudo mkdir /mnt/openwrt/boot
pi@raspberrypi:~ $ sudo mount -v /dev/loop0p1 /mnt/openwrt/boot
mount: /dev/loop0p1 mounted on /mnt/openwrt/boot.

now test your container

pi@raspberrypi:~ $ sudo systemd-nspawn --directory=/mnt/openwrt

Now we need to setup container as a service to start on boot up.

copy all the contents from /mnt/openwrt and place in this directory
cp -r /mnt/openwrt /var/lib/machines/ . ---dont forget the -r ------

For network config you need 2 files in /etc/systemd/network/
./Bridge.netdev

[NetDev]

Name=br0
Kind=bridge

./bridge.network

[Match]
Name=br0

[Network]
DNS=192.168.1.1
Address=192.168.1.2/24
Gateway=192.168.1.1

The ip addresses can be whatever, this is what machinectl will see and show in machinectl -a
Now setup your config file:

/etc/systemd/nspawn/openwrt.nspawn

[Exec]
PrivateUsers=yes

[Files]

[Network]
Private=yes
VirtualEthernet=yes
Bridge=br0

When you are ready to deploy you need to add Interface=eth0 to the above config file
Here I made sure the Bridge was assigned in .nspawn conf of the single container.

Now start it up: machinectl start openwrt

you should be able to SSH into the ip 192.168.1.1 or whatever you put in the .network config file
if you can not, then manual startup the container
first kill the one running
machinectl kill openwrt give it about 5-10 sec to shutdown check with machinectl -a
now run systemd-nspawn --directory=/var/lib/machines/openwrt

Once in the container on openwrt add bridge for interfaces.
vi /etc/config/network
go to line with option ifname then add 'eth0 host0' save [ Esc / : / wq / Enter]
then terminate container session
Hold Ctrl and press ] key rapidly.
If you check ip a you will see that vb-openwrt now has an ip address, 192.168.1.x

Now that is as far as i have gotten so far. Working on trying to bind ports and mount essential "host' directories is still needed. If anyone wants to contribute to testing or knows of some advice, it would be much appreciated.

@dlakelan has been working with me on this and im sure he is wanting to see this fully working.

2 Likes

Testing on debian9 host...( not Pi ) so far pretty easy...

Needed;

sudo brctl addbr brNUM

And spitting ( but working-ish );

Failed to mount n/a on /fs/nspawn/openwrt/sys/fs/selinux (MS_BIND ""): No such file or directory
Failed to mount n/a on /fs/nspawn/openwrt/sys/fs/selinux (MS_RDONLY|MS_NOSUID|MS_NODEV|MS_NOEXEC|MS_REMOUNT|MS_BIND ""): Invalid argument

But I haven't rebooted ( installed rewrote bootfiles ) and nor did I mount "boot" within the rootfs...

Ive never operated on plain Debian so unsure of a few things.

are you running from command line systemd-nspawn or as service machinectl ?
if you are running from command you need to add --network-veth to create a bridge, you can further insure the bridge ID by adding --network-bridge=br0

shouldnt need to use bridge utils

you should be copying the contents of the mounted image to the /var/lib/machines/openwrt directory.

nor did I mount "boot" within the rootfs...

I dont believe its really necessary but it was what was in a tutorial by another contributor so i stuck with it. I was able to start a container without it before but i didnt get as far into things as i am now with it in. I think it has something to do with issuing reboot command or something.

1 Like

@blee Did you succeed to bring up br-lan on the container (openwrt)? I can't see it. Here is the output of ip link.

root@openwrt64:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether dc:a6:32:75:ac:d9 brd ff:ff:ff:ff:ff:ff
3: host0@if7: <BROADCAST,MULTICAST,M-DOWN> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether c2:4e:81:92:26:bc brd ff:ff:ff:ff:ff:ff
4: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:e0:4c:68:00:48 brd ff:ff:ff:ff:ff:ff

When run /etc/init.d/network restart, I got an error:

root@openwrt64:~# /etc/init.d/network restart
/etc/rc.common: /lib/functions/procd.sh: line 54: can't create /var/lock/procd_network.lock: nonexistent directory

After manully make a dir, another error occurred

root@openwrt64:~# /etc/init.d/network restart
Failed to connect to ubus
Failed to connect to ubus
Failed to connect to ubus

I too had issues with messages like "Failed to connect to ubus". I didn't debug it further, but someone familiar with the OpenWrt boot system might be able to improve on this outcome. I think what's happening is that systemd isn't booting the container in a way compatible with the assumptions/mechanisms in place for OpenWrt.

one issue may be that OpenWrt wants to create device files or similar stuff and it's not allowed in the container?

I'm not really sure

What is your /etc/config/network

What hardware device are you running this on?
what target image are you using? Please post link to download page as well.

What kernel is your host machine?

/etc/config/network

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdd1:e403:c127::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

I'm running it on the raspberry pi 4 with the official image 2020-02-13-raspbian-buster-lite.img, and then installed raspbian-nspawn-64. So the kernel is 64bit one supported offcially. The openwrt root file system is downloaded from openwrt link.

First you need to add option ifname 'eth0, host0'
This will create the bridge between host0 and container. This will then create br-lan. eth0 set to bridge by itself cant make a bridge.
I assume you have --network-veth in your setup?
Why did you choose that image?

Yes, I'm using --network-veth. The complete commmand is sudo systemd-nspawn --machine=openwrt64 --network-interface=eth0 --network-interface=eth1 --network-veth.

Which image do you mean? 2020-02-13-raspbian-buster-lite.img or openwrt-19.07.1-armvirt-64-default-rootfs.tar.gz? No special reason, just want to use official images.

I tried use option ifname 'eth0, host0', but still no luck.

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdd1:e403:c127::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0, host0'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

Yes, I'm using --network-veth . The complete commmand is sudo systemd-nspawn --machine=openwrt64 --network-interface=eth0 --network-interface=eth1 --network-veth .

The RPi only has a single eth0 so adding eth1 may or may not cause issues. remove this interface. If you want to create an interface you need, use the virtual network option.

Which image do you mean? 2020-02-13-raspbian-buster-lite.img or openwrt-19.07.1-armvirt-64-default-rootfs.tar.gz ? No special reason, just want to use official images.

Talking about openwrt, you raspbian image is fine. I used the sysupgrade version for RPi2 and had good experience. RPi3 img kept running into issues so I stayed with RPi2. If you so intend on using a 64bit image, make sure you are modifying the config.txt file on your raspbian boot partition to 64bit kernel.
http://downloads.openwrt.org/releases/19.07.2/targets/brcm2708/bcm2709/

for the br-lan, to better help check things. only launch container this way:

sudo systemd-nspawn --machine=openwrt64 network-veth

I would prefer you use --directory but this should work.

This will simply provide a virtual bridge. modify the bridge interface to have br0, host0. you should then be able to ping the static address. if you can, go ahead and install luci then from browser log in. go to interfaces and see if br-lan is visible.

RPi4 ---> {br0,host0} ----->openwrt . this will be how they talk to one another. 

i am running from memory at the moment because i dont have this setup currently.

after checking with the vm (x86) img version. it does not work properly.
Use the RPi image and let me know how it goes.

I have installed raspbian-nspawn-64 so the kernal is 64bit and confirmed in the config.txt.

I'm using a USB NIC so I have eth1 on host but that's not the problem, I still don't have br-lan even with sudo systemd-nspawn --machine=openwrt64 --network-veth, maybe I should try vanilla raspbian 32 image and openwrt32.

I think the issue is that openwrts boot system isn't bringing up some important bit, and so it doesn't initialize the network on the container... hence you don't get br-lan

the key here is getting OpenWrt to boot inside the container properly. then the complaints about ubus would go away and br-lan would get created by openwrt

1 Like

To add, i am not an expert on openwrt but it appears they dont have a 64bit version for rpi.
Yes, i suggest using 32bit as the container relies on the host os for key parts when booting up. Host =32bit, container=32bit. Keeping the kernel at 32bit wont hurt performance as many say. Its just too small of a device to really benefit much.

I wonder if anyone here got WireGuard working inside container with OpenWRT? I did, and even routing to other WireGuard peers works fine. What I can't get working is 0.0.0.0/0 route via one of WireGuard peers that serve as "VPN Server" in this particular setup.

OpenWRT running inside container can let's say "traceroute openwrt.org" via this "server" peer, but OpenWRT "clients" (that use this OpenWRT container as their default route) can't.

A little bit more information about this setup:
Other wg peers (including OpenWRT on hardware and inside virtual machines) can route traffic via "server" peer just fine. So there is no issues with "server" side configuration.
I tried to set this peer as default gateway instead of using 0.0.0.0/0 route created automatically by WireGuard. This didn't fly too.
lan interface, wireguard interface and firewall configuration is identical to what I use on hardware router running OpenWRT, yet for some reason it behave differently in container.
I thought lack of net.ipv4.ip_forward=1 on host system could cause issues - tried to enable it, but it didn't help.

So I asking if anyone done something similar and got working setup? What else to try?