I too had issues with messages like "Failed to connect to ubus". I didn't debug it further, but someone familiar with the OpenWrt boot system might be able to improve on this outcome. I think what's happening is that systemd isn't booting the container in a way compatible with the assumptions/mechanisms in place for OpenWrt.
one issue may be that OpenWrt wants to create device files or similar stuff and it's not allowed in the container?
I'm not really sure
What is your /etc/config/network
What hardware device are you running this on?
what target image are you using? Please post link to download page as well.
What kernel is your host machine?
/etc/config/network
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fdd1:e403:c127::/48'
config interface 'lan'
option type 'bridge'
option ifname 'eth0'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
I'm running it on the raspberry pi 4 with the official image 2020-02-13-raspbian-buster-lite.img, and then installed raspbian-nspawn-64. So the kernel is 64bit one supported offcially. The openwrt root file system is downloaded from openwrt link.
First you need to add option ifname 'eth0, host0'
This will create the bridge between host0 and container. This will then create br-lan. eth0 set to bridge by itself cant make a bridge.
I assume you have --network-veth in your setup?
Why did you choose that image?
Yes, I'm using --network-veth. The complete commmand is sudo systemd-nspawn --machine=openwrt64 --network-interface=eth0 --network-interface=eth1 --network-veth.
Which image do you mean? 2020-02-13-raspbian-buster-lite.img or openwrt-19.07.1-armvirt-64-default-rootfs.tar.gz? No special reason, just want to use official images.
I tried use option ifname 'eth0, host0', but still no luck.
config interface 'loopback'
option ifname 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fdd1:e403:c127::/48'
config interface 'lan'
option type 'bridge'
option ifname 'eth0, host0'
option proto 'static'
option ipaddr '192.168.1.1'
option netmask '255.255.255.0'
option ip6assign '60'
Yes, I'm using
--network-veth. The complete commmand issudo systemd-nspawn --machine=openwrt64 --network-interface=eth0 --network-interface=eth1 --network-veth.
The RPi only has a single eth0 so adding eth1 may or may not cause issues. remove this interface. If you want to create an interface you need, use the virtual network option.
Which image do you mean?
2020-02-13-raspbian-buster-lite.imgoropenwrt-19.07.1-armvirt-64-default-rootfs.tar.gz? No special reason, just want to use official images.
Talking about openwrt, you raspbian image is fine. I used the sysupgrade version for RPi2 and had good experience. RPi3 img kept running into issues so I stayed with RPi2. If you so intend on using a 64bit image, make sure you are modifying the config.txt file on your raspbian boot partition to 64bit kernel.
http://downloads.openwrt.org/releases/19.07.2/targets/brcm2708/bcm2709/
for the br-lan, to better help check things. only launch container this way:
sudo systemd-nspawn --machine=openwrt64 network-veth
I would prefer you use --directory but this should work.
This will simply provide a virtual bridge. modify the bridge interface to have br0, host0. you should then be able to ping the static address. if you can, go ahead and install luci then from browser log in. go to interfaces and see if br-lan is visible.
RPi4 ---> {br0,host0} ----->openwrt . this will be how they talk to one another.
i am running from memory at the moment because i dont have this setup currently.
after checking with the vm (x86) img version. it does not work properly.
Use the RPi image and let me know how it goes.
I have installed raspbian-nspawn-64 so the kernal is 64bit and confirmed in the config.txt.
I'm using a USB NIC so I have eth1 on host but that's not the problem, I still don't have br-lan even with sudo systemd-nspawn --machine=openwrt64 --network-veth, maybe I should try vanilla raspbian 32 image and openwrt32.
I think the issue is that openwrts boot system isn't bringing up some important bit, and so it doesn't initialize the network on the container... hence you don't get br-lan
the key here is getting OpenWrt to boot inside the container properly. then the complaints about ubus would go away and br-lan would get created by openwrt
To add, i am not an expert on openwrt but it appears they dont have a 64bit version for rpi.
Yes, i suggest using 32bit as the container relies on the host os for key parts when booting up. Host =32bit, container=32bit. Keeping the kernel at 32bit wont hurt performance as many say. Its just too small of a device to really benefit much.
I wonder if anyone here got WireGuard working inside container with OpenWRT? I did, and even routing to other WireGuard peers works fine. What I can't get working is 0.0.0.0/0 route via one of WireGuard peers that serve as "VPN Server" in this particular setup.
OpenWRT running inside container can let's say "traceroute openwrt.org" via this "server" peer, but OpenWRT "clients" (that use this OpenWRT container as their default route) can't.
A little bit more information about this setup:
Other wg peers (including OpenWRT on hardware and inside virtual machines) can route traffic via "server" peer just fine. So there is no issues with "server" side configuration.
I tried to set this peer as default gateway instead of using 0.0.0.0/0 route created automatically by WireGuard. This didn't fly too.
lan interface, wireguard interface and firewall configuration is identical to what I use on hardware router running OpenWRT, yet for some reason it behave differently in container.
I thought lack of net.ipv4.ip_forward=1 on host system could cause issues - tried to enable it, but it didn't help.
So I asking if anyone done something similar and got working setup? What else to try?
Hi all,
I got a lot of help from the earlier posts in this thread - so thanks to all of you.
I started a separate thread for a specific problem I'm having running openwrt using nspawn on a Pi4 and was asked about if/how I got nspawn/openwrt working on the pi and perhaps contribute to this thread. I'm not actually sure if it is working properly - others have probably gotten further. There are some strange aspects I'll cover later, but it is definitely starting up, including the luci website, and basic router functionality also appears to be working (besides the problem I describe in the other thread) - so I'll share what I've managed so far. Perhaps others can confirm and/or give me appropriate tests to run.
The other thread I created is No IPV4 using Raspberry pi as router - if you have any suggestions for troubleshooting IPv4 / routing / firewall / iptable vs nftable issues please take a look.
In brief, I'd like to use my PI as a gigabit WAN router and if it really doesn't need too much CPU, then I'll also use it for Kodi and maybe office tasks at the same time (hey - 2 HDMIs - why not). I've been really happy with OpenWRT to date in running our hostel network, so I'm happy to keep using it - but my old TP-Link 1043NDs struggle to reach ~170MBps with SQM on my upgraded gigabit fiber.
For initial basic testing, I decided to dedicate a port on a 1043 and try to get the Pi running as a router. I'm using the pi's built-in eth0 with my existing local VLAN11 (eth0.11) acting as the WAN input to the Pi. The openwrt container should route VLAN11 onto VLAN16 and back onto the same physical cable to the same switch. So the basic topology is: Internet <-> ONT <-> OpenWRT TP-Link 1043nd router <-> a 2nd 1043nd running as a switch/AP <-> Raspberry pi <-> back to the 2nd 1043nd and a test WiFi on a different VLAN.
My steps were:
mkdir /mnt/openwrt
sudo losetup -P -f /home/pi/OpenWRT/openwrt-19.07.4-bcrm2708-bcm2709-rpi-2-ext4-factory.img
sudo mount /dev/loop0p2 /mnt/openwrt
mkdir /mnt/openwrt/boot
sudo mount /dev/loop0p1 /mnt/openwrt/boot
(note: at this point you can edit the config files in /mnt/openwrt/etc/config)
sudo systemd-nspawn --directory=/mnt/openwrt --boot --network-interface=eth0 --network-veth
Be sure to use --boot. Without it, the terminal will display the basic openwrt text splash screen and login prompt - just as if you connected via ssh - but nothing works. With the --boot option, nspawn only displays basic information on how to kill the container, but (almost) everything just worked on the actual ethernet port - it's just a bit harder to test.
I connected a cross-over cable between my laptop and the pi, set my laptop's IP-address as 192.168.1.10, browsed to 192.168.1.1 - and luci was there waiting for me. Note: veth works as well (a bit more on that later) but again you need to configure it in luci (or by editing the various config files) before you can use it.
With luci, I could change the password, do basic configuration, and modify the interface to use DHCP and connect to my existing VLAN. At this point, the connection broke down, I plugged it into my already configured switch, checked the DHCP leases for openwrt, and reconnected to luci on that address - and after that everything gets a bit easier.
For my test, I created a wan and wan6 interface on the pi to connect to the existing vlan 11 subnet which also connects to the internet (eth0.11). I then assigned lan to vlan 16 and added a static IP and DHCP, and added the normal firewall routing rules. On my existing openwrt 1043nd switch, I put VLAN 11 and 16 on the switch port to the Pi, added a test interface and wlan for vlan16 (just static IP with the Pi's IP as the gateway and no DHCP) and then got my laptop onto that wlan for further testing.
So that's where I am now. Basically - everything works except IPv4 beyond the local subnet. So DHCP, IPv6 DNS and the IPv6 internet all work, but I can't ping on IPv4 beyond the local VLAN16 subnet or browse the IPv4 internet - although these all work when I ssh into the pi's openwrt - more on that in the other thread.
A few minor notes:
So basically, I'm not sure if the system is doing a proper boot - obviously the container does a lot of magic, and I'm not well versed in the linux or openwrt boot process.
My next goal is to get ipv4 working and then to install sqm and repeat all the basic configuration I have on my existing router.
I'll post again if I learn anything else I think may be useful or if I can help someone else get this far.
VMWare has announced ESX for ARM. I have no idea if it will run on a RPI4, or if OpenWRT will run under it, but it might be a viable alternative for what you are wanting to do.
It's a bit easier to do it like this in my opinion:
[Exec]
PrivateUsers=0
[Network]
Bridge=br0
One could ask why I propose to clone openwrt container if you can use single openwrt container? In my opinion openwrt container with untouched default settings could be useful for future deployments on other RaspberryPi's or other ARM boards. To do so export it with "machinectl export-tar openwrt openwrt.tar" (you also could use this to backup whole thing) and import with "machinectl import-tar openwrt.tar" (although you still need to create /etc/systemd/nspawn/openwrt.nspawn manually).
I also have to note that if you would like to use WireGuard inside OpenWRT container - you have to put wireguard into /etc/modules on host, to make it available inside container.
Thanks for those suggestions. I may look into the initialization settings - perhaps it can help with the boot process. In the end - I was able to get OpenWRT running - as far as I can tell including the firewall and ipv4 comes up. I haven't had much time to work on it in the last month but the key point was to run OpenWRT on an iptables OS - I used UbuntuMATE instead of Raspberry OS. The other issue was getting the firewall settings to come up and persist - I'm still not sure on the cause but logging into LuCi and clicking on Status->Firewall would trigger the processing of the firewall configuration after which I could save it using iptables-save and then use iptables-restore during the boot process. So now I have a Pi that boots into MATE with OpenWRT running in a container and the firewall starts up. Pretty happy overall. Now to work on Kodi.
More at:
https://forum.openwrt.org/t/no-ipv4-using-raspberry-pi-as-router/80730/31
Thanks to all forum members.
It's about 2 years later... My how time flies... And OpenWrt now has nftables based firewall. Has anyone tried systemd-nspawn of nftables based OpenWrt from Debian or another nftables host?
I've got a system running Debian installed at a remote location and want to run routing in a container. I'm waffling between debootstrap of a machine with simple nftables firewall and systemd-networkd setup, or putting OpenWrt in a container and having a more router oriented OS with Luci.
Not exactly what you're asking, but for my core router I run OpenWRT as an LXD container inside Alpine Linux 3.17 on x86_64. Painless to set up and nftables works fine out of the box. So far the only stock thing that doesn't work is the "Connections" tab in luci-statistics, because CONFIG_NF_CONNTRACK_PROCFS isn't configured in the kernel by default in most general distributions these days.
Can anyone share the steps to setup openwrt 22 in a container?
I'm using the rootfs.tar.gz and upon booting there's that failsafe issue and dbus/uhttpd not started issue.
$ sudo systemd-nspawn -bD openwrt
Spawning container openwrt on /mnt/ramdrive/openwrt.
Press Ctrl-] three times within 1s to kill container.
Failed to resize receive buffer: Operation not permitted
Press the [f] key and hit [enter] to enter failsafe mode
Press the [1], [2], [3] or [4] key and hit [enter] to select the debug level
f
- failsafe -
Generating 1024 bit rsa key, this may take a while...
Public key portion is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQCmGIdChOrozvBFRwUqYoFrwAKDDVhG2vPcCsjlsrgHhy0tnmzyqdKc6rKD6gKhyPkExQtKjTjclGhBp63wYU6NTYbSOxz6BdsTIxr8bMPH/lhj4KCFbsxeSnZWbT6Ir9E0yoq2U50HwVPljcJ1S7tp+WMThEuJxP8btfMFlTIE1w== root@openwrt
Fingerprint: SHA256:o+qPEAQYeXDImigxDiYDFhMoI3myJFtb+Cpy3PGaZn4
BusyBox v1.35.0 (2023-01-03 00:24:21 UTC) built-in shell (ash)
_______ ________ __
| |.-----.-----.-----.| | | |.----.| |_
| - || _ | -__| || | | || _|| _|
|_______|| __|_____|__|__||________||__| |____|
|__| W I R E L E S S F R E E D O M
-----------------------------------------------------
OpenWrt 22.03.3, r20028-43d71ad93e
-----------------------------------------------------
================= FAILSAFE MODE active ================
special commands:
* firstboot reset settings to factory defaults
* mount_root mount root-partition with config files
after mount_root:
* passwd change root's password
* /etc/config directory with config files
for more help see:
https://openwrt.org/docs/guide-user/troubleshooting/
- failsafe_and_factory_reset
- root_password_reset
=======================================================
root@openwrt:/# ps
PID USER VSZ STAT COMMAND
1 root 968 S /sbin/init
3 root 1040 S /sbin/procd -h /etc/hotplug-preinit.json
4 root 1508 S /bin/sh /etc/preinit
52 root 1124 S lock /tmp/.failsafe
60 root 940 S dropbear -r /tmp/dropbear_failsafe_host_key
66 root 1508 S /bin/sh /etc/preinit
67 root 1124 S lock -w /tmp/.failsafe
68 root 1136 S ash --login
78 root 1132 R ps
root@openwrt:/#
I tested to use LXC to run OpenWrt which works, but with some limitations (probably setting issue which I haven't figured out)
Why not run with LXC/Docker?