Hi all,
I got a lot of help from the earlier posts in this thread - so thanks to all of you.
I started a separate thread for a specific problem I'm having running openwrt using nspawn on a Pi4 and was asked about if/how I got nspawn/openwrt working on the pi and perhaps contribute to this thread. I'm not actually sure if it is working properly - others have probably gotten further. There are some strange aspects I'll cover later, but it is definitely starting up, including the luci website, and basic router functionality also appears to be working (besides the problem I describe in the other thread) - so I'll share what I've managed so far. Perhaps others can confirm and/or give me appropriate tests to run.
The other thread I created is No IPV4 using Raspberry pi as router - if you have any suggestions for troubleshooting IPv4 / routing / firewall / iptable vs nftable issues please take a look.
In brief, I'd like to use my PI as a gigabit WAN router and if it really doesn't need too much CPU, then I'll also use it for Kodi and maybe office tasks at the same time (hey - 2 HDMIs - why not). I've been really happy with OpenWRT to date in running our hostel network, so I'm happy to keep using it - but my old TP-Link 1043NDs struggle to reach ~170MBps with SQM on my upgraded gigabit fiber.
For initial basic testing, I decided to dedicate a port on a 1043 and try to get the Pi running as a router. I'm using the pi's built-in eth0 with my existing local VLAN11 (eth0.11) acting as the WAN input to the Pi. The openwrt container should route VLAN11 onto VLAN16 and back onto the same physical cable to the same switch. So the basic topology is: Internet <-> ONT <-> OpenWRT TP-Link 1043nd router <-> a 2nd 1043nd running as a switch/AP <-> Raspberry pi <-> back to the 2nd 1043nd and a test WiFi on a different VLAN.
My steps were:
- Get the standard RPi4 Rasberry Pi 32-bit Buster image working (a full 64-bit installation should also work if you are willing to go without luci initially and also don't want, say, full Kodi on your pi)
- Install systemd / nspawn.
- Download the 32-bit ext4 Pi2 image (do not use squashfs images - another mistake I made):
http://downloads.openwrt.org/releases/19.07.4/targets/brcm2708/bcm2709/openwrt-19.07.4-brcm2708-bcm2709-rpi-2-ext4-factory.img.gz - Unzip and place the img file on the pi - I used: /home/pi/OpenWRT/openwrt-19.07.4-...
- Create the relevant outer mount point - I used:
mkdir /mnt/openwrt - (I forget if I also had to install losetup.)
- The boot folder needs to get mounted inside the openwrt folder - see @blee 's post (17 on this thread) - basically mount the main partition and create the boot folder, then mount the boot partition:
sudo losetup -P -f /home/pi/OpenWRT/openwrt-19.07.4-bcrm2708-bcm2709-rpi-2-ext4-factory.img
sudo mount /dev/loop0p2 /mnt/openwrt
mkdir /mnt/openwrt/boot
sudo mount /dev/loop0p1 /mnt/openwrt/boot
(note: at this point you can edit the config files in /mnt/openwrt/etc/config)
- Assuming systemd-nspawn is installed and using the folders I describe above, the following line will then launch the openwrt container:
sudo systemd-nspawn --directory=/mnt/openwrt --boot --network-interface=eth0 --network-veth
Be sure to use --boot. Without it, the terminal will display the basic openwrt text splash screen and login prompt - just as if you connected via ssh - but nothing works. With the --boot option, nspawn only displays basic information on how to kill the container, but (almost) everything just worked on the actual ethernet port - it's just a bit harder to test.
-
I connected a cross-over cable between my laptop and the pi, set my laptop's IP-address as 192.168.1.10, browsed to 192.168.1.1 - and luci was there waiting for me. Note: veth works as well (a bit more on that later) but again you need to configure it in luci (or by editing the various config files) before you can use it.
-
With luci, I could change the password, do basic configuration, and modify the interface to use DHCP and connect to my existing VLAN. At this point, the connection broke down, I plugged it into my already configured switch, checked the DHCP leases for openwrt, and reconnected to luci on that address - and after that everything gets a bit easier.
-
For my test, I created a wan and wan6 interface on the pi to connect to the existing vlan 11 subnet which also connects to the internet (eth0.11). I then assigned lan to vlan 16 and added a static IP and DHCP, and added the normal firewall routing rules. On my existing openwrt 1043nd switch, I put VLAN 11 and 16 on the switch port to the Pi, added a test interface and wlan for vlan16 (just static IP with the Pi's IP as the gateway and no DHCP) and then got my laptop onto that wlan for further testing.
So that's where I am now. Basically - everything works except IPv4 beyond the local subnet. So DHCP, IPv6 DNS and the IPv6 internet all work, but I can't ping on IPv4 beyond the local VLAN16 subnet or browse the IPv4 internet - although these all work when I ssh into the pi's openwrt - more on that in the other thread.
A few minor notes:
- The pi has no switch device, so that configuration screen isn't available and adding a VLAN is less obvious. You can either directly edit the /etc/config/network file or use luci as follows: wherever luci displays a device connection drop-down, at the very bottom just type in the full vlan address (e.g. "eth0.16"). Afterwards, luci will display that VLAN option among the others.
- Inside openwrt, the veth virtual adapter appears as "host". I used luci to bridge it to my vlan16 interface. At this point you have a weird reverse host/guest situation where the guest is providing internet to the host - but it works. The little cable-connect icon will appear in the upper right corner of the raspberry and the network will start working as well as it works elsewhere (so no ipv4 yet).
- Almost all configuration changes stick - so they get written back to the image. But I can't change the name of the openwrt instance. I can edit it and it goes into the config file but luci and the ssh login continue to display "openwrt". There is a brief error which is displayed on the website indicating some type of read-only filesystem (but the name is changed in the file - I assume "openwrt" is the default and some error occurs during the initialization process).
- I also can't reboot inside the container. If the container tries to reboot, nspawn exits with a TTY error. That TTY error persists and nspawn fails until I reboot the pi host. If I simply kill the container using ctrl+] 3-times, I can restart it.
So basically, I'm not sure if the system is doing a proper boot - obviously the container does a lot of magic, and I'm not well versed in the linux or openwrt boot process.
My next goal is to get ipv4 working and then to install sqm and repeat all the basic configuration I have on my existing router.
I'll post again if I learn anything else I think may be useful or if I can help someone else get this far.