Another option is to run OpenWrt through Linux containers. If you compare network type options provided by Linux Containers (LXC) and Docker many options are similar, however one option seems exclusive for LXC and that is support for the network type "phys".
The "lxc.net.[i].type" setting is used to
specify what kind of network virtualization to be used for the container". And "phys" is described as "phys: an already existing interface specified by the lxc.net.[i].link is assigned to the container. Source
To run OpenWrt in LXC I need it compiled for the ARM architecture.Using the default build instructions, I would have needed something similar to the target "x86\generic target", but there there is no such option for ARM. Closest I can find is a target aimed for "QEMU ARM virt machine".
Once I figure out how to build OpenWrt for the ARM architecture, I hope to get inspiration from the work done over at https://github.com/mikma/lxd-openwrt how to build LXD images from OpenWrt rootfs tarballs
Questions:
Is LXC network type "phys" a good option?
How to compile OpenWrt to fit my needs (ARMv8 64 bit)?
Decided to shut down the "container initiative" when I learned that early work has been done to add support for an (openwrt) target referred to as "mesongx".
However, I can´t risk the mental health of my family by using pretty untested software for providing access to internet. Hence I´ll have to wait until the support matures for the mesongx target.
As for now I´m releasing my Haswell NUC of its media-center duties, swapping it with this Odroid-C2 targeting https://coreelec.org/. Thought about adding an additional USB NIC to the NUC and run a virtual firewall (Opnsense etc..) in ESXI, but you have to resort to unofficial drivers to get USB NICs working in ESXI. Stupid risks. Again. I simply have to force myself to always decide on robust solutions for this crucial piece of functionality such as the internet connection.
I am using an OpenWrt 18.06 lxc container on an Odroid C1 as my internet access router. In front of the Odroid C1 a managed switch is used the combine the incomming stream from my provider and the outgoing streams ( lan and guest network ) and pump it via different VLAN into the Openwrt Container. Working without any problem. The base system on the Odroid C1 is a Debian Stretch System.
I'm glad you posted, because I have a great deal of interest in achieving something similar to you, on Ubuntu 18.04 server running on a raspberry pi 3b+!
Would you mind if I asked some questions about how you set your container up, as I'm relatively new to lxc and keep having issues with the procd process stalling, due to issues with mounting?
I've mostly followed the following tutorial, link, adjusting certain things to suit my requirements, such as the configuration of the interfaces.
I find that the init.sh script provided doesn't work, returning an error message:
mount: mounting devtmpfs on /dev failed: Permission denied
waiting for rest of boot up...
wait...
wait...
wait...
which just goes on until I lxc stop -f container.
When I start the container, I get the following message to stdout on the host's prompt (but not on prompts through ssh), followed by a stream of continuous lines:
/etc/preinit: line 1: can't create /dev/kmsg: Operation not permitted
open: No such file or directory
open: No such file or directory
open: No such file or directory
...
which, again, just goes on until I stop the container.
lxc info --show-log openwrt produces only one line of output:
lxc openwrt 20190512114614.153 WARN conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
My setup uing the raw lxc tools and not the lxd setup from your referenced tuturial
can you show you lxc log file which is created during the lxc-start startup in the container root directory ?
My lxc config file is nothing special
lxc.network.type = veth
lxc.network.mtu = 2000
lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = xx:xx:xx:xx:xx
lxc.network.ipv4 = 0.0.0.0/24
lxc.network.name = lan0
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = ifb0
lxc.network.name = ifb0
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = vlan4
lxc.network.name = guest
lxc.network.type = phys
lxc.network.flags = up
lxc.network.link = vlan2
lxc.network.name = fibre
lxc.rootfs = /var/lib/lxc/base_images/openwrt-18.06.0
lxc.utsname = openwrt
#lxc.hook.pre-start=/var/lib/lxc/openwrt/linkHostModules.sh
#lxc.hook.post-stop=/var/lib/lxc/openwrt/linkHostModules.sh
lxc.hook.mount = /var/lib/lxc/openwrt-chaos_calmer/modules-mount.sh
lxc.tty = 1
lxc.pts = 4
lxc.mount = /var/lib/lxc/openwrt-chaos_calmer/fstab
lxc.arch = armv6l
# Permanently tweaked resource settings
lxc.cgroup.cpu.shares=256
lxc.cgroup.memory.limit_in_bytes = 32M
lxc.cgroup.memory.memsw.limit_in_bytes = 48M
lxc.cgroup.cpuset.cpus = 3
lxc.cgroup.devices.deny = a
# Allow any mknod (but not using the node)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
#lxc.cgroup.devices.allow = c 4:0 rwm
#lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
#fuse
lxc.cgroup.devices.allow = c 10:229 rwm
#tun
lxc.cgroup.devices.allow = c 10:200 rwm
#full
lxc.cgroup.devices.allow = c 1:7 rwm
#hpet
lxc.cgroup.devices.allow = c 10:228 rwm
#ppp
#lxc.cgroup.devices.allow = c 108:0 rwm
#cryptodev
lxc.cgroup.devices.allow = c 10:57 rwm
#rtc
lxc.cgroup.devices.allow = c 252:0 rwm
lxc.start.auto = 0
lxc.start.delay = 10
lxc.group=onboot
lxc.aa_profile = unconfined
the Openwrt image in /var/lib/lxc/base_images/openwrt-18.06.0 directory is a matching, in my case ARM Cortex A5, unpacked image. I using three interfaces fibre, guest and lan ( without bridging )
Sorry for the gap in replying, I had a weird issue where the firmware for the onboard broadcomm wifi chip stopped working, which effectively halted me working any further on this...
Yeah, if anyone else is interested in replicating this setup, I found the underlying cause for the problem, via this forum topic (link).
Apparently, the issue is that the openwrt image - not realising it's in a container - tries to mount /dev/* itself, thus over-mounting the /dev* setup by LXD. stgraber provides the solution, which is simply to unmount the second mount:
umount -l /dev
which fixes everything, no need to use mknod .
Someone has made a build script for building openwrt images suitable for lxc, although it doesn't have support form aarch32 yet. You can checkout the patch he uses that stops procd from trying to mount /dev/* here though (link).
edit: It seems this got picked up by the people maintaining procd, who have made changes to detect when running in a container (link) and modify behaviour accordingly, so hopefully just doing a fresh image build should provide a container-friendly image.
edit2: And trying a freshly compiled image, ls /dev gives