I am trying to use LXC with OpenWrt 18.06.2. I have compiled all the required kernel options and are able to launch after some trial and error containers. However, the containers have no network access! Since LXC-Network is not available, I probably have to create a bridge manually and forward it to WAN. Is there anywhere an introduction on how to do this?
I am working on x86_64 on OpenWRT 18.06.2 but networking is probably similar across plattforms.
/etc/lxc/default.conf
lxc.net.0.type = empty
/etc/lxc/lxc.conf
lxc.lxcpath = /srv/lxc
The config does not contain any networking because I wanted to know how to set it up. Ideally I would like to set-up an masqeraded/independent bridge for all containers and then forward traffic to that bridge as requested. Similar to the set-up on Ubuntu that I have previously used.
Kernel configuration not found at /proc/config.gz; searching...
lxc-checkconfig: unable to retrieve kernel configuration
Try recompiling with IKCONFIG_PROC, installing the kernel headers,
or specifying the kernel configuration path with:
CONFIG=<path> lxc-checkconfig
I was wondering about that because I used the option
PACKAGE_kmod-ikconfig [=m]
So I do not understand why lxc-checkconfig cannot find my kernel configuration. However, when I point lxc-config manually to the .config in buildroot I get a multitude of error messages:
I am really struggeling to understand a lot of these errors, because I have included the options as modules that lxc-checkconfig is complaining about. Here are 2 examples:
I have already included the package in the confirguration CONFIG_PACKAGE_kmod-veth=m
I think that is the correct one. My problem is the configuration of the interface.
You're already using your own buildroot? If so, navigate e.g. to /source/bin/targets/x86/64/packages copy the kmod-veth module via scp to your router and install it directly via opkg.
In general, all required (kernel) modules should be included with 'y' in your build.
OK, thank you. Will change and get back to you. However, quick question: There is normally a difference in compiling Kernel between 'y' and 'm' in that 'm' can be loded at runtime whereas 'y' is included in the kernel even if it is not needed. So making everything 'y' might lead to a bloated kernel. So how does this work with OpenWrt?
VIRTUALBOX HOST
there is an additional setup "if" your LXC-HOST-OS is inside a VIRTUALBOX in that the LAN interface MUST be set to PROMISC-ALL! ( in the VBOX-GUI )
Here is my lxc-checkconfig ( missing | not loaded ) for brevvity.
Thank you so much for your help. Apologies for my delay in getting back. I had to compile and try over and over again but it is now working! I have network access in my containers!
One remaining question is the bridge: Currently the container use the "br-lan" bridge which is also used for the Wifi clients. How can I set-up a seperate bridge with a seperate IP range for the containers and forward traffinc from the WAN to that bridge?
these might be optional but i've put them in my image so;
ip-full
ip-bridge
( in network -> routing maybe )
dnsmasq-full ( in base system )
kmod-tun ( not required for basic )
kmod-gre ( not required for basic )
openvswitch, macvlan.... are kinda cool but leave them for now....
change the bridge interface back to lxcbr0 or whatever it was and see if the container is getting ip's - they should be a new range - you might need lxc-netSOMETHING for that ....... ( yet to try this i've been working manually - you might need to reboot the router to be on the safe side - not mandatory but expedites a few low level things )
let me know how that goes....
Basically like you mention, with the caveat that the system likes m and not y so much.... the kernel is usually a fixed size on most router partitions so like most *nix's these days..... and to not break routers m is used wherever possible ( not required for boot + network access )
For extroots and x86, personally I prefer y..... but you can run into trouble with the build system wanting it to be m and a few other sideways conflicts....
-if you have space and use it all the time use y faster ( or m updatable )
-if you need it for boot use y
-otherwise use m
For anybody that is reading this post and runs into similar problems, here is a summary of the steps that I took:
You need to compile OpenWRT with the required kernel settings. The binary image unfortunately does not have these settings. An introduction can be found here: http://www.gnuton.org/blog/2016/02/lxc-on-openwrt/
The buildroot is very tricky: Most kernel modules (inkl. the kmod-veth) that is needed for network acces are not installed in the image. You need to manually install them with opkg filename form the packages subdirectory of of the target directory. You need to manually install BEFORE opkg update because otherwise OpenWRT will install the version from the server that is not compatible with your kernel.
Use the settings from wulfy23 for cat /etc/lxc/default.conf and cat /srv/lxc/debian-jessie/config
Haha, then I have to compile again. However, I think I only need to recompile for the Kernel Mods. Normal apps (like all the LXC stuff) I can just install with opkg from the openwrt server.