OpenWrt ARM64 QUICK LXC HOWTO/GUIDE (LMS in Debian System in LXC Container)

Usage note:

Running unprivileged LXC containers under OpenWRT 21.02.1 and later; apart from the usual requirements for unprivileged containers (which I won't duplicate here unless someone asks), an issue occurs in at least some images, including OpenWRT and Alpine Linux, that the container will fail to start, reporting a permissions error mounting /proc in the container.

When started in the foreground error will contain the following text:

lxc: Operation not permitted - Failed to mount "proc"

or

Operation not permitted - Failed to mount "proc" onto "/usr/lib/lxc/rootfs/proc"

The solution is to remount both /proc and /sys, (which will also produce the error) in the host, changing the noatime option to relatime. The initial /proc and /sys mount options in the host are not modifiable without building from source, so I found it simplest to do all the work in /etc/rc.local, including starting the containers I want auto-started. (To do the latter as part of the init.d process would cause these things to occur in the wrong order.)

# Change noatime to relatime for /proc and /sys
mount -o remount,rw,nosuid,nodev,noexec,relatime proc /proc
mount -o remount,rw,nodev,noexec,relatime sysfs /sys
...
lxc-start -n your-container-name-here
...

Posted here so it'll come up in a forum search in a related context.
Edited Feb 2024: updated, this problem still exists in 23.05, added the current error message to help this come up in a search.

3 Likes

Do you mind sharing the config? I cannot seem to find OpenWrt specific steps.

What are you having difficulty with? My configuration doesn't contain anything OpenWRT-specific. The thread above does contain OpenWRT-specific details, though. Unprivileged containers started by root are pretty straightforward; I didn't do anything different from what's described in this doc. (Well, I use macvlans instead of veth and I don't create a user bridge, but that's not specific to OpenWRT either and not necessarily relevant to your concerns.)

It was a user error. It is all working beautifully so far!

hello i'm getting this error


lxc-start myLMS 20220927195219.848 ERROR    start - ../src/lxc/start.c:start:2197 - Exec format error - Failed to exec "/sbin/init"
lxc-start myLMS 20220927195219.849 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)
lxc-start myLMS 20220927195219.892 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc-start myLMS 20220927195219.893 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start myLMS 20220927195219.895 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:309 - To get more details, run the container in foreground mode
lxc-start myLMS 20220927195219.896 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start myLMS 20220927195219.899 ERROR    start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "myLMS"

** MY lxc-checkconfig**

LXC version 5.0.1
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/cpu
/sys/fs/cgroup/cpuacct
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/net_cls
/sys/fs/cgroup/pids
/sys/fs/cgroup/rdma
/sys/fs/cgroup/systemd

Cgroup v2 mount points:


Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: missing
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

My /lxc/config/common.conf

# Default configuration shared by all containers

# Setup the LXC devices in /dev/lxc/
lxc.tty.dir = lxc

# Allow for 1024 pseudo terminals
lxc.pty.max = 1024

# Setup 4 tty devices
lxc.tty.max = 4

# Drop some harmful capabilities
lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio

# Ensure hostname is changed on clone
lxc.hook.clone = /usr/share/lxc/hooks/clonehostname

# Default unified cgroup configuration
#
# CGroup allowlist
# lxc.cgroup2.devices.allow = a
## Allow any mknod (but not reading/writing the node)
# lxc.cgroup2.devices.allow = c *:* m
# lxc.cgroup2.devices.allow = b *:* m
## Allow specific devices
### /dev/null
# lxc.cgroup2.devices.allow = c 1:3 rwm
### /dev/zero
# lxc.cgroup2.devices.allow = c 1:5 rwm
### /dev/full
# lxc.cgroup2.devices.allow = c 1:7 rwm
### /dev/tty
# lxc.cgroup2.devices.allow = c 5:0 rwm
### /dev/console
# lxc.cgroup2.devices.allow = c 5:1 rwm
### /dev/ptmx
# lxc.cgroup2.devices.allow = c 5:2 rwm
### /dev/random
# lxc.cgroup2.devices.allow = c 1:8 rwm
### /dev/urandom
# lxc.cgroup2.devices.allow = c 1:9 rwm
### /dev/pts/*
# lxc.cgroup2.devices.allow = c 136:* rwm
### fuse
# lxc.cgroup2.devices.allow = c 10:229 rwm

# Setup the default mounts
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0

# Block some syscalls which are not safe in privileged
# containers
lxc.seccomp.profile = /usr/share/lxc/config/common.seccomp

# Lastly, include all the configs from /usr/share/lxc/config/common.conf.d/
lxc.include = /usr/share/lxc/config/common.conf.d/

First keep up the good work , running LXC on openwrt is just awesome.

I'm still on ver LXC 4.10 on the friendly NanoPI R5s , with 4GB memory and 16 Gb internal SSD.
Hopefully there will be an official OpenWrt release for NanoPI R5s in the future I need to wait for it.

I have had some issue with lxc-snapshot and rsync , my fix was to compile rsync 3.2.6 , but again
I'm not on the real openwrt Release...

You need to run with the suggestion options to get more error messages, I encountered same problem but then using this method I found out that the default bridge interface used by container doesn't exist on OpenWrt, and the lxc-create command won't auto create that bridge which is the cause of issue.

For simplicity I changed the container config to use the OpenWrt default LAN bridge (br-lan) for testing and then it runs!

Is this guide still working with 23.05?

I tried to start an LXC container to use casaOS in openWRT but docker won't run inside the LXC-Container: failed to start daemon: Devices cgroup isn't mounted

it seems the Container still uses cgroups v1 (tmpfs) and openWRT is missing /sys/fs/cgroups/devices

OpenWrt 23.05.0, r23497-6637af95aa on Raspberry Pi 4

My issue seems to look a lot like this one, which should be fixed though: https://github.com/openwrt/packages/issues/13052

I think you need the 'cgroupfs-mount' package.

'cgroup-tools' might also be a good idea, but is probably not need.
'lxc-unprivileged' might also be worth a try.

Im have all of em installed, and LXC works fine-ish for me.

The only thing that doesn't seem to work is "unprivileged container", I couldn't start.

Yeah im struggling with the same with in an unprivileged LXC container and getting podman to run an unprivileged container inside that, but at the moment for me, it seems to come down to not understanding how to map guid and suid properly after they have already been mapped for the unprivileged LXC container.

You mean unprivileged LXC or docker container?

LXC works fine for me ... follow the instructions higher up in the thread and the archlinux wiki for more detailed instructions, the relevant parts are the guid and suid mapping and as stated somewhere above remounting /proc and /sys.

2 Likes

Ok figured that one out ...

You need more uid/gid.
Ok, so that one container has only 65536 which makes it impossible for it to further delegate 65536 to a nested container.

Realised whats going on while reading this tread https://discuss.linuxcontainers.org/t/unable-to-run-rootless-docker-podman-under-a-rootless-lxd-container/15276/3.

Maybe I was too used to setup this with Proxmox (and it already did something to help) so I didn't notice about this mapping thing.

Hi, did you manage to get your LXC running? I encountered the same error (running a privileged Debian bookworm LXC).

I followed the instruction in the wiki and have set lxc.net.0.link = br-lan in /srv/lxc/<container>/config.

My log:

root@OpenWrt:~# lxc-start -n myLMS --logfile ./test.log --logpriority INFO -F
lxc-start: myLMS: ../src/lxc/sync.c: sync_wait: 34 An error occurred in another process (expected sequence number 7)
lxc-start: myLMS: ../src/lxc/start.c: __lxc_start: 2107 Failed to spawn container "myLMS"
lxc-start: myLMS: ../src/lxc/tools/lxc_start.c: main: 306 The container failed to start
lxc-start: myLMS: ../src/lxc/tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

root@OpenWrt:~# cat test.log 
lxc-start myLMS 20240212220309.663 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver nop
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "[all]"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
lxc-start myLMS 20240212220309.664 INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
lxc-start myLMS 20240212220309.664 INFO     start - ../src/lxc/start.c:lxc_init:881 - Container "myLMS" is initialized
lxc-start myLMS 20240212220309.664 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:748 - File exists - Creating the final cgroup 8(lxc.monitor.myLMS) failed
lxc-start myLMS 20240212220309.664 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:808 - File exists - Failed to create monitor cgroup 8(lxc.monitor.myLMS)
lxc-start myLMS 20240212220309.664 INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1391 - The monitor process uses "lxc.monitor.myLMS-1" as cgroup
lxc-start myLMS 20240212220309.664 INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1499 - The container process uses "lxc.payload.myLMS" as inner and "lxc.payload.myLMS" as limit cgroup
lxc-start myLMS 20240212220309.665 INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNS
lxc-start myLMS 20240212220309.665 INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWPID
lxc-start myLMS 20240212220309.665 INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWUTS
lxc-start myLMS 20240212220309.665 INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWIPC
lxc-start myLMS 20240212220309.665 INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNET
lxc-start myLMS 20240212220309.665 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3232 - Invalid argument - Ignoring cgroup2 limits on legacy cgroup system
lxc-start myLMS 20240212220309.665 INFO     start - ../src/lxc/start.c:do_start:1204 - Unshared CLONE_NEWCGROUP
lxc-start myLMS 20240212220309.670 INFO     network - ../src/lxc/network.c:netdev_configure_server_veth:655 - Retrieved mtu 1500 from br-lan
lxc-start myLMS 20240212220309.676 INFO     network - ../src/lxc/network.c:netdev_configure_server_veth:721 - Attached "vethHtgmud" to bridge "br-lan"
lxc-start myLMS 20240212220309.676 INFO     conf - ../src/lxc/conf.c:setup_utsname:876 - Set hostname to "myLMS"
lxc-start myLMS 20240212220309.676 INFO     network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4019 - Finished setting up network devices with caller assigned names
lxc-start myLMS 20240212220309.713 INFO     network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4035 - Finished setting up network devices with kernel assigned names
lxc-start myLMS 20240212220309.713 INFO     conf - ../src/lxc/conf.c:mount_autodev:1220 - Preparing "/dev"
lxc-start myLMS 20240212220309.713 INFO     conf - ../src/lxc/conf.c:mount_autodev:1281 - Prepared "/dev"
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:mount_entry:2436 - No such file or directory - Failed to mount "/sys/fs/fuse/connections" on "/usr/lib/lxc/rootfs/sys/fs/fuse/connections" (optional)
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:lxc_fill_autodev:1318 - Populating "/dev"
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:lxc_fill_autodev:1406 - Populated "/dev"
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:lxc_transient_proc:3804 - Caller's PID is 1; /proc/self points to 1
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:lxc_allocate_ttys:1110 - Finished creating 4 tty devices
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:lxc_setup_ttys:1073 - Finished setting up 4 /dev/tty<N> device(s)
lxc-start myLMS 20240212220309.720 INFO     conf - ../src/lxc/conf.c:setup_personality:1946 - Set personality to "0lx0"
lxc-start myLMS 20240212220309.721 NOTICE   conf - ../src/lxc/conf.c:lxc_setup:4511 - The container "myLMS" is set up
lxc-start myLMS 20240212220309.721 NOTICE   utils - ../src/lxc/utils.c:lxc_drop_groups:1367 - Dropped supplimentary groups
lxc-start myLMS 20240212220309.721 NOTICE   start - ../src/lxc/start.c:start:2194 - Exec'ing "/sbin/init"
lxc-start myLMS 20240212220309.722 ERROR    start - ../src/lxc/start.c:start:2197 - Exec format error - Failed to exec "/sbin/init"
lxc-start myLMS 20240212220309.722 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)
lxc-start myLMS 20240212220309.842 INFO     network - ../src/lxc/network.c:lxc_delete_network_priv:3680 - Removed interface "vethHtgmud" from "br-lan"
lxc-start myLMS 20240212220309.843 ERROR    start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "myLMS"
lxc-start myLMS 20240212220309.843 WARN     start - ../src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 37 for process 3066
lxc-start myLMS 20240212220309.844 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start myLMS 20240212220309.844 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

Turns out I made a silly mistake: I was experimenting with LXC in a x64 OpenWrt VM, and I pulled the arm64 version of Debian LXC image, lol.

I encountered this error immediately afterwards:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

I added lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=1 to the config file of the LXC container. (as documented in the Debain wiki)
Then the LXC would start properly. I am running as root though, so this might not be helpful to rootless LXC users

1 Like

This somehow works for privileged containers (ArchLinux guest in my test) launched by root. It fails to work (still same systemd cgroup error) for unprivileged containers launched by root.

Didn't test non-root users as I won't use this scenario.

Earlier I stumbled on another workaround, uninstall 'cgroupfs-mount', discovered by @twisteroidambassador here: https://github.com/openwrt/packages/issues/22586#issuecomment-2134724686

From my tests, uninstalling 'cgroupfs-mount' works for both privileged and unprivileged containers (test guest ArchLinux for ARM).

I don't know what's going on but will keep it this way for now. Hope this tip helps some folks landing on this thread in future.