Podman cgroup struggles /sys/fs/cgroup/devices missing - need help

Hi, I get the following error when I try to start a container with podman.

I installed Podman according to the wiki

podman run hello-world
Error: crun: open `/sys/fs/cgroup/devices/libpod_parent/libpod-61324e0d1667790eddb9ed97cb3338c7055e9be411c52d7885423111c231d222`: No such file or directory: OCI runtime attempted to invoke a command that was not found
podman run --name blocky --replace -v /srv/container_conf/blocky/config.yaml:/app/config.yml -p 4000:4000 -p 53:53/udp spx01/blocky
Error: crun: open `/sys/fs/cgroup/devices/libpod_parent/libpod-9ae2e00b6dcde8ef9501f282e03fe1d48129ada033b7de45c07aeae1c0a23d3c`: No such file or directory: OCI runtime attempted to invoke a command that was not found

To me, it feels like I'm missing some package or something, or maybe a kernel setting, but it looks to me like all needed cgroup features are enabled.

But I'm somehow missing /sys/fs/cgroup/devices

ls -la /sys/fs/cgroup/
drwxr-xr-x   10 root     root           220 Dec 28 20:00 ./
drwxr-xr-x    7 root     root             0 Dec 28 19:28 ../
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 blkio/
-rw-r--r--    1 root     root             6 Dec 28 20:00 cgroup.subtree_control
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 cpu/
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 cpuacct/
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 cpuset/
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 memory/
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 net_cls/
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 pids/
dr-xr-xr-x    3 root     root             0 Dec 28 19:28 rdma/

output of this check-config.sh

/tmp# ./check-config.sh
info: no config specified, searching for kernel config ...
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: missing
- CONFIG_CGROUP_FREEZER: missing
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: missing
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing

To the state of my system.
I did build a minimal 23.05.2 release following this guide Build official release with minimal kernel fix (kernel vermagic mismatch) - #3 by julianjm to include and remove some patches for the nanopi R4S.
To be specific, I followed walmartshopper's suggestions from here NanoPi R4S-RK3399 is a great new OpenWrt device - #730 by walmartshopper but decided to stick with the open source NIC/LAN driver.

Then I used the resulting image builder to create my actual image with the desired packages.
The currently running one was created with:

make image \
         PROFILE="friendlyarm_nanopi-r4s" \
         PACKAGES=" luci-mod-dashboard nano-full fish zram-swap kmod-lib-zstd  bash htop luci-app-statistics luci-app-sqm bottom tsping luci luci-ssl luci-app-firewall luci-app-opkg luci liblua libubus libubus-lua libuci-lua lua luci-base luci-lib-ip luci-lib-jsonc luci-lib-nixio luci-mod-admin-full luci-theme-bootstrap rpcd uhttpd luci-proto-wireguard luci-proto-ipv6 luci-mod-system luci-app-acme luci-app-dawn luci-app-ddns luci-app-hd-idle luci-app-ksmbd luci-app-upnp luci-mod-admin-full rsync f2fs-tools xz tar gnupg kmod-ikconfig kmod-veth block-mount luci-app-dockerman conmon crun catatonit netavark podman external-protocol kmod-fuse cgroupfs-mount cgroup-tools" \
         FILES="" \
         DISABLED_SERVICES="" \
         ROOTFS_PARTSIZE="1004"

What I'm actually trying to achieve ...

I want openwrt with nftables, preferably working repos and cake-autorate.
+
Running a few containers, some will move to a home server in the future, some will stay.
At the moment, I mainly want blocky and a Grafana instance in some sort of container that can auto update by pulling a new image.

It's been a 3 weeks journey so far, just because I wanted to run the cpu at the advertised clock speed and a fast sd card ...

If anybody has suggestions for a different OS that makes a decent router and can run a few containers ... Im kinda frustrated, so open to suggestions.

Or if the general vibe is bro just use docker and ditch nftables, let me know to.

I guess I'll have to recompile the kernel, but does that really mean that podman only works on custom-builds?

Well, time to download the official image and test ...
I'll be back - gotta turn a compost first before it gets too dark tho.

Ok I tried it with the official image and got exactly the same results, which isn't surprising given that I basically was running an official kernel plus some tiny patches.

So I guess podman does not work with the default kernel, is this intended or a bug?

I just flashed a custom kernel with every possible cgroup option enabled, and now podman works.

podman run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/
/tmp/check-config.sh
info: no config specified, searching for kernel config ...
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: missing
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: missing
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: enabled
- CONFIG_SECURITY_APPARMOR: missing

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.