Trouble running containers with podman

Installing and Using OpenWrt
1

Hey everyone.
I recently purched a turris mox as my new router, it works fine so far and i'm happy about it.
However i also want it to perform some basic server tasks (i.E. running a pihole) in containers.
That's were i run into problems.
Docker is not cutting it for me because it does to much network magic meaning that i ended up with a pihole exposed to the entire internet.
So my new plan is to use podman and a dedicated network namespace, but i can't get podman to run containers. I've installed it and crun but when i run
podman run --rm -it ubuntu:rolling
it fails with
Error: container create failed (no logs from conmon): EOF

I'd be grateful for any hints how to fix this.
Some more logs/versions:

DISTRIB_ID='TurrisOS'
DISTRIB_RELEASE='6.4.4'
DISTRIB_REVISION='r16872+128-42374bcee6'
DISTRIB_TARGET='mvebu/cortexa53'
DISTRIB_ARCH='aarch64_cortex-a53'
DISTRIB_DESCRIPTION='TurrisOS 6.4.4 42374bcee6b51b78848c7031900e908d2d7fe74d'
DISTRIB_TAINTS='busybox'
root@turris:~# podman -v
podman version 3.4.4
root@turris:~# uname -r
5.15.135

Debug log:

INFO[0000] podman filtering at log level debug
DEBU[0000] Called run.PersistentPreRunE(podman run --rm --runtime crun --log-level debug -it ubuntu:rolling)
DEBU[0000] Merged system config "/etc/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /tmp/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding graph root "/tmp/lib/containers/storage" with "/var/lib/containers/storage" from database
DEBU[0000] Overriding static dir "/tmp/lib/containers/storage/libpod" with "/var/lib/containers/storage/libpod" from database
DEBU[0000] Overriding volume path "/tmp/lib/containers/storage/volumes" with "/var/lib/containers/storage/volumes" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=tmpfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend none
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] configured OCI runtime uxc initialization failed: no valid executable found for OCI runtime uxc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/crun"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman-bridge.conflist
INFO[0000] Found CNI network cni-podman1 (type=macvlan) at /etc/cni/net.d/cni-podman1.conflist
INFO[0000] Found CNI network lan (type=macvlan) at /etc/cni/net.d/lan.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 7
DEBU[0000] Pulling image ubuntu:rolling (policy: missing)
DEBU[0000] Looking up image "ubuntu:rolling" in local containers storage
DEBU[0000] Trying "ubuntu:rolling" ...
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Trying "docker.io/library/ubuntu:rolling" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] Found image "ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage
DEBU[0000] Found image "ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82)
DEBU[0000] Looking up image "docker.io/library/ubuntu:rolling" in local containers storage
DEBU[0000] Trying "docker.io/library/ubuntu:rolling" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] Found image "docker.io/library/ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage
DEBU[0000] Found image "docker.io/library/ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82)
DEBU[0000] Looking up image "ubuntu:rolling" in local containers storage
DEBU[0000] Trying "ubuntu:rolling" ...
DEBU[0000] Trying "docker.io/library/ubuntu:rolling" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] Found image "ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage
DEBU[0000] Found image "ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82)
DEBU[0000] Inspecting image 3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] Looking up image "ubuntu:rolling" in local containers storage
DEBU[0000] Trying "ubuntu:rolling" ...
DEBU[0000] Trying "docker.io/library/ubuntu:rolling" ...
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] Found image "ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage
DEBU[0000] Found image "ubuntu:rolling" as "docker.io/library/ubuntu:rolling" in local containers storage ([overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82)
DEBU[0000] Inspecting image 3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] Inspecting image 3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82
DEBU[0000] using systemd mode: false
DEBU[0000] Adding exposed ports
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] Allocated lock 7 for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/run/containers/storage:overlay.mountopt=nodev]@3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] exporting opaque data as blob "sha256:3f9cf3a31fbf871322b774402c7c25b800d0e93c222d5a44339f2d3a99dede82"
DEBU[0000] created container "6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828"
DEBU[0000] container "6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828" has work directory "/var/lib/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata"
DEBU[0000] container "6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828" has run directory "/run/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata"
DEBU[0000] Handling terminal attach
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] backingFs=tmpfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Cached value indicated that volatile is being used
DEBU[0000] overlay: mount_data=nodev,lowerdir=/var/lib/containers/storage/overlay/l/XVPUQJM6BARDLHQMLLG7NESBOI,upperdir=/var/lib/containers/storage/overlay/ef20f93c3abc8bd27411e2c4eae424b626539a69c9512fc84e732360cd1f4abb/diff,workdir=/var/lib/containers/storage/overlay/ef20f93c3abc8bd27411e2c4eae424b626539a69c9512fc84e732360cd1f4abb/work,volatile
DEBU[0000] Made network namespace at /run/netns/cni-959ccb76-2477-4d1f-ece3-ee2b603c1ec6 for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
INFO[0000] Got pod network &{Name:bold_sanderson Namespace:bold_sanderson ID:6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 NetNS:/run/netns/cni-959ccb76-2477-4d1f-ece3-ee2b603c1ec6 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}
INFO[0000] Adding pod bold_sanderson_bold_sanderson to CNI network "podman" (type=bridge)
DEBU[0000] mounted container "6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828" at "/var/lib/containers/storage/overlay/ef20f93c3abc8bd27411e2c4eae424b626539a69c9512fc84e732360cd1f4abb/merged"
DEBU[0000] Created root filesystem for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 at /tmp/lib/containers/storage/overlay/ef20f93c3abc8bd27411e2c4eae424b626539a69c9512fc84e732360cd1f4abb/merged
DEBU[0000] [0] CNI result: &{0.4.0 [{Name:cni-podman0 Mac:b6:a1:3d:75:d8:3f Sandbox:} {Name:vethdf9731d0 Mac:e6:02:4b:98:83:84 Sandbox:} {Name:eth0 Mac:d6:d1:43:ab:e6:f1 Sandbox:/run/netns/cni-959ccb76-2477-4d1f-ece3-ee2b603c1ec6}] [{Version:4 Interface:0x400020a678 Address:{IP:10.88.0.15 Mask:ffff0000} Gateway:10.88.0.1}] [{Dst:{IP:0.0.0.0 Mask:00000000} GW:<nil>}] {[]  [] []}}
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription
DEBU[0000] Setting CGroup path for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 to /libpod_parent/libpod-6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Workdir "/" resolved to host path "/tmp/lib/containers/storage/overlay/ef20f93c3abc8bd27411e2c4eae424b626539a69c9512fc84e732360cd1f4abb/merged"
DEBU[0000] Created OCI spec for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 at /var/lib/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 -u 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 -r /usr/bin/crun -b /var/lib/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata -p /run/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata/pidfile -n bold_sanderson --exit-dir /run/libpod/exits --full-attach -l k8s-file:/var/lib/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata/ctr.log --log-level debug --syslog -t --conmon-pidfile /run/containers/storage/overlay-containers/6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev --exit-command-arg --events-backend --exit-command-arg none --exit-command-arg --syslog --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828]"
DEBU[0000] Cleaning up container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
DEBU[0000] Tearing down network namespace at /run/netns/cni-959ccb76-2477-4d1f-ece3-ee2b603c1ec6 for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
INFO[0000] Got pod network &{Name:bold_sanderson Namespace:bold_sanderson ID:6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 NetNS:/run/netns/cni-959ccb76-2477-4d1f-ece3-ee2b603c1ec6 Networks:[{Name:podman Ifname:eth0}] RuntimeConfig:map[podman:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}
INFO[0000] Deleting pod bold_sanderson_bold_sanderson from CNI network "podman" (type=bridge)
DEBU[0000] unmounted container "6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828"
DEBU[0001] Removing container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
DEBU[0001] Removing all exec sessions for container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
DEBU[0001] Cleaning up container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828
DEBU[0001] Network is already cleaned up, skipping...
DEBU[0001] Container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 storage is already unmounted, skipping...
DEBU[0001] Container 6fc414d98a435e55c461a15ee9ae9bc5279c223de9f6e00612643434aa785828 storage is already unmounted, skipping...
DEBU[0001] ExitCode msg: "container create failed (no logs from conmon): eof"
Error: container create failed (no logs from conmon): EOF
2

I am not sure if the following still applies but have a look at this answer from @slh

3 Likes
3

here's a great article detailing how to deal with Docker networking under OpenWRT. I know it's not the direct answer to your question, but still may be a solution. I've been running Docker on my OpenWRT with zero problems for about a year now.

https://paul-mackinnon.medium.com/openwrt-raspberry-pi-docker-vlan-project-9cb1db10684c

here's an excerpt of my docker-compose.yml:

networks:
  default:
    driver: macvlan
    driver_opts:
      parent: eth2.70
    ipam:
      config:
        - subnet:   10.70.70.0/24
          gateway:  10.70.70.1

services:
  whoami:
    image: containous/whoami:latest-arm.v8
    container_name: whoami
    restart: always
    networks:
      default:
        ipv4_address: 10.70.70.201
    dns: "10.70.70.1"