I finally had some time to look into this. I started by stopping the dockerd and containerd processes. Looking at the startup processes for these two I found they are called using:
/usr/bin/dockerd --config-file=/tmp/dockerd/daemon.json
containerd --config /var/run/docker/containerd/containerd.toml --log-level warn
The dockerd config file:
// /tmp/dockerd/daemon.json
{ "data-root": "\/opt\/docker\/", "log-level": "warn", "iptables": true }
The containerd config file:
## /var/run/docker/containerd/containerd.toml
version = 0
root = "/opt/docker/containerd/daemon"
state = "/var/run/docker/containerd/daemon"
plugin_dir = ""
disabled_plugins = ["cri"]
oom_score = 0
[grpc]
address = "/var/run/docker/containerd/containerd.sock"
tcp_address = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[ttrpc]
address = ""
uid = 0
gid = 0
[debug]
address = "/var/run/docker/containerd/containerd-debug.sock"
uid = 0
gid = 0
level = "warn"
[metrics]
address = ""
grpc_histogram = false
[cgroup]
path = ""
[plugins]
[plugins.linux]
shim = "containerd-shim"
runtime = "runc"
runtime_root = "/opt/docker/runc"
no_shim = false
shim_debug = false
I started up just the containerd process with the same command and tried pulling an image with ctr (had to use --address flag as the config has a non-standard grpc address:
ctr --address /var/run/docker/containerd/containerd.sock images pull docker.io/pihole/pihole:latest
I got a similar warning as when I pulled the image with docker, but maybe a bit more helpful:
INFO[0132] apply failure, attempting cleanup error="failed to extract layer sha256:a672338d660a6060c60383d3a65ead6c2509a97c9e528d718e1f169fbba8d3a4: write /opt/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs/etc/pihole/gravity.db: no space left on device: unknown" key="extract-787942071-lwRd sha256:78a7561898940c3e5b7ee8c6e0eef589d533b6f50c582140bea9ddbc36ac7d36"
WARN[0132] extraction snapshot removal failed error="write /opt/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db: no space left on device: unknown" key="extract-787942071-lwRd sha256:78a7561898940c3e5b7ee8c6e0eef589d533b6f50c582140bea9ddbc36ac7d36"
ctr: failed to extract layer sha256:a672338d660a6060c60383d3a65ead6c2509a97c9e528d718e1f169fbba8d3a4: write /opt/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs/etc/pihole/gravity.db: no space left on device: unknown
No space left on disk is the key part I noticed. So I modified the containerd.toml config to point to my external drive, rather than /opt/.
version = 0
root = "/mnt/sda1/docker/containerd/daemon"
...
...
[plugins.linux]
shim = "containerd-shim"
runtime = "runc"
runtime_root = "/mnt/sda1/docker/runc"
With these changes the image pull with ctr worked!
I then modified the docker config to point to the same shared folder:
{ "data-root": "\/mnt\/sda1\/docker\/", "log-level": "warn", "iptables": true }
then I stopped containerd and started dockerd using the default command shown above. The image pull worked!
So I guess something is happening with available space on the /opt/ folder location. There was some talk about available RAM, but perhaps this is a way to allow more space for the actual disk usage I think even having more disk space, doesn't mean that dockerd and containerd will use them? I'm not sure why this would have changed with v22 either.