I am using OpenWrt 22.03.0, r19685-512e76967f on my Pi4. There I have Docker installed and also already used. I was already able to start containers successfully.
Today I wanted to create a new container and after running docker pull I got the following error message.
failed to register layer: operation not supported
I already uninstalled docker, deleted all files and reinstalled it but unfortunately no change.
What can be the reason for this? How can I fix this?
Any ideas? I already reinstalled OpenWrt on my Pi but I still have the same problem. Did anything change in one of the latest version?
-----------------------------------------------------
OpenWrt 22.03.0, r19685-512e76967f
-----------------------------------------------------
root@OpenWrt:~# docker pull mbentley/omada-controller:latest
latest: Pulling from mbentley/omada-controller
f8c66abba3b0: Pull complete
a318825ec34e: Extracting [==================================================>] 3.795kB/3.795kB
2461ad21fbec: Download complete
c3d9801e20fc: Download complete
4f4fb700ef54: Download complete
failed to register layer: operation not supported
I just found out that is has something to do with containerd. I have to start containerd manually and afterwards run dockerd.
Then I am able to pull images. After a restart I have to do the whole thing again and all my containers does not exist anymore.
Maybe it is a general docker problem? Is anybody running docker and OpenWrt running on the latest version?
I'm also having the same issue. Could you walk through the commands you used to get it up and running? I couldn't quite figure the right order of things out.
Having to do it every time a reboot happens seems like a pain. Could the process be added to the startup script?
Just downgraded to OpenWrt 21.02.4. For the next releases I will definitely not update immediately, but wait a bit.
Are youngt2 and me the only ones using Docker or do the others not have any problems with it?
I'm having this issue as well. Just upgraded from version 21, it was working fine there.
I think it may have to do with the lack of RAM. Memory usage is pretty full on my device, it may be causing some issues.
I also tried downloading it with docker save (from a different device), but it wouldn't load properly. When loading it on the router, it would get to a certain point, but then it would fail.
The funny thing is that docker pull hello-world functioned just fine. Based on this post, docker needs an amount of memory equal to the amount used by the image. Usually this isn't a problem, but when using docker on a device with such low memory, there just isn't enough available for it to complete.
I'm going to try and set up swap on an external drive I have, and see if that works. I'll update when I'm done.
Based on other posts I've read about similar issue (non openwrt) it could also be a permission problem. Docker tries to write the file it's extracting and cannot. I'm not really sure how to diagnose or troubleshoot either though.
The ram swap sounds like a good idea to start!
Unfortunately my router Askey RT4230W REV6 doesn't have anything before 22.03 that I could find to roll back to.
Same issue here, used to work, now it doesn't, identical error
Not a memory issue.
I don't think it's a RAM problem either. I use a Pi4 with 4GB of RAM, so there should be more than enough available.
running containerd directly gives me some errors... "failed to load cni during init, please check CRI plugin status before setting up network for pods error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Do you have a config in this location? if so, would you mind sharing it?
edit; Downgrading to 21.02.5 has resolved the issues for me, this must be a bug
I got similar errors, but I thought it had something to do with a plugin that wasn't enabled.
Has anyone had luck with docker on 21.03?
I can confirm the same however im not sure if its miss configurated on my side:
what I did on my mochabin is add my 1tb ext4 m.2 drive to /overlay via block-mount tools perhaps something breaks that permission wise to /opt/? (not sure)
when I try docker compose up -d for the lancache the same error happens failed to register layer: operation not supported
Then I saw in this topic it might be related to containerd so I tried to see what shows up there and I see this error:
ERRO[2022-11-23T13:55:13.829910556Z] failed to load cni during init, please check CRI plugin status before setting up network for pods error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
so it complains about a config which is indeed missing, I'm very new to this but is this a bug ?
I finally had some time to look into this. I started by stopping the dockerd and containerd processes. Looking at the startup processes for these two I found they are called using:
/usr/bin/dockerd --config-file=/tmp/dockerd/daemon.json
containerd --config /var/run/docker/containerd/containerd.toml --log-level warn
The dockerd config file:
// /tmp/dockerd/daemon.json
{ "data-root": "\/opt\/docker\/", "log-level": "warn", "iptables": true }
The containerd config file:
## /var/run/docker/containerd/containerd.toml
version = 0
root = "/opt/docker/containerd/daemon"
state = "/var/run/docker/containerd/daemon"
plugin_dir = ""
disabled_plugins = ["cri"]
oom_score = 0
[grpc]
address = "/var/run/docker/containerd/containerd.sock"
tcp_address = ""
tcp_tls_cert = ""
tcp_tls_key = ""
uid = 0
gid = 0
max_recv_message_size = 16777216
max_send_message_size = 16777216
[ttrpc]
address = ""
uid = 0
gid = 0
[debug]
address = "/var/run/docker/containerd/containerd-debug.sock"
uid = 0
gid = 0
level = "warn"
[metrics]
address = ""
grpc_histogram = false
[cgroup]
path = ""
[plugins]
[plugins.linux]
shim = "containerd-shim"
runtime = "runc"
runtime_root = "/opt/docker/runc"
no_shim = false
shim_debug = false
I started up just the containerd process with the same command and tried pulling an image with ctr (had to use --address flag as the config has a non-standard grpc address:
ctr --address /var/run/docker/containerd/containerd.sock images pull docker.io/pihole/pihole:latest
I got a similar warning as when I pulled the image with docker, but maybe a bit more helpful:
INFO[0132] apply failure, attempting cleanup error="failed to extract layer sha256:a672338d660a6060c60383d3a65ead6c2509a97c9e528d718e1f169fbba8d3a4: write /opt/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs/etc/pihole/gravity.db: no space left on device: unknown" key="extract-787942071-lwRd sha256:78a7561898940c3e5b7ee8c6e0eef589d533b6f50c582140bea9ddbc36ac7d36"
WARN[0132] extraction snapshot removal failed error="write /opt/docker/containerd/daemon/io.containerd.metadata.v1.bolt/meta.db: no space left on device: unknown" key="extract-787942071-lwRd sha256:78a7561898940c3e5b7ee8c6e0eef589d533b6f50c582140bea9ddbc36ac7d36"
ctr: failed to extract layer sha256:a672338d660a6060c60383d3a65ead6c2509a97c9e528d718e1f169fbba8d3a4: write /opt/docker/containerd/daemon/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs/etc/pihole/gravity.db: no space left on device: unknown
No space left on disk is the key part I noticed. So I modified the containerd.toml config to point to my external drive, rather than /opt/.
version = 0
root = "/mnt/sda1/docker/containerd/daemon"
...
...
[plugins.linux]
shim = "containerd-shim"
runtime = "runc"
runtime_root = "/mnt/sda1/docker/runc"
With these changes the image pull with ctr worked!
I then modified the docker config to point to the same shared folder:
{ "data-root": "\/mnt\/sda1\/docker\/", "log-level": "warn", "iptables": true }
then I stopped containerd and started dockerd using the default command shown above. The image pull worked!
So I guess something is happening with available space on the /opt/ folder location. There was some talk about available RAM, but perhaps this is a way to allow more space for the actual disk usage I think even having more disk space, doesn't mean that dockerd and containerd will use them? I'm not sure why this would have changed with v22 either.
By the way I didn't test actually running any containers.
Also those files get reset any time you reboot. I don't know enough about openwrt to make it so that doesn't happen.
its strange for me since I fusioned my 1tb drive to /overlay
I can see /opt/ has 1tb
yet I have the same error, I don't get it why it doesn't like /opt/, I either don't know enough of openwrt 
I met the same problems on OpenWrt 22.03.2 . Now I'v got it.
I have a device /dev/sdb1
I "mount /dev/sdb1 /overlay " to enlarge overlay space.
so when I run docker ,it showed :operation not supported(I guess the error is from the linux, not docker's).
Beacuse overlay is read-only filesystem.
so I re fdisk the /dev/sdb device , I create 2 partitons , one is /dev/sdb1 1G, and another /dev/sdb2 20G
mount /dev/sdb1 /overlay
mount /dev/sdb2 /opt/docker
now I can run docker succesfully!
This is an interesting idea. Might be the best solution as you and @xize has mentioned it seems even mounting a larger overlay does not solve the problem.
Thanks!
Device Boot Start End Sectors Size Id Type
/dev/sda1 2048 2099200 2097153 1G 83 Linux
/dev/sda2 2101248 23072768 20971521 10G 5 Extended
I created two partitions on a usb-stick.
When I try run the following commands:
mount /dev/sda1 /overlay
mount /dev/sda2 /opt/docker
I get the following error.
mount: mounting /dev/sda1 on /overlay failed: Invalid argument
Can anybody please explain how I have to mount them.
Thank you.
Did you follow this guide?
