Docker engine on OpenWrt

Please feel free to collaborate here

2 Likes

Hello. You don't need glibc or docker to run your images. See podman. The most complete list of required options for kernel are listed in check-config.sh. I haven't tried it on openwrt yet, but you shouldn't receive any heavy issues on this way.

1 Like

So should we port podman to OpenWrt instead?

1 Like

Interesting that this got bumped just as I was looking into a few Docker-related items.

The link you posted regarding collaboration isn't really relevant, as that is discussing providing docker images that run OpenWRT in a container. I don't think @mcbridematt understood the goal of the thread either, since that's what he is providing and isn't what the OP was looking for.

This thread started discussing using OpenWRT as a docker engine host - which IS supported in git master and has been for quite a while (first merged in July - https://github.com/openwrt/packages/commits/master/utils/docker-ce ). I just discovered this last week, and strangely, there's NO discussion of this fact anywhere I can find. Any Google search for Docker and OpenWRT is looking at the opposite use case (OpenWRT within a docker container), or using a Docker container to facilitate building OpenWRT on hosts that don't play nice.

I can say that docker-ce running on an aarch64 Raspberry Pi 3b+ runs pretty well. The main issue is that you really need an additional storage device beyond the SD card, at least on pi, because sysupgrade will clobber the partition table, so you can't readily put a third partition on the sd card for persistent storage outside of the rootfs. If you've got Docker images there, it's way too much for sysupgrade to try and back up into RAM when the rootfs gets nuked.

5 Likes

Have a look at the first link - I did exactly that. The 'OpenWrt inside Docker' came in parallel.

I got Docker Engine running by repackaging the prebuilt binaries distributed by Docker (putting the full source build in the 'too hard basket' due to Go and cross-compiling), some months later someone figured out how to build the whole thing from source - hence the docker-ce package - https://github.com/openwrt/packages/pull/8397.

As it turns out the 'slim docker' I was hoping for already exists: https://www.balena.io/engine/ . I haven't had a chance to revisit any of this, I was also hoping to split out runc/containerd/etc so they could be integrated with procd.

1 Like

Sorry to bump this 3-month old thread, but I need to build some lightweight docker containers and it would be great to build them on my OpenWRT router.

I am not a tech guy, and I see there is no official docker-ce packages on the 19.07 repository. Could we expect to get it sometime or is it too difficult to implement docker in OpenWRT?

Thanks in advance and keep up the good work.

1 Like

subscribed

Support for arm cpus was added in November 2019 by juppin https://github.com/openwrt/packages/commit/a6be45ba66 and you are using 19.07 which was branched July 2019 from master.
So no docker for arm on 19.07, you have to use the snapshot builds to get docker on arm.

2 Likes

Thank you.

I have 19.07.03 flashed (May 2020), but I cannot find any docker or docker-ce package at the official repository through opkg.

I have a cortex a9 (arm) processor.

Where should I look for it? I found this:

But I do not know how to install it on my router.

The package doesn't exist in 19.07, it is currently only on master.

2 Likes

Thank you.

I have located the ipk.

docker-ce_19.03.8-2_arm_cortex-a9_vfpv3.ipk

Could I just download it and push it to my 19.07.3 openwrt router?

I have never installed any package other than using the software tab on Luci.

Installing across versions wouldn't be recommended because they are expecting specific system libraries and other things that are built into the OpenWRT version.

1 Like

Thank you again.

So my best bet is to wait for a 19.07.4 or 20.XX release to have it included in the stable branch, right?

It seems docker-ce was not included in 19.07.4, I cannot find it in the LuCi package search function.

I hope it will be included in 20.XX

Been waiting for this too.
I've got plenty of headroom on my router though. Using an x86_64. Interested if anyone has used the static binaries from Docker.

Hello, found this thread and is anybody else trying to run the new docker-ce on openWRT?!?
I got a aarch64 host with 2GB of RAM and a 64GB SDCard, so I thought I'd give it a whirl.
docker on x86_64 wouldn't even be worth asking about, that's too easy.

what I installed:

root@n3 opkg list-installed | egrep -i "container|docker|cgroup"
cgroup-tools - 0.41-3
cgroupfs-mount - 2020-06-26-05494281-1
containerd - 1.3.7-1
docker-ce - 19.03.13-1
docker-compose - 1.27.4-1
libcgroup - 0.41-3
luci-app-dockerman - v0.5.13
luci-lib-docker - git-20.211.27004-141857a
lxc-cgroup - 4.0.2-4
python3-docker - 4.3.1-1
python3-dockerpty - 0.4.1-3

yessir, I haz /sys/fs/cgroup mounted:

root@n3 df -hTP
Filesystem           Type            Size      Used Available Capacity Mounted on
/dev/root            ext4           58.8G    468.6M     58.3G   1% /
tmpfs                tmpfs         997.7M    964.0K    996.7M   0% /tmp
tmpfs                tmpfs         512.0K         0    512.0K   0% /dev
/dev/mmcblk0p1       ext4           15.7M     12.1M      3.3M  78% /boot
cgroup               tmpfs         997.7M         0    997.7M   0% sys/fs/cgroup
/dev/root            ext4           58.8G    468.6M     58.3G   1% /opt/docker

yessir, I haz a snapshot release,

because that's all you can do with a rockchip armv8/A53/aarch64 cpu, and docker-ce won't show up in a prod release till later -


root@n3 uname -a
Linux n3 5.4.70 #0 SMP PREEMPT Mon Oct 12 18:59:58 2020 aarch64 GNU/Linux

root@n3 egrep -i "pretty|release" /etc/os-release 
PRETTY_NAME="OpenWrt SNAPSHOT"
OPENWRT_RELEASE="OpenWrt SNAPSHOT r14694-e788e9bd87"

what I did:

root@n3 /etc/init.d/dockerd start

meanwhile chatty little bird rsyslog says:

Oct 14 01:00:07 n3 dockerd-init: Adding docker default interface to network uci config (docker)
Oct 14 01:00:07 n3 dockerd-init: Adding docker default bridge device to network uci config (docker0)
Oct 14 01:00:07 n3 dockerd-init: Adding docker default firewall zone to firewall uci config (docker)
Oct 14 01:00:07 n3 dockerd-init: Updating network uci config option "172.18.0.1/24" for docker default bridge (docker0)
Oct 14 01:00:08 n3 dockerd-init: Updating network uci config option "172.18.0.1/24" for docker default bridge (docker0)
Oct 14 01:00:09 n3 dockerd[1898]: time="2020-10-14T01:00:09.081074086Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Oct 14 01:00:09 n3 modprobe: failed to find a module named aufs
Oct 14 01:00:09 n3 dockerd[1898]: time="2020-10-14T01:00:09.094011984Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Oct 14 01:00:09 n3 dockerd[1898]: time="2020-10-14T01:00:09.577416251Z" level=warning msg="Your kernel does not support swap memory limit"
Oct 14 01:00:09 n3 dockerd[1898]: time="2020-10-14T01:00:09.577614309Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Oct 14 01:00:09 n3 dockerd[1898]: time="2020-10-14T01:00:09.577653978Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Oct 14 01:00:10 n3 dockerd[1898]: failed to start daemon: Devices cgroup isn't mounted

also:

opkg list | egrep -i "aufs|devmap"

whattheheck is aufs? devmapper?

we aint got no multipathd and lvm around here. This is a single SDCard rootdisk! Not even SPI/eMMC/flash!

Do I haveta supply arcane cgroups kernel boot args like we do in CentOS-land?

as things slip way from real docker to moby to podman?

There is currently some work happening to get podman to work with uxc, which basically replaces Docker with a much lower footprint.

1 Like

Try configuring docker to use overlay2 instead of aufs, it's the newer and better storage method.

wow, very interesting! just to let you know, the PR you mentioned has been merged now.

I had the same issues with a snapshot pi build.... took a long time to figure out and dont know why.... but the fix for me was

Update cgroup startup - comment out section

vi /etc/init.d/cgroupfs-mount

#        if mountpoint -q /sys/fs/cgroup; then
#                umount /sys/fs/cgroup/
#        fi

update dockerd start order from 25 -> 60

vi /etc/init.d/dockerd

        START=60

Update startup to setup tmpfs

        mount -t tmpfs -o size=32k tmpfs /run

If no longer needed then great but if stuck, try the above....