Docker extracting image - very slow

zawier · May 27, 2023, 6:55pm

Hi,
So basically my problem is similar to case: like this. Where in big shurtcut when pulling image from dockerhub it can takes forever .. Finally always it finish it's job but extracting over all night is too much

At linked post users claims that sollution is switching to faster sd card solves a problem, which I really doubt that! And here is why (btw I have switched to a brand new one):
I have NonoPi R4S with 6cores on CPU and 4GB of RAM - for a such device like it it's nice, right? Additionally I have equipped it in additional 1TB SSD - so we can assume it won't be faster that that. My net speed is ~ 750MiB/s.

Already I have a few images pulled and up running. It works ok. But while pulling a new image then it stock (acctually work but reallllllllllyyyyyy slow). Please keep in mind that!

I changed SD card. First attempt to pull - WOW it did the trick!! And after system reboot it back again to a snail mode And because of that I don't belive SD hint helps here.

But I was tinkering around. Probably I supost start from fact that during first start on a brand new SD:

root@FriendlyWrt:~# blkid
/dev/mmcblk1p8: LABEL="rootfs" UUID="ff313567-e9f1-5a5d-9895-3ba130b4a864" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="rootfs" PARTUUID="b2af085d-a675-48c6-c437-f6d557ff4744"
/dev/mmcblk1p9: LABEL="userdata" UUID="269e9152-5cb6-035b-9c34-e1ed87cca310" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="userdata" PARTUUID="2d9e7b61-1b31-47e7-ee0d-8cec26d42ef6"
/dev/sda1: UUID="6D2A-9E26" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="50098e3a-01"
/dev/sdb1: LABEL="RedNateck_NanoPi" UUID="cce28d02-24e0-488e-a848-f60d0d28e7c1" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="587fb637-01"
/dev/mmcblk1p3: PARTLABEL="misc" PARTUUID="43784a32-a03d-4ade-92c6-ede64ff9b794"
/dev/mmcblk1p1: PARTLABEL="uboot" PARTUUID="b750e44e-833f-4a30-c38c-b117241d84d4"
/dev/mmcblk1p6: PARTLABEL="kernel" PARTUUID="1cac805f-726a-495a-fd35-821355a6e7e8"
/dev/mmcblk1p4: PARTLABEL="dtbo" PARTUUID="000b305f-484a-4582-9090-4ad0099d47bd"
/dev/mmcblk1p2: PARTLABEL="trust" PARTUUID="a1c81622-7741-47ad-b846-c6972488d396"
/dev/mmcblk1p7: PARTLABEL="boot" PARTUUID="2bfee623-d83c-426a-ab80-21732c9bb7d3"
/dev/mmcblk1p5: PARTLABEL="resource" PARTUUID="24eeb649-277f-4c11-ffeb-d9f20027a83b"

/dev/sdb1 here is my SSD and it's crutial. Sda1 is another flash drive.

block info command:

Disk /dev/sdb: 953.87 GiB, 1024209543168 bytes, 2000409264 sectors
Disk model: 400-01T-G2
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x587fb637

Device     Boot Start        End    Sectors   Size Id Type
/dev/sdb1        2048 2000396287 2000394240 953.9G 83 Linux

So we know what type of disc it is. Let's proceed then.

When I'm able to find that disk then pulling images from docker hub is jus a bullet speed! 1-2 min for images like 1GB (including extraction time).

There is down side here - while I'm able to get mounted SSD my data in containeres seams to be lost.. Because non of app doesn't recognize my credentials any more...

But lets reboot this sucker!
Suddenly blkid lost /dev/sdb1 after waking up. And something interesting happened:

and

with set up like that my data from containers are backed!

My docker setup point on that location because of:

root@FriendlyWrt:~# docker info
Client:
Context:    default
Debug Mode: false
Plugins:
compose: Docker Compose (Docker Inc., v2.15.1)

Server:
Containers: 5
Running: 3
Paused: 0
Stopped: 2
Images: 13
Server Version: 20.10.22
Storage Driver: fuse-overlayfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version: de40ad0
Kernel Version: 5.15.78
Operating System: OpenWrt 22.03.3
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 3.747GiB
Name: FriendlyWrt
ID: W6EH:KEAU:CLUM:HWVM:G5WX:RUFY:U5EX:ZH3O:7XHT:I7WQ:M5RN:4V2J
Docker Root Dir: /mnt/sdb1/Docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Docker Root Dir: /mnt/sdb1/Docker - I just wanted to store images, containers and volumes outside of slow micro SD card.

Please check out what happened with mount bellow:

root@FriendlyWrt:~# mount |grep sdb1
overlay on /mnt/sdb1/Docker type overlay (rw,noatime,lowerdir=/root,upperdir=/data/root,workdir=/data/work)
fuse-overlayfs on /mnt/sdb1/Docker/fuse-overlayfs/6c314d6a7350e936f4fe2e04d0f864492b396ed594724c82541dfc8179c2375d/merged type fuse.fuse-overlayfs (rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
fuse-overlayfs on /mnt/sdb1/Docker/fuse-overlayfs/24dde8dd5b57f25b122b31890b7c0167a9f64734be94065f832580c8c56dfd6f/merged type fuse.fuse-overlayfs (rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
fuse-overlayfs on /mnt/sdb1/Docker/fuse-overlayfs/0f2bab965354b03fc0141af31204a2cf9a1aad7d95ba1979264c5ff6ea637c46/merged type fuse.fuse-overlayfs (rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)

How to get rid of that overlay but in the same time be able to not loose data from my current containers? Because of that overlay pulling images causing performance issue. And I'm sure about that after my investigation

Personally I find particular topic very interesting, but I think I chaising my tail currently

Any idea?
Thanks in advance!

psherman · May 27, 2023, 6:58pm

This is a form of OpenWrt and is not the same as openwrt. You will probably get better information by asking on their user forums.

zawier · May 27, 2023, 7:15pm

Yeap, FriendlyWrt but it's based on OpenWrt as well and in the end of the day it's linux, right? FriendlyWrt from my point is different with a theme. Other things I assume are the same or at least similar. Besides this particular forum is helpfull. In case of FriendlyWrt you have to be a Chinese to find interesting topics I send an email to person who created an image, but haven't received any respond. That's why I'm here..

psherman · May 27, 2023, 7:28pm

Yes and no. There are very significant differences. That's like saying that OpenWrt is basically the same as Ubuntu or Debian or BSD. In fact, even those three big distros have pretty significant differences relative to each other.

That is the goal of this forum. We're not trying to be unhelpful, but we just don't know the nuances of FriendlyWrt... and due to the differences between OpenWrt and the forks/offshoots, quite often there is a need for specific knowledge of those forks -- it's not generic.

You can always use online translators and/or find other forums that specialize in FriendlyWrt that are in a language that is more familiar to you.

Alternatively, you can consider installing Official OpenWrt if your device is supported.

zawier · May 27, 2023, 7:45pm

You naver translated anything from Chinese, didn't you Good luck with that..

Well, for my point of view is more like I bought car and asked how to drive it. My question is not like " how to drive exactly Toyota Prius". Believe making a simplification like that hopefully I'm able to receive any advice.

Please, instead keep talking about differences lets focus on main problem.
In past I've tried OpenWrt , but faced issues with WiFi, plus other problems and returned to FriendlyWrt. I would like to stick to id. Even because of trivial reason - I realy love Argon theme

Come on, I don't expect anybody to be a proficient with my case. I just hope somebody might point me on key direction. That's all.

psherman · May 27, 2023, 7:55pm

Maybe it's better to look at it like: you bought a car that has a problem you need to fix.

So, if we continue with the car analogy, let's say that the fuel/air mixture isn't working properly leading to poor engine performance. We would need to know more details about the inner workings of your car -- for example, diesel vs gasoline, fuel-injected vs naturally aspirated, regular vs turbo. Now, let's say that this forum specialized in modern gasoline cars with fuel injection and yours uses a carburetor... you'd probably want the advice of someone who knows how to fix carburetor issues rather than some generic advice.

The main problem is something to do with FriendlyWrt working slowly with the docker image you're extracting. Can you confirm that the same problem occurs if you use official OpenWrt? If so, that makes it a more generic problem to be solved here. But, if it is unique to your FriendlyWrt environment, it is the domain knowledge of those familiar with that firmware that you need to seek.

zawier · May 27, 2023, 8:03pm

No I can't. I don't have right now OpenWrt as you noticed. But believe he can! while all symptoms are the same. He has OpenWrt. Looks exactly like my case!

That user I assume posted too quickly his sollution (or downloaded images which he was interested into and never back again to topic - on devices like that usually you run 1-5 containers max).

psherman · May 27, 2023, 8:07pm

So... let's go with the solution from that very thread:

zawier · May 27, 2023, 8:18pm

As I wrote, I have changed my sd card for a brand new. If you haven't noticed I wrote also that:

First boot and pulling images done the trick!
but after restart it happened again (performance issue)

I described that as best I as I could. In my opinion sd card here is not causing a problem but auto mounting external SSD as overlay!

That's why my guess is that probably avoiding that behavior solves a problem.

So probably I should ask : how to change it?

psherman · May 27, 2023, 8:22pm

Does that new card have significantly higher performance than the previous card?

The specifics of the issue cannot be attributed to OpenWrt specifically (as compared to changes made for FriendlyWrt or other configuration issues) unless there is a reference point and a means of comparison with consistency of the variables.

Can you install OpenWrt on an x86 machine (including as a VM) or a Raspberry Pi or similar and try it there? No it won't use the same hardware as your FriendyWrt device, but at least you'll be able to see if the same performance issues arise after a reboot.

zawier · May 27, 2023, 8:36pm

Peter, one thing came on my mind: before adding external hdd it was working fine. There wasn't problem with extraction (for write speed I've read 20-70Mb/s -so fine). But after runing a docker pulling x images, runing containers , adding SSD, moving all related with docker (except docker run files etc) to a new directory and assigning new dir as main docker work folder started that.

Somehow my system when figure it out that my SSD is full of containers - it "adopted" that part as it's own main overlay..

Sure I can, but also doubt in finding out spectacular discovery, because architecture is different there as well. My is arm64v6.

I really believe we should focus on that overlay here.
How do you change mounting specific disk under OpenWrt? How do you prevent it being mounted as something else (overlay)?

psherman · May 27, 2023, 8:43pm

Storage mounting documentation (for OpenWrt -- ymmv using forks) is here:

OpenWrt will not mount a drive as an overlay unless it is specifically added to the fstab as an overlay partition. See the extroot example to see how overlay is specified in the fstab.