Host system is Arm64 ,docker container is amd x86

joneswu456 · May 2, 2024, 7:39am

Hi all,
I encountered a trouble.My host system is arm64/kernel 5.4.213 for openwrt ,and docker container is amd64 x86 for ubuntu 22.04 that kernel is 5.4.0-169-generic I think,but it will map to real host system 5.4.213.
I want to insmod a haipci driver from docker container,but it alway told me below:
/lib/modules/5.4.213# insmod hailo_pci.ko
insmod: ERROR: could not insert module hailo_pci.ko: Function not implemented
how come?
The environment as below:
Host Dokcer Container
Openwrt Ubuntu 22.04
Arm64 Amd64
Kernel:5.4.213 Kernel: 5.4.0-169-generic

Thanks.

joneswu456 · May 2, 2024, 7:40am

I found that is ioctl restriction ,“Error in ioctl: Function not implemented” as run hailortcli benchmark to read/write /dev/hailo0 ,but I really don’t understand what’s problem.As below is running command:
docker run ‘–cap-add=SYS_RAWIO’ ‘–cap-add=cap_chown’ ‘–cap-add=cap_sys_admin’ ‘–cap-add=sys_nice’ ‘–cap-add=net_raw’ ‘–cap-add=cap_sys_rawio’ ‘–cap-add=cap_sys_module’ ‘–cap-add=NET_ADMIN’ --platform linux/amd64 ‘–net=host’ --device /dev/hailo0:/dev/hailo0:rw --volume /usr/bin/qemu-x86_64:/usr/bin/qemu-x86_64 -v /dev:/dev:rw -v /lib/modules:/lib/modules:rw -v /lib/firmware:/lib/firmware -v /lib/udev/rules.d:/lib/udev/rules.d -v /mnt/sda1/shared_with_docker/:/local/shared_with_docker:rw --name max1 -ti hailort:4.16.0
.I stuck the issue a couple of weeks.

joneswu456 · May 2, 2024, 7:40am

I integrated a buildx into my docker platform also can work it,but I encounted ioctl issue as using "hailortcli benchmark " to read/write /dev/hailo0.I worried about maybe some missing in host or container,so I need to help it.Thanks
As below is detailed:

./hailortcli benchmark …/resnet_v1_50.hef

Starting Measurements…
Measuring FPS in hw_only mode
Error in ioctl: Function not implemented
[HailoRT] [error] Failed query driver info, errno 38
[HailoRT] [error] CHECK_SUCCESS_AS_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT CLI] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36) - Failed creating vdevice
[HailoRT CLI] [error] CHECK_EXPECTED_AS_STATUS failed with status=HAILO_DRIVER_FAIL(36) - hw_only measuring failed

joneswu456 · May 2, 2024, 7:42am

root@ROOter:/proc# docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.13.1)

Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 4
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version: de40ad0
Kernel Version: 5.4.213
Operating System: OpenWrt 19.07-SNAPSHOT
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 876.4MiB
Name: ROOter
ID: 2C67:46UP:4ON5:QIIA:3DLN:ONGT:PIJY:ZCFD:TXJU:VLRG:46XE:Z7CQ
Docker Root Dir: /mnt/sda1/docker/tmp/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
root@ROOter:/proc#

slh · May 2, 2024, 8:05am

Before even diving into x86_64 emulated on ARM…

Running OpenWrt in a container (contrary to a full system emulation, such as qemu/kvm or similar) is not a supported configuration<fullstop>.

…exactly for the reasons you are experiencing (among others). A container isn't allowed to load kernel modules, change sysctl settings or monitor hardware devices by design - and that's before even thinking about arm64 vs x86_64 (which can't work either).

zekica · May 2, 2024, 8:17am

Note: this is either an old, unsupported version of OpenWrt or it is a 3rd party software based on OpenWrt, so this forum is probably the wrong place to ask for help.

Anyway, what happens when you try to load the module from the host system, not from inside the container?

brada4 · May 2, 2024, 8:18am

You need to run virtual machine (emulated in your chosen configs) to load modules in guest system.

joneswu456 · May 2, 2024, 8:44am

Thanks, zekica,
Now,I used the hailortcli benchmark in the arm64 host system that is work,but I also need to use the same command to run it in the amd64_x86 target system in docker container for Mulit-arch.Therefore,I got the ioctl restriction ,“Error in ioctl: Function not implemented”.
I stuck the issue a couple of weeks.
I really don't understand what happens.So,I need to help.Thanks,Also posted the issue to Docker Community Forums ,but they said maybe to ask the maintainers of the openwrt docker package.tks.

brada4 · May 2, 2024, 8:55am

There is no bug, it works like expected, processor will not run binaries compiled for a different processor. You need aarch64 container and load kmods in the docker host before running container.

zekica · May 2, 2024, 9:21am

How are you running x86_64 binaries on arm64? Do you use box64, qemu-user or?

I don't think any of them implement the insmod syscalls, as they don't make sense
Even if possible, you still need to insert the module from the host.
What you really need is to bind mount the /dev/halio0 inside the container.

joneswu456 · May 2, 2024, 9:27am

joneswu456 · May 2, 2024, 9:36am

Thanks,zekica,
Because of a python3 enviroment need to setup in ubuntu 22.04 for amd64_x86 in a docker container.The python3 need to read/write hailort device /dev/hailo0 via a hailort library.But the physical hw inside arm64 host system.
Tks

zekica · May 2, 2024, 9:42am

Can you try bind mounting the /dev/halio0 inside the container?

brada4 · May 2, 2024, 9:47am

to sum up

ask your container packager to supply multi-arch image.

joneswu456 · May 2, 2024, 10:00am

brada4 · May 2, 2024, 10:37am

Install arm64 container and ioctls will be 1:1

joneswu456 · May 9, 2024, 5:21am

Thanks,I see,but unfortunately,our partner's development environment has been implemented on amd64_x86,so that is why I need to port a docker container of amd64_x86.

psherman · May 9, 2024, 5:50am

As stated previously, the firmware you are attempting to use is not from the official OpenWrt project, and the details of your attempts suggest that it will not be possible.

Since you are using firmware not provided here, please reach out to the maintainers of the code that you are working with.

psherman · May 9, 2024, 6:50am

This topic was automatically closed after 60 minutes. New replies are no longer allowed.