Host system is Arm64 ,docker container is amd x86

Hi all,
I encountered a trouble.My host system is arm64/kernel 5.4.213 for openwrt ,and docker container is amd64 x86 for ubuntu 22.04 that kernel is 5.4.0-169-generic I think,but it will map to real host system 5.4.213.
I want to insmod a haipci driver from docker container,but it alway told me below:
/lib/modules/5.4.213# insmod hailo_pci.ko
insmod: ERROR: could not insert module hailo_pci.ko: Function not implemented
how come?
The environment as below:
Host Dokcer Container
Openwrt Ubuntu 22.04
Arm64 Amd64
Kernel:5.4.213 Kernel: 5.4.0-169-generic


I found that is ioctl restriction ,“Error in ioctl: Function not implemented” as run hailortcli benchmark to read/write /dev/hailo0 ,but I really don’t understand what’s problem.As below is running command:
docker run ‘–cap-add=SYS_RAWIO’ ‘–cap-add=cap_chown’ ‘–cap-add=cap_sys_admin’ ‘–cap-add=sys_nice’ ‘–cap-add=net_raw’ ‘–cap-add=cap_sys_rawio’ ‘–cap-add=cap_sys_module’ ‘–cap-add=NET_ADMIN’ --platform linux/amd64 ‘–net=host’ --device /dev/hailo0:/dev/hailo0:rw --volume /usr/bin/qemu-x86_64:/usr/bin/qemu-x86_64 -v /dev:/dev:rw -v /lib/modules:/lib/modules:rw -v /lib/firmware:/lib/firmware -v /lib/udev/rules.d:/lib/udev/rules.d -v /mnt/sda1/shared_with_docker/:/local/shared_with_docker:rw --name max1 -ti hailort:4.16.0
.I stuck the issue a couple of weeks.

I integrated a buildx into my docker platform also can work it,but I encounted ioctl issue as using "hailortcli benchmark " to read/write /dev/hailo0.I worried about maybe some missing in host or container,so I need to help it.Thanks
As below is detailed:

./hailortcli benchmark …/resnet_v1_50.hef

Starting Measurements…
Measuring FPS in hw_only mode
Error in ioctl: Function not implemented
[HailoRT] [error] Failed query driver info, errno 38
[HailoRT] [error] CHECK_SUCCESS_AS_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36)
[HailoRT CLI] [error] CHECK_EXPECTED failed with status=HAILO_DRIVER_FAIL(36) - Failed creating vdevice
[HailoRT CLI] [error] CHECK_EXPECTED_AS_STATUS failed with status=HAILO_DRIVER_FAIL(36) - hw_only measuring failed

root@ROOter:/proc# docker info
Context: default
Debug Mode: false
buildx: Docker Buildx (Docker Inc., v0.13.1)

Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 4
Server Version: 20.10.22
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version: de40ad0
Kernel Version: 5.4.213
Operating System: OpenWrt 19.07-SNAPSHOT
OSType: linux
Architecture: aarch64
CPUs: 4
Total Memory: 876.4MiB
Name: ROOter
Docker Root Dir: /mnt/sda1/docker/tmp/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Before even diving into x86_64 emulated on ARM…

Running OpenWrt in a container (contrary to a full system emulation, such as qemu/kvm or similar) is not a supported configuration<fullstop>.

…exactly for the reasons you are experiencing (among others). A container isn't allowed to load kernel modules, change sysctl settings or monitor hardware devices by design - and that's before even thinking about arm64 vs x86_64 (which can't work either).


Note: this is either an old, unsupported version of OpenWrt or it is a 3rd party software based on OpenWrt, so this forum is probably the wrong place to ask for help.

Anyway, what happens when you try to load the module from the host system, not from inside the container?

You need to run virtual machine (emulated in your chosen configs) to load modules in guest system.

1 Like

Thanks, zekica,
Now,I used the hailortcli benchmark in the arm64 host system that is work,but I also need to use the same command to run it in the amd64_x86 target system in docker container for Mulit-arch.Therefore,I got the ioctl restriction ,“Error in ioctl: Function not implemented”.
I stuck the issue a couple of weeks.
I really don't understand what happens.So,I need to help.Thanks,Also posted the issue to Docker Community Forums ,but they said maybe to ask the maintainers of the openwrt docker package.tks.

There is no bug, it works like expected, processor will not run binaries compiled for a different processor. You need aarch64 container and load kmods in the docker host before running container.

How are you running x86_64 binaries on arm64? Do you use box64, qemu-user or?

  1. I don't think any of them implement the insmod syscalls, as they don't make sense
  2. Even if possible, you still need to insert the module from the host.
  3. What you really need is to bind mount the /dev/halio0 inside the container.

Because of a python3 enviroment need to setup in ubuntu 22.04 for amd64_x86 in a docker container.The python3 need to read/write hailort device /dev/hailo0 via a hailort library.But the physical hw inside arm64 host system.

Can you try bind mounting the /dev/halio0 inside the container?

1 Like

to sum up

ask your container packager to supply multi-arch image.

Install arm64 container and ioctls will be 1:1

1 Like

Thanks,I see,but unfortunately,our partner's development environment has been implemented on amd64_x86,so that is why I need to port a docker container of amd64_x86.

As stated previously, the firmware you are attempting to use is not from the official OpenWrt project, and the details of your attempts suggest that it will not be possible.

Since you are using firmware not provided here, please reach out to the maintainers of the code that you are working with.

This topic was automatically closed after 60 minutes. New replies are no longer allowed.