OpenWrt ARM64 QUICK LXC HOWTO/GUIDE (LMS in Debian System in LXC Container)

@erdoukki thanks a lot, like a lot. I deeply appreciate your kindness. It's working well. The thing is this router only supports up to 19.07, so I must find another router that supports 21.02. But thanks a lot. You're very kind. I'm trying now to test 'motion'. It has been developed for many years and somehow it is 'natively' in Openwrt, but not LMS. I've installed it but now must find a way to configure and use. A great day to you.
Camus

Great guide, thank you!

@erdoukki - You might want to mention that the changes to /usr/share/lxc/config/common.conf will not persist across updates. If users are compiling their own, one solution to this is to provide that file in $BUILDROOT/files/usr/share/lxc/config/common.conf so that it gets packages up in the image.

It does beg the question though, whether or not to include the file with cgroups commented out by default. Is there any kernel/arch that needs them?

Yes, I think that is mostly like a bug... and reporting it to the OpenWrt community, or, better, propose a fix (like the commented v1 cgroups) will be welcome...

1 Like

Another issue, which can be easily resolved :

LUCI-APP-LXC error : 
/usr/lib/lua/luci/dispatcher.lua:427: /etc/config/luci seems to be corrupt, unable to find section 'main'
# service lxc-auto enable
# service lxc-auto boot

Root cause : Cgroup namespace: required

Merged yesterday…
Thanks !

1 Like

@erdoukki - I copied your content to the wiki: https://openwrt.org/lxc_openwrt_host

I plan to generalize it over the next few days. Feel free to help.

1 Like

@erdoukki - Can you help me with that first table. It is meant to show the host arch mapping to what lxc calls their arches. For example, OpenWRT x68_64 host is called amd64 in lxc's world. I listed the other lxc arches in the 2nd column, but I don't know what the corresponding OpenWRT host arches are called for many of them so I used "tdb" for "to be determined."

Hi,

Thank you very much for sharing knowledge and making a Wiki page about how we can use LXC technology with OpenWRT, this is really amazing.

As a constructive feedback though, may I raise some points please? Like:

  • owrt v19 lacks LXC support by default, luckily in v21 it is already included
  • some packages listed above is not required per se to run an LXC container, on a default v21 you can run LXC without for example kmod-ikconfig, lxc-checkconfig. Or you don't need to tweak cgroup mounts hence no need any package for that.
  • there is a big problem with current versions of lxc* packages, e.g. owrt v21 moved to use cgroups v2 but packages assumes v1. Tools like lxc-top will not work as cannot read v2 attributes.
  • most problem is with package lxc-configs (as is mentioned already) as it is assuming v1 config. Which causes scary error messages when try to start a container.
  • I think packages could be improved to check/set dependencies better, e.g. /run is a requirement which does not exists in default owrt layout.

In short I think userland tools should be updated to work together with cgroups v2 setup. v1 is legacy now so should those tools catch up instead of tweaking owrt to use v1.

My 2 cents.

Thanks again for the great work!

Not sure how to handle it aside from providing a warning.

Yes, some are not needed. Perhaps we can categorize into a suggested and optional but suggested may depend on use-case. I would think the hard dependencies are already considered by packagers.

I do not find errors using lxc-top on my system build from latest master (post v21.02.1):

Container                   CPU          CPU          CPU                                BlkIO        Mem
Name                       Used          Sys         User                    Total(Read/Write)       Used
pihole                     0.00         0.00         0.00    509.99 GiB(509.99 GiB/509.99 GiB)    0.00   
TOTAL 1 of 1               0.00         0.00         0.00    509.99 GiB(509.99 GiB/509.99 GiB)    0.00   

I do not believe that is correct any more since PR#16549 was merged a few days ago. Please correct me if I am wrong.

I do not understand:

# ls -l /run
lrwxrwxrwx    1 root     root             8 Sep 18 13:40 /run -> /var/run

@erdoukki - Is the delay working correctly? I thought the daemon would behave by waiting for the pid to disappear waiting UP TO 30 sec, but that is not the case. It waits 30 s regardless of the exit status of lxc-stop it seems.

I was referring to 21.02 release, not any snapshot version. The PR16549 was merged 19 hours ago to master for example. My point is that 21.02 release by its own can run containers, which is really great, with a very few additional packages easily.

But - at least on x86 - the 21.02 release layout does not include e.g. /run and lxc-top due to v1 vs v2 problem produces this output:

lxc-top 2>&1 | tee t.txt
cat t.txt
Unable to read cgroup item memory.limit_in_bytes
Unable to read cgroup item memory.memsw.usage_in_bytes
Unable to read cgroup item memory.memsw.limit_in_bytes
Unable to read cgroup item memory.kmem.usage_in_bytes
Unable to read cgroup item memory.kmem.limit_in_bytes
Unable to read cgroup item cpuacct.usage
Unable to read cgroup item cpuacct.stat
Unable to read cgroup item cpuacct.stat
Unable to read cgroup item blkio.throttle.io_service_bytes
Unable to read cgroup item blkio.throttle.io_serviced
Unable to read cgroup item memory.usage_in_bytes
Unable to read cgroup item memory.limit_in_bytes
Unable to read cgroup item memory.memsw.usage_in_bytes
Unable to read cgroup item memory.memsw.limit_in_bytes
Unable to read cgroup item memory.kmem.usage_in_bytes
Unable to read cgroup item memory.kmem.limit_in_bytes
Unable to read cgroup item cpuacct.usage
Unable to read cgroup item cpuacct.stat
Unable to read cgroup item cpuacct.stat
Unable to read cgroup item blkio.throttle.io_service_bytes
Unable to read cgroup item blkio.throttle.io_serviced
^[[H^[[J^[[7m^[[1mContainer CPU CPU CPU BlkIO Mem
Name Used Sys User Total(Read/Write) Used
^[[0mxxxxx 0.00 0.00 0.00 0.00 ( 0.00 / 0.00 ) 0.00
yyyyy 0.00 0.00 0.00 0.00 ( 0.00 / 0.00 ) 0.00
TOTAL 2 of 2 0.00 0.00 0.00 0.00 ( 0.00 / 0.00 ) 0.00

Your example with 500GiB read/write and everything else is zero maybe is valid but I cannot verify from here, but my lxc-top on my system looks bad.

** Edit: /run does not exist in default layout but lxc-auto init script creates it actually. And lxc-auto is a great recommended package but not essential, probably would be better to include /run creation into package lxc. Because when you manually create a container and try to start it will fail due to missing /run.

Yes but still not present in 21.02

@darksky can you made a PR for 21.02 which CherryPick the commit of the merged one, please ?

I do not test it...
I'll try to give some tests... or look at the init.d !

1 Like

and

That should wait up to the $timeout defined as I read it.

# time lxc-stop -n pihole
lxc-stop -n pihole  0.00s user 0.01s system 1% cpu 0.851 total

So for me it stops very quickly. But using the deamon as written does not (I have a 10 sec delay defined for it in /etc/config/lxc-auto):

# time /etc/init.d/lxc-auto stop
/etc/init.d/lxc-auto stop  0.02s user 0.00s system 0% cpu 10.018 total

BUT:

So with out a for loop, it will just wait the defined value.

I would think it needs to watch the exit code of lxc-stop sleeping for 1 second up to the $maxtimeout then quit when it is gone, no?

EDIT: I took a stab at it here: https://github.com/openwrt/packages/pull/16658

1 Like

What about 0 value ?
No timeout at all...

A real cherry-pick is better !

https://github.com/openwrt/packages/pull/16549 commit as reference will be sufficient and get more fast merge...

I discovered this with my Fail2Ban package :
https://github.com/openwrt/packages/pull/16592 cherry-picked from
https://github.com/openwrt/packages/pull/15098

Problem with that is if the container is doing something on shutdown, the init system on the host could kill it and you would lose data. Honoring the timeout is better.

Done, https://github.com/openwrt/packages/pull/16660