OpenWrt ARM64 QUICK LXC HOWTO/GUIDE (LMS in Debian System in LXC Container)

erdoukki · June 23, 2021, 4:32pm

OpenWrt ARM64 QUICK LXC HOWTO/GUIDE

Based on 21.02-RC
TESTED on ARM64 MVEBU ESPRESSOBIN et ESPRESSOBIN ULTRA

Install some necessary tools

opkg install xz tar gnupg

Install some necessary kernel modules

opkg install kmod-ikconfig kmod-veth

install some cgroups necessary tools

opkg install cgroupfs-mount cgroup-tools

install all lxc stuff

opkg install liblxc luci-app-lxc lxc lxc-attach lxc-auto lxc-autostart lxc-cgroup lxc-checkconfig lxc-common lxc-config lxc-configs lxc-console lxc-copy lxc-create lxc-destroy lxc-device lxc-execute lxc-freeze lxc-hooks lxc-info lxc-init lxc-ls lxc-monitor lxc-monitord lxc-snapshot lxc-start lxc-stop lxc-templates lxc-top lxc-unfreeze lxc-unprivileged lxc-unshare lxc-user-nic lxc-usernsexec lxc-wait rpcd-mod-lxc

Check your kernel configs

root@ultra:~# lxc-checkconfig 
LXC version 4.0.5
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu
/sys/fs/cgroup/cpuacct
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/pids
/sys/fs/cgroup/rdma
/sys/fs/cgroup/systemd

Cgroup v2 mount points: 


Cgroup v1 freezer controller: missing
Cgroup v1 clone_children flag: enabled
Cgroup device: missing
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, not loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: missing
CONFIG_IP6_NF_TARGET_MASQUERADE: missing
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: missing
CONFIG_INET_DIAG: missing
CONFIG_PACKET_DIAG: missing
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

Create a LXC container for LMS based on Debian BUSTER

root@ultra:~# lxc-create --name myLMS --template download -- --dist debian --release buster --arch arm64
Setting up the GPG keyring
ERROR: Unable to fetch GPG key from keyserver
lxc-create: myLMS: lxccontainer.c: create_run_template: 1616 Failed to create container from template
lxc-create: myLMS: tools/lxc_create.c: main: 319 Failed to create container myLMS

FIXME ERROR: Unable to fetch GPG key from keyserver

in case of GPG error :
Two solutions, force a gpg keyserver or disable GPG validation

first solution, force a gpg keyserver

opkg install gnupg2-utils gnupg2-dirmngr

or

opkg install gnupg-utils

root@ultra:~# DOWNLOAD_KEYSERVER="pgp.mit.edu" lxc-create --name myPOD --template download -- --dist debian --release buster --arch arm64
Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

Or second solution with disabling gpg validation

root@ultra:~# lxc-create --name myLMS --template download -- --dist debian --release buster --arch arm64 --no-validate
Downloading the image index
WARNING: Running without gpg validation!
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Debian buster arm64 (20210623_05:24) container.

To enable SSH, run: apt install openssh-server
No default root or user password are set by LXC.

List container and status

root@ultra:~# lxc-ls -f
NAME  STATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED 
myLMS STOPPED 0         -      -    -    false

FIX CGROUPS V1 ERROR

root@ultra:~# lxc-start -n myLMS
lxc-start: myLMS: lxccontainer.c: wait_on_daemonized_start: 860 Received container state "ABORTING" instead of "RUNNING"
lxc-start: myLMS: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: myLMS: tools/lxc_start.c: main: 311 To get more details, run the container in foreground mode
lxc-start: myLMS: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

THE FIX (comments all cgroups legacy parts)

root@ultra:~# nano /usr/share/lxc/config/common.conf
# Default legacy cgroup configuration
#
# CGroup allowlist
#lxc.cgroup.devices.deny = a
## Allow any mknod (but not reading/writing the node)
#lxc.cgroup.devices.allow = c *:* m
#lxc.cgroup.devices.allow = b *:* m
## Allow specific devices
### /dev/null
#lxc.cgroup.devices.allow = c 1:3 rwm
### /dev/zero
#lxc.cgroup.devices.allow = c 1:5 rwm
### /dev/full
#lxc.cgroup.devices.allow = c 1:7 rwm
### /dev/tty
#lxc.cgroup.devices.allow = c 5:0 rwm
### /dev/console
#lxc.cgroup.devices.allow = c 5:1 rwm
### /dev/ptmx
#lxc.cgroup.devices.allow = c 5:2 rwm
### /dev/random
#lxc.cgroup.devices.allow = c 1:8 rwm
### /dev/urandom
#lxc.cgroup.devices.allow = c 1:9 rwm
### /dev/pts/*
#lxc.cgroup.devices.allow = c 136:* rwm
### fuse
#lxc.cgroup.devices.allow = c 10:229 rwm

START CONTAINER

root@ultra:~# lxc-start -n myLMS

root@ultra:~# lxc-ls -f
NAME  STATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED 
myLMS RUNNING 0         -      -    -    false

STOP CONTAINER

root@ultra:~# lxc-stop -n myLMS

root@ultra:~# lxc-ls -f
NAME  STATE   AUTOSTART GROUPS IPV4 IPV6 UNPRIVILEGED 
myLMS STOPPED 0         -      -    -    false

ADD NETWORK CONFIG IN LXC CONTAINER

root@ultra:~# nano /srv/lxc/myLMS/config`

...
# Network configuration
#lxc.net.0.type = empty
lxc.net.0.type = veth
lxc.net.0.link = br-lan
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:FF:DD:BB:CC:01

START CONTAINER

root@ultra:~# lxc-start -n myLMS

root@ultra:~# lxc-ls -f
NAME  STATE   AUTOSTART GROUPS IPV4          IPV6                               UNPRIVILEGED 
myLMS RUNNING 0         -      192.168.1.188 fdc5:f7f:d0b5:0:2ff:ddff:febb:cc01 false

DEBIAN POSTINSTALL SSH + SUDO FIXME

root@ultra:~# lxc-attach -n myLMS

root@myLMS:~#

$ adduser admin
$ apt install sudo
$ addgroup admin sudo
$ apt install ssh -y
$ ip a

root@myLMS:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:ff:dd:bb:cc:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.188/24 brd 192.168.1.255 scope global dynamic eth0
       valid_lft 42908sec preferred_lft 42908sec
    inet6 fdc5:f7f:d0b5:0:2ff:ddff:febb:cc01/64 scope global dynamic mngtmpaddr 
       valid_lft forever preferred_lft forever
    inet6 fe80::2ff:ddff:febb:cc01/64 scope link 
       valid_lft forever preferred_lft forever

$ exit

LXC-AUTO AUTOBOOT AT BOOTUP FIXME

uci show lxc-auto
uci add lxc-auto container
uci set lxc-auto.@container[-1].name=myLMS
uci set lxc-auto.@container[-1].timeout=30
uci show lxc-auto
uci commit lxc-auto

access with SSH to Debian system

ssh admin@myLMS

ssh admin@192.168.1.188

---8<---

install LMS inside Debian system

REFS :

sudo apt install wget -y
sudo apt-get install libio-socket-ssl-perl
wget http://downloads.slimdevices.com/LogitechMediaServer_v8.1.1/logitechmediaserver_8.1.1_all.deb
sudo dpkg -i logitechmediaserver_8.1.1_all.deb

ACCESS LMS WEBUI

http://mylms.local:9000/

http://192.168.1.188:9000

erdoukki · June 23, 2021, 4:39pm

anon50098793 · June 23, 2021, 4:39pm

cool... a few minor tweaks would get you a unifi-controller also...

eg

wget https://get.glennr.nl/unifi/install/unifi-5.13.32.sh
bash unifi-5.13.32.sh

erdoukki · June 23, 2021, 4:44pm

yes, or any service or webapp a Debian (arm64) could host...!
preferably, each one in its own container !

erdoukki · June 23, 2021, 4:52pm

you can also cross mount a share point or a folder with :

MOUNT SHARE INSIDE LXC

add in /srv/lxc/myLMS/config
...
lxc.mount.entry = /mnt/SHARE /srv/lxc/myLMS/rootfs/mnt/SHARE none bind,create=dir 0 0

erdoukki · June 23, 2021, 5:20pm

take care, the script you give is adding swap !
swap can (will) kill SSD, eMMC, and any memory hard drive !!!

erdoukki · June 24, 2021, 6:42am

DISABLE MYSQUEEZEBOX (CLOUD) FOR LMS IN DEBIAN

admin@myLMS:~$ sudo nano /etc/default/logitechmediaserver 
...
SLIMOPTIONS="--nomysqueezebox"

erdoukki · June 24, 2021, 10:55am

an alternative with zram ?

opkg update
opkg install kmod-zram zram-swapfile

root@ultra:~# swapon -s
Filename				Type		Size	Used	Priority
/dev/zram0                              partition	507900	0	-2

root@ultra:~# free -h
              total        used        free      shared  buff/cache   available
Mem:        1017176      266520       85568       78604      665088      606516
Swap:        507900           0      507900

root@ultra:~# service zram status

Gathering stats info for zram device "zram0"

Z-RAM
-----
Block device              - /sys/block/zram0
Device size               - 496 MiB
Compression algo          - [lzo] lzo-rle lz4 zstd 

DATA
----
Original data size        - 0.00 MiB
Compressed data size      - 0.00 MiB
Compress ratio            - 69.42 

MEMORY
------
Memory used, total        - 0.00 MiB
Allocator overhead        - 0.00 MiB
Allocator efficiency      - 1.44 %
Maximum memory ever used  - 0.00 MiB
Memory limit              - 0.00 MiB

PAGES
-----
Same pages count          - 0
Pages compacted           - 0
Free pages discarded      - 126975
root@ultra:~#

camus · June 24, 2021, 2:59pm

@erdoukki you should be installed into some kind of a hall-of-fame in this forum. What a genius and most valued member of this forum. Thanks a lot! I will give this a try. My router may not work, it's old but running 21.02-RC. I have exroot to a USB, however. Anyone care to say if it will work with an exroot, please? Meanwhile, I will now go on the market for a device like yours, ARM64 MVEBU ESPRESSOBIN. What is that, anyway? Thanks a lot!

erdoukki · June 24, 2021, 3:10pm

EspressoBin Ultra with 8GB eMMC
https://www.globalscaletechnologies.com/p-88-espressobin-ultra.aspx
With a SSD of 1TB’added…

There is also the standard v7 with a 4GB eMMC…
But you’ll need to add a box and a power supply…

You might also look at the RPi 4 which is now supported in OpenWrt (not tested myself…)

camus · June 24, 2021, 3:20pm

@erdoukki I sent their link to a friend, who found and sent me this. Has this been your experience, any of it at all?
Poor design, horrible reliability, ZERO support

Rezension aus den Vereinigten Staaten vom 13. Dezember 2018

Stil: V5 64 Bit

This product seems like a good idea, but the implementation has been terrible. I've been looking for a low-cost single board computer with multiple gigabit ethernet ports to use as a home internet gateway. While the specs of this board fit the bill perfectly, the board itself is an incredible disappointment.

First of all, the design was poorly done, with wrong-gender connectors on the board, making it necessary to find and modify rare cables to make it work. The V5 design uses a male Molex connector for power, and the V7 design does something similar with the SATA power connector. Good luck finding the proper cables...

Secondly, the operating system is not well supported. There are a couple of flavors of linux available, notably Arch Linux and Armbian, but neither is anywhere close to being bug free for this board. The board will only report a single MAC address for all of the physical ports, updates to the board's U-Boot can easily brick the system, and the instructions for recovering from any of these problems are either nonexistant or inaccurate.

Finally, the website support for this product is a joke. Questions usually go unanswered, and when answers ARE given, they're generally wrong.

An internet gateway has to work 24/7. Reliability is foremost in a device that you intend to install and then hopefully forget about for the next few years. Everything about this board is as if someone took a great idea, then handed it off to an idiot to implement. Do not waste your money on this board.

erdoukki · June 24, 2021, 4:45pm

Not agreed, I used more than few for years !
I like them also because it is full opensource software system now…
The ultra may missing the uboot part for now but it may come soon.
The problem is’only with the wifi of the ultra, still not working on 21.02-RC !
The wifi is optional on espressobin v7.

This review is known, but mostly partially false in my feelings.
But it is’only my personal point of view.
These development board are good home routeur.

anon50098793 · June 28, 2021, 8:18am

fwiw.. I tested the guide on a rpi4 and mostly went ok...

i'd probably add a section re:storage and options for bind mounting over /srv or similar...

thanks again...

camus · June 28, 2021, 6:03pm

@erdoukki I'm trying now to use the guide and got stuck here:
" * opkg_install_cmd: Cannot install package cgroupfs-mount."
But I'm doing this on 19.07, however, the message in luci says:

Perhaps you had installed it in a special way and can help with that, please?
Thanks a lot.

erdoukki · June 28, 2021, 6:05pm

It will only work on 21.02-RC !

camus · June 28, 2021, 6:08pm

@erdoukki Thanks a lot. So now I give up, as I just continued and for those like me who may attempt it on 19.07 again, your next roadbloack is this:

root@OpenWrt:~# lxc-checkconfig

--- Namespaces ---

Namespaces: required

Utsname namespace: missing

Ipc namespace: required

Pid namespace: required

User namespace: missing

Network namespace: missing

--- Control groups ---

Cgroups: missing

Cgroup v1 mount points:

Cgroup v2 mount points:

Cgroup v1 systemd controller: /usr/bin/lxc-checkconfig: line 167: printf \033[1;31m: not found

Cgroup v1 freezer controller: /usr/bin/lxc-checkconfig: line 174: printf \033[1;31m: not found

Cgroup namespace: requiredCgroup device: missing

Cgroup sched: missing

Cgroup cpu account: missing

Cgroup memory controller: missing

--- Misc ---

Veth pair device: enabled, loaded

Macvlan: enabled, not loaded

Vlan: enabled, not loaded

Bridges: enabled, not loaded

Advanced netfilter: enabled, not loaded

CONFIG_NF_NAT_IPV4: enabled, loaded

CONFIG_NF_NAT_IPV6: enabled, not loaded

CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded

CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded

CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loadedCONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded

FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---

checkpoint restore: missing

CONFIG_FHANDLE: missing

CONFIG_EVENTFD: enabled

CONFIG_EPOLL: enabled

CONFIG_UNIX_DIAG: missing

CONFIG_INET_DIAG: missing

CONFIG_PACKET_DIAG: missing

CONFIG_NETLINK_DIAG: missing

File capabilities: enabled

Note : Before booting a new kernel, you can check its configuration

usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig

anon50098793 · June 28, 2021, 6:34pm

19.x supports debootstrap(chroot) but needs some massaging... there are some posts on the forum about it

for LXC you have to jump through 99 hoops in a buildroot... there are some posts about this too...

camus · June 28, 2021, 11:43pm

@erdoukki thanks. but I've installed, by running this command:

opkg install liblxc luci-app-lxc lxc lxc-attach lxc-auto lxc-autostart lxc-cgroup lxc-checkconfig lxc-common lxc-config lxc-configs lxc-console lxc-copy lxc-create lxc-destroy lxc-device lxc-execute lxc-freeze lxc-hooks lxc-info lxc-init lxc-ls lxc-monitor lxc-monitord lxc-snapshot lxc-start lxc-stop lxc-templates lxc-top lxc-unfreeze lxc-unprivileged lxc-unshare lxc-user-nic lxc-usernsexec lxc-wait rpcd-mod-lxc

And all the other commands before that last command from your guide.

How can I remove all of that. I'm keeping the router running until, well, until you assure me on what to do. And for that, I thank you, very much.
Camus

camus · June 28, 2021, 11:47pm

@anon50098793 thanks a lot for that. as I'm rather new to all this, and without any mention of the oil I'd need to use to do the massaging, I'd leave for the experienced folks to play with. But I like that this thread to @erdoukki HowTO has much interest in the community. LMS is a great server and I hope to see the day it's made native to OpenWrt, with a luci app and all. That would be most, most wonderful indeed. I keep hoping a dev would see my many similar pleas here and, well, just do it!

erdoukki · June 29, 2021, 6:24am

you just have to install a new firmware file, which will replace your all system, so no need of remove the actual system...
just switch 19.07 with 21.02 ( take care that is still a development phase with the actual Release Candidate ) !

if you really need to clean your actual 19.07.x,
just do opkg remove on package you have installed !

LXC is not supported without building your own firmware in 19.07.x
It will be supported by default in 21.02...