NanoPi R4S-RK3399 is a great new OpenWrt device

For Developers
1280

Docker is something OpenWrt really needs on opkg since these new devices are much more CPU/RAM/storage overhead now. The FriendlyWrt builds and @anaelorlinski build both have it, it's a game changer.

Which Ubiquiti image are you using? Assuming this is for unify-controller, is it this one? [https://hub.docker.com/r/jacobalberty/unifi] I'm looking to do this since I run a U6-Lite on my network.

edit: okay just didn't know OpenWrt had this page until I searched: https://openwrt.org/docs/guide-user/virtualization/docker_host this is great.

1281

one word of warning thou. when you want to update the images? do it from an ssh window. the luci window times out after 90seconds or so. Image pulls can take 2-5mins. its 750mb.

once the image is downloaded you can use the luci plugin instead.

1 Like
1282

Thanks for the tip, looks like a fun project for this weekend

1 Like
1283

oh and don't forget to change docker to using your 3rd partition. I mount it as /opt and let docker use /opt/docker.

Info
Docker Version 20.10.18
Api Version 1.41
CPUs 6
Total Memory 3.78 GB
Docker Root Dir /opt/docker (23.96 GB Available)
Index Server Address https://index.docker.io/v1/
1 Like
1284

also I ran into this little gotcha with the last unifi upgrade. I believe its a clash with the new java vm?

you need to remove the memory limit options in order for it to boot. take out both the optional tagged environment options from either your yaml file or docker cli command.

:edit: also to simplify things I run the controller in host mode

    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
      - MEM_LIMIT=1024 #optional
      - MEM_STARTUP=1024 #optional
2 Likes
1285

Thanks this is all great info, what is the exact command you enter from SSH to pull the image if you don't mind? Never pulled a large image like this before, (I'm going to put the info in the Openwrt docker wiki too).

edit: thinking it's simply: docker pull lscr.io/linuxserver/unifi-controller:latest

I'm wondering if there are any smaller/more optimized images for unify-controller on arm64 like the first link I posted, I'm going to check the size of a few of them, but will start with the one you use.

1286

same as regular docker commands.

docker pull lscr.io/linuxserver/unifi-controller

root@OpenWrt:/opt# docker pull lscr.io/linuxserver/unifi-controller 
Using default tag: latest
latest: Pulling from linuxserver/unifi-controller
e23f04a611b4: Pull complete 
0e05dbb9a313: Pull complete 
2d3b5e7ce942: Pull complete 
ac9f18d9fe68: Pull complete 
9565b61600cd: Downloading [==============================>                    ]  200.6MB/332.3MB
5621d1988847: Download complete 
a1e613774ba0: Download complete

I think I tried a different image at start. I think hot.io but swapped to linuxserver as they also do other images I use elsewhere. The image works for me and until it stops working I prob won't change.

1 Like
1287

I have played around with the custom images today and I learned a few things that might be interesting to other people who want to switch from FriendlyWRT:

1. You can't install any kernel module packages

This is the biggest deal breaker for me.

I installed @anaelorlinski's R4S custom image and then I tried to install the package kmod-sched-ctinfo for SQM with QoS... and I got an error like...

Collected errors:
 * satisfy_dependencies_for: Cannot satisfy the following dependencies for kmod-sched-ctinfo:
 * 	kernel (= 5.10.x-XXXXXXXXXX) * 
 * opkg_install_cmd: Cannot install package  kmod-sched-ctinfo.

It turns out OpenWRT prevents you from installing kernel modules that were compiled with a different config than the kernel.

You can read more about vermagic mismatches in this blog post.

It means that if you use a custom build for your R4S, you need to make sure it includes at build time all the kernel modules you may want to use in the future.

So that's probably only possible if you build the image yourself. And if you ever need to add another module, you will have to rebuild the image.

If you match the config of the official openwrt builds exactly, you may get a kernel that has a matching vermagic hash, but I don't know if that's going to be possible or if it would cause instability since we do want to actually patch the kernel in the custom builds (that's why we are running custom builds in the first place).

Perhaps, I will try to make that work. But not sure if it's worth the effort.

Expanding storage is unstable

I had a lot of trouble with expanding storage. Many times the R4S would not come online after expanding the storage.

Sometimes it worked and sometimes not.

And that's problematic, because...

You have to expand the storage every time you update/reflash the firmware

When you update the firmware in luci, it deletes everything, then restores only the files backed up by sysupgrade.

So you have to expand the storage again and risk breaking the system.

Then you have to find your SD card reader dongle, reflash the firmware, expand again, install the backup...

And having to do this each time you want to install a new kernel module makes it a huge hassle :frowning:

1288

I have asked for kmod-sched-ctinfo to be including in @anaelorlinski next build so hopefully this will fix this issue, but you can use 22.03.2 if needed these work fine, or snapshot works also.

1289

For the 1 could be due to the olpkg sources.

For the point 2, is the same with OpenWrt. You have to build your image already with the needed size.

1 Like
1290

I have the nanopi r4s with 4GB of ram. Do you think it is ok to run omada-controller (it takes some ram ~700MB but almost no cpu) and SWAG? I have it running on a rpi4 on my network but I would like to move these two dockers for the router since swag is used to reverse proxy some other hosts and because I think it makes sense to have omada-controller on a central unit related to the network stuff.

1291

The very latest release (2023.01.06) builds all the kmods and bundle them into the image as a local package feed. Also it is frozen to 22.03.3 commit.

This means that you should be able in many cases to install packages from 22.03.3 official build that rely on kmods without getting this error anymore. Just follow the normal procedure opkg update / opkg install.

The images become slightly bigger but it is a good trade off as the SD card space is cheap, avoiding to bloat the image by default with many installed modules but still having them available around. All the available packages are located in /ipks
Let me know if you get better success. Note that kernel modules built from packages repo are not included (for example antfs will not install), but this should still cover 99% of the usecases.

9 Likes
1292

hey guys

I'm not a R4S / RK3399 or R6S / RK3588 user but
I'm a R2S / RK3328 user.

openwrt v22 + docker on /opt + ext4/overlay fs with SDCard boot works flawlessly with the GA/prod release. I should be sending coffee/beer/Venmo tips to the responsible peoples/devs.
I know how hard life is on the snapshot releases, I did it for long enough that I avoid it in the present, for blood pressure reasons.

If somebody who knows the magic and innards of what the buildbot does, maybe you can take a look at what prevents the 3399 or 3588 from being clean and easy out of the box like the R2S (aarch64 cortex-A53 rk3328).

I am considering an RK3588 purchase (I like the extensive I/O options) but I know the longer I wait the easier it'll be to adopt. Those on the bleeding edge... bleed.

1293

Thank you! This is an AMAZING solution and so far is working very well. It's also only using 14 mb of extra storage, which for the NanoPis with SD cards is no problem at all.

Thank you for sharing your solution with all of us.

1 Like
1294

What is the difference in performance between this version and the official one?

1295

@huangjinqiu:
I just performed 5 reboots in a row on my spare R4S, running the latest SquashFS snapshot built today (2022-01-15) ... no problems at all. With recent builds I never faced any soft reboot problems. I can try it with a build you face problems with if you want.

Maybe it is a problem with your SD card and/or the power supply? Just guessing.

1296

Not sure if it is soft reboot issue or something else. But it does not happen every reboot. My Nanopi R4S reboots at 5am daily. This problem happened this morning again. The LAN led was off (which I think is different from soft reboot issue) and there was no internet. I had to unplug it and do a hard reboot. I do sysupgrade regularly, especially when I encountered this problem. I can't recall the exact build number.

1297

Specific processor compilation options are enabled for each model. Don't expect very high performance boost over official build in general.
There are no overclocking patches like in ImmortalWrt to boost frequency of cores.

1298

just got one today. would have chosen 1gb model if it was available...

OpenWrt 22.03.3, r20028-43d71ad93e
 -----------------------------------------------------
root@OpenWrt:~# cat /proc/meminfo
MemTotal:        3967352 kB
MemFree:         3801208 kB
MemAvailable:    3810160 kB

installed adblock with 140k list, sqm, netdata, stubby. 4gb seems a bit overkill

1299

The price difference is negligible and the extra RAM allows for LOTS of conntrack entries, iptables list rules, etc. - more RAM is never a bad thing. I don't know why they even sell the 1GB model tbh.

2 Likes