My "workaround" on NanoPi R6S with vanilla OpenWrt

For Developers
1

Screenshot 2023-01-31 03.41.56
Screenshot 2023-01-31 03.41.561920×1594 240 KB

The NanoPi R6S with dual 2.5G LAN + 1GB LAN, and 8-core CPU + 8GB ram is a really great device, however no official OpenWrt build yet, same as most people here I also want to use the vanilla one, and here is what I did:

Use DietPi as the underlying OS, it has FriendlyElec kernel (according to DietPi team) but a trim down version of Debian Bullseye.

Install ARM version Proxmox (PiMox 7), grab the OpenWrt ARM64 rootfs from release page to run as container! It works well for me.

And then tested a passthrough of USB WiFi and it also worked so this part is quite successful.

I didn't assign all resources to OpenWrt LXC, so now I can add something like PiHole, and maybe Tailscale (tried to run TS directly on OpenWrt but it doesn't work well with nftables, a separate container might be better), and something I couldn't think of at this moment.

3 Likes
2

Hello!

That's impressive, thank you for sharing! It would be interesting to know if you have tested the throughput of ethernet interfaces, it would be normal not to have 100% or the maximum values but if the loss is not too high, your solution is a good one. Maybe having FriendlyElec kernel is something not totally desiderable - I have read somewhere that there are some security issues, see here:

Thanks!!

3

I haven't got time to do a full test due to out of country this week, but technically it should be close to 100% because this is a container which has very little overhead.

1 Like
4

I really like this idea but am having trouble accomplishing this on my r6s.
I've installed the most basic DietPi for the r6s via SD, then properly copied it to MMC.
Then after disabling ipv6, using a static IP, and making sure the SSH server is set to OpenSSH, I follow the manual installation instructions here:

Which bombs out because:

dpkg: dependency problems prevent configuration of proxmox-ve:
 proxmox-ve depends on pve-manager; however:
  Package pve-manager is not configured yet.

Now, step 6 says "Make sure the kernel-headers are installed." which is impossible for this build afaik, but that doesn't seem to be the hold up. Any clarification as to how you got this on there and running?

Edit
Spoke too soon. Should have googled a bit more. Fixed by making my /etc/hosts file look like this:

root@r6s:~# cat /etc/hosts
#127.0.0.1 localhost
192.168.1.5 r6s pvelocalhost
#::1 localhost ip6-localhost ip6-loopback
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters
5

Yes, the PiMox 7 install script is unable to change hostname on DietPi (haven't checked why), so you can change the DietPi's hostname manually before installation.

6

One more note, on my setup the default r8169 module couldn't bring up the 2.5G NIC properly, I have to blacklist it and use r8125.

7

Thank you for the extra help!
I just had a look at the modules and will pick this back up on the weekend. I could be wrong but it does seem odd to me that it even tries using the r8169 module since the r6s has 2x RTL8125BG and 1x RTL8211F (1Gb). I didn't poke around with any of the other friendlyarm images outside of the openwrt one, so I don't know what modules their images include, but blacklisting the r8169 module makes sense.

root@r6s:~# lspci
0003:30:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0004:40:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3588 (rev 01)
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
root@r6s:~# lspci -vs 03:31:00 | awk '!/Cap/'
0003:31:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Device tree node: /sys/firmware/devicetree/base/pcie@fe180000/pcie@30/pcie@30,0
        Flags: fast devsel, IRQ 116
        I/O ports at 100000 [size=256]
        Memory at f3200000 (64-bit, non-prefetchable) [size=64K]
        Memory at f3210000 (64-bit, non-prefetchable) [size=16K]
        Kernel driver in use: r8169
        Kernel modules: r8169, r8125

root@r6s:~# lspci -vs 04:41:00 | awk '!/Cap/'
0004:41:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
        Subsystem: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller
        Device tree node: /sys/firmware/devicetree/base/pcie@fe190000/pcie@40/pcie@40,0
        Flags: fast devsel, IRQ 106
        I/O ports at 1000 [size=256]
        Memory at f4200000 (64-bit, non-prefetchable) [size=64K]
        Memory at f4210000 (64-bit, non-prefetchable) [size=16K]
        Kernel driver in use: r8169
        Kernel modules: r8169, r8125
root@r6s:~# dmesg | grep eth0
[    8.181042] rk_gmac-dwmac fe1c0000.ethernet eth0: PHY [stmmac-1:01] driver [RTL8211F Gigabit Ethernet] (irq=POLL)

I'm thinking the r8169 module was included in the dietpi image for other devices.. like the R4S, which uses an R8111H

8

I tried to search around and found that normally r8169 should also cover Realtek 8125B, however I found it never works on my DietPi setup, so eventually I just blacklist it and go straight to r8125 and then it works!

3 Likes
9

I am wondering how much is the maximum SQM pull?

10

There is a FriendlyWRT version for the R6S? I read a review the other day that said they installed the FriendlyWRT version. The latest version is based on OpenWRT 22.03.3 (or maybe it's 22.03.2 - don't remember for sure) So, why not run that until the R6S is supported in OpenWRT? Honest question.

11

In my case it’s because of security. I don’t know if a manufacturer-made operating system/firmware has some… “uncontrolled” code (maybe that’s not the most correct expression, but I guess you get the idea). I’ll have my internal network and personal data behind the router, protected by a vpn that I will configure by myself, and don’t want to leave any loose end. I trust openwrt but I don’t totally trust any other firmware made by manufacturers.

But if I’m wrong, please anyone correct me. Maybe Friendlyelec firmware is reliable in terms of security.

1 Like
12

Running the vanilla OpenWrt, in case FriendlyElec doesn't update fast enough in future when new version comes, I can still use this way to update. But of course the best would be having mainline linux kernel support so that vanilla OpenWrt can do bare metal install on it.

13

That's what I needed to know. Thanks!

14

Could you share your experience, please.
How did you install OpenWrt LXC on ARM version Proxmox (PiMox)?

I tried this article
https://virtualizeeverything.com/2022/05/23/setting-openwrt-in-proxmox-lxc/

pct create 201 /var/lib/vz/template/cache/OpenWRT.tar.xz \
--arch arm64 \
--hostname OpenWrt-21.02 \
--rootfs local:201 \
--memory 1024 \
--cores 2 \
--ostype unmanaged \
--unprivileged 1

but on PiMox unsuccess.

What image of OpenWrt did you use?
What command did you to create a LXC?
Thank you for your answers.

15

During my time of testing, I downloaded the 22.03.3 default root FS images here.

And this was how I created the LXC (which looks almost the same as yours):

pct create 102 /var/lib/vz/template/cache/openwrt-22.03.3-armvirt-64-default-rootfs.tar.gz --arch arm64 --hostname openwrt-lxc --rootfs local:1 --memory 2048 --cores 2 --ostype unmanaged --swap 0 --unprivileged 1

So what error you've got? Cannot create the container or it doesn't boot?

For PiMox 7 I used DietPi Bullseye image (at that time FriendlyElec didn't have Bullseye image), now I can see that Armbian has the image as well, just build up from there. However at that time the 2.5G NIC didn't work with r8169 driver, which I had to blacklist it and use r8125 instead.

16 
pct create 201 \
/var/lib/vz/template/cache/openwrt-22.03.3-armvirt-64-default-rootfs.tar.gz \
--arch arm64 \
--hostname openwrt-lxc \
--rootfs local:1 \
--memory 2048 \
--cores 2 \
--ostype unmanaged \
--swap 0 \
--unprivileged 1;

echo $?;
return 0
lxc added

then I had added NICs and run
Console:
Connected to tty 1
Type <Ctrl+4 q> to exit the console, <Ctrl+4 Ctrl+4> to enter Ctrl+4 itself

black screen without text

Summary:
Status running
HA State none
Node pve
CPU usage 0.03% of 2 CPU(s)
Memory usage 0.32% (6.49 MiB of 2.00 GiB)
SWAP usage N/A
Bootdisk size 1.36% (13.23 MiB of 973.42 MiB)

17

Hardware:
Orange PI 5 16GB
NVMe
2 x USB NIC

OS:

Armbian_23.02.2_Orangepi5_bullseye_legacy_5.10.110.img.xz

NVMe setting:

Pimox installation:
https://www.reddit.com/r/Proxmox/comments/104z8pt/proxmox_arm_running_on_the_orange_pi_5_with_nvme/

XLC test from this article:

Works

Virtual Machive OpenWrt arm64 uefi

Works

18

I think unless you compile your own kernel, otherwise the shipped kernel usually doesn't have KVM support which will cause huge performance hit on VM.

By default there is no video output so normal PVE console will show you nothing even you are trying to view it like other LXC/VM, you need to use other command to enter into the container console directly.

19

Thank you for your answers.
I will try to build kernel with KVM support by these manuals:
https://docs.armbian.com/Developer-Guide_Build-Preparation/
https://www.shincbm.com/linux/kvm/arm64/2019/03/22/arm64-kvm-armbian.html

One more questions:

  1. How did you use pihole, as docker container inside openwrt?
  2. What usb wifi device do you use?
  3. How did you passthrough usb to lxc?
20

pct enter 201

BusyBox v1.35.0 (2023-01-03 00:24:21 UTC) built-in shell (ash)

~ #

uname -a
Linux openwrt-lxc 5.10.110-rockchip-rk3588 #23.02.2 SMP Fri Feb 17 23:59:20 UTC 2023 aarch64 GNU/Linux

Connected to openwrt lxc to use command:
pct enter container_id