RB5009UPr+S+IN PCIe/NVME

For Developers
1

Hi all together,

Firstly, despite having long experience in developing embedded systems, this is my very first project with OPENWRT and I never build an OPENWRT image up to now.

Secondly - let me describe my goals:

I am working for a company, which is building airborne geophysical exploration devices and i am mainly developing hard and software for this kind of systems.
For an autonomous and spatially distributed sensor network which would use ETHERNET/POE for communication with and powering the connected sensors, we where looking for an "intelligent" POE-switch with a certain computing power which enables it to be the central data acquisition device for the sensor network running our own acquisition software.
The question was whether to develop our own hardware or to find an off-the-shelve device which suits (most of) our requirements.

The closest fit in hardware was the RB5009 with its powerful MARVELL ARMADA, the 120W POE option, USB 3.1 storage and the opportunity to run OPENWRT as LINUX OS which could also be able to run our own software.

So we ordered one devices to test the flashing and to test the USB storage speed.

I must say, it was relatively easy to get OPENWRT (release 24.10.0) running and i was a bit pleased, how easy it is to install additional software packages using OPKG.

At this point, i must say, that USB storage would not be our preferred storage option but as the speed is good enough, we would come around with that.

Much more interesting would be a PCIe/NVME disk and as i have seen that there is an unpopulated NVME header i was quite happy :wink:

So i ordered the appropriate connector and put it onto the board adding some R's and C's to be ready for an NVME test.

But unluckily, there is obviously no PCIe/NVME support in the kernel :frowning:

So my big questions would be:

  1. How can i get PCIe/NVME support (for ARMADA 7040) in the kernel image?
  2. Or, is there someone who could compile me an image with this stuff in?
  3. Has anybody else tried to get the NVME on RB5009 working?

Any help or any hint is appreciated!

Many thanks in advance and best regards,

Marco.

2

Never tried NVME

kmod-nvme is a kernel module that provides support for NVM Express (NVMe) block devices, which are typically PCIe-based. You can download and install it thru openwrt.
Let me how it works out.

3

Hi SteeledPick,

I already tried to install the kmod-nvme package using the following command after "opkg update":

opkg install kmod-nvme
Installing kmod-nvme (6.6.73-r1) to root...
Downloading https://downloads.openwrt.org/releases/24.10.0/targets/mvebu/cortexa72/kmods/6.6.73-1-7061dcd9b893eb42c2b98c42b2e51dee/kmod-nvme_6.6.73-r1_aarch64_cortex-a72.ipk
Installing kmod-hwmon-core (6.6.73-r1) to root...
Downloading https://downloads.openwrt.org/releases/24.10.0/targets/mvebu/cortexa72/kmods/6.6.73-1-7061dcd9b893eb42c2b98c42b2e51dee/kmod-hwmon-core_6.6.73-r1_aarch64_cortex-a72.ipk
Configuring kmod-hwmon-core.
Configuring kmod-nvme.

Then i tried to start the module with "modprobe nvme"

But when I look for a running NVME module using "lsmod | grep nvme", i get nothing.

Also listing the installed package files from kmod-nvme shows no kernel modules:

opkg files kmod-nvme
Package kmod-nvme (6.6.73-r1) is installed on root and has the following files:
/etc/modules.d/30-nvme
/etc/modules-boot.d/30-nvme

"lspci" doesn't even give any controller hardware!

So i think there is no pci(e) support build in...

Best regards, Marco.

4

It was worth a shot. Check the logs and grep to see if any "pcie down" ............hmmm.
Now I want to get a router with an nvme slot

5

the only sentence concerning pci i get out of dmesg:

dmesg | grep -i pci
[    0.024589] PCI: CLS 0 bytes, default 64

there is no controller mentioned - nothing...

So my intention was to build my own image trying to get the pci and nvme support in the kernel.

I already cloned the OPENWRT git repo and tried to start with a somehow "standard" config for RB5009 (from which the openwrt-24.10.0-mvebu-cortexa72-mikrotik_rb5009-squashfs-factory.ubi or the openwrt-24.10.0-mvebu-cortexa72-mikrotik_rb5009-squashfs-sysupgrade.bin are made from) but I have no real glue how to do this.

In the menuconfig i can find the processor system and then the rb5009 profile but when i start with this all other config are more or less empty. So i dont know which configurations are mandatory...

6

Well, you gotta enable PCIe first in the DTS, but for that you must also know which SERDES lane is connected there in order to change its function to PCIe

1 Like
7

Hi robimarko,

i made slight progress reverse engineering my RB5009 :slight_smile:

  1. I was able to build my own openwrt image with changed LINUX-DT.
  2. I found the PERST pin using gpiod. It is line 12 on the gpiochip2 (mpp43).

For testing i put the following node into DT:

&cp0_pcie2 {
    status = "okay";
    phys = <&cp0_comphy5 2>;
    phy-names = "cp0-pcie2-x1-phy";
    reset-gpio = <&cp0_gpio2 12 GPIO_ACTIVE_HIGH>;
};

But, when the pcie is initialized with comphy mode 2 on one of the free comphys (1,2,4,5 - comphy 2 is SFI0, comphy3 is USB3), i get either an error (-22) (on comphy 1,2,4) or the boot process got stuck (on comphy5).

Is ther perhaps a missing initialization? I read somewhere, that the comphys of ARMADA 7040 have to be initialized prior to kernel startup e.g. in the UBOOT?

8

U-Boot will print during boot what the SERDES lanes are configured to, probably that lane is either used for something else or it needs to be configured