(WIP) Porting OpenWrt to Cavium CN6640-SNIC10E Octeon II NIC

Do get me a photo of the underside too, please. Thanks.

I suspect the heatsink is thermal adhesive'd to the SoC, so no use trying to pull it open to get a look at what, if any, PHY there is under there.

Good to see that you have a serial connector on it already. You should be able to get into U-boot that way; do please report what comes up when you do load it up (remember to blacklist liquidio if you are using Linux).

Hello there,

I hope this is the right forum....

I have bought a "10GB Dual Port CAVIUM CN6640-SNIC10E-1.1-G PCI-E" on ebay. I guess this has been a mistake for my plan to upgrade my network to 10GB, because I cannot get it to work.

Should I trash it, or can I somehow get it to run under Win10 or Ubuntu 20.04?

Help appreciated. Thanks in advance.

It's not really the right forum, but let me explain the situation to you.

Under Linux, the liquidio driver written by Cavium is supposed to push the firmware /usr/lib/firmware/liquidio/lio_210sv_nic.bin to the card. This is a MIPS64 ELF executable built using Cavium's SDK's "Cavium Simple Executive"; it initializes any hardware that the U-boot bootloader hasn't already set up, and is supposed to talk to the liquidio Linux driver to create virtual interfaces.

However, on this card, the driver fails to load it properly. There are multiple issues; one is that the cards you will find on the internet don't have a U-boot version capable of receiving the firmware. Replacing U-boot allows the driver to get further -- see the Reddit thread above where I've described those issues in part. Still, I haven't gotten it to work with the driver and lack the knowledge, resources and time to fix that. Frankly, it's a lost cause. I haven't tracked out every issue to the point where I can get the firmware to start correctly.

My goal in this thread is to build OpenWrt properly on the device. In the far future this could possibly allow me to replace the firmware liquidio uses, so that the card can boot full Linux, more like /lib/firmware/liquidio/lio_23xx_vsw.bin, which uses a Linux kernel and a Simple Executive working cooperatively on a specific LiquidIO card; working cooperatively avoids the overhead of Linux system calls, but get the development advantages of Linux to e.g. reload stalled cores.

I don't ever plan to reproduce that cooperatively-run environment. I don't even know where I'd start. The Simple Executive and the documentation to build it aren't GPL-licensed. In addition, Cavium has so little interest in actually maintaining the drivers for their cards that their octeon-ethernet driver was briefly dropped from the Linux kernel's staging tree.

Thank you for the insights. I consider it a lost cause then. Will resell my adapter on ebay as "defective" or "expert only". Maybe I'll find someone who enjoys the hassle.

Totally a level-headed response. Best of luck!

First of all, let me thank you for the work you've done on this already. I have ordered a few of those cards on eBay as well, hoping one day they can replace my current HA router setup.

I took a look at your snic10e-ethernet branch, and based my own snic10e branch on it. Since master will move to kernel 5.10, I decided to move everything to 5.10 first.

I modified the VSC848x driver to compile with newer kernels, but some things were over my head, so those parts of the driver are commented out for now.

For those who want to try, here's how to boot OpenWrt on the device via the Octeon SDK.

  1. Prepare
git clone https://github.com/Hurricos/OCTEON-im8724-SDK.git
cd OCTEON-im8724-SDK.git
source ./env-setup OCTEON_CN63XX
  1. Build u-boot
cd bootloader/u-boot/
make octeon_snic10e_config
  1. Create u-boot env file

For some reason I'm unable to abort autoboot via the console. To work around this, create a file named env, with the following content:

bootdelay=-1

  1. Start u-boot via oct-remote-boot
export OCTEON_REMOTE_PROTOCOL=PCI:0
oct-remote-boot --envfile=env u-boot-octeon_snic10e.bin
  1. Load and boot the OpenWrt image built from my snic10e-5.10 branch
oct-remote-load 0x20000000 /home/stijn/openwrt-octeon-snic10e-initramfs-kernel.bin
oct-remote-bootcmd "bootoctlinux numcores=8"

Hope this helps other people who want to experiment with these cards.

1 Like

Looks like the fix for the hang on XAUI init is also relevant for the SNIC10E. Updated my staging tree.

Let me start by saying: Feel free to submit code into my trees; you may use https://git.laboratoryb.org/user/sign_up or just e-mail me patches.

Yes, Aaro Koskinen's patchset for the CN68XX series also contains one patch to prevent that hang on XAUI init. I believe there are some hardware queues that get initialized by that reset. Not sure. There are also some PKO / SSO differences between CN66XX and CN68XX. What that actually means, I do not know; I am just repeating the complaints in the response to that patchset.

If I find time I will pull your 5.10 branch and compile and take a look. Porting forward VSC848X was completely over my head: the use of the OF_MEMORY_ACCESSOR framework does not port forward easily as the concepts have been completely paved over with a different way of dealing with / mapping OpenFirmware / Device Tree objects into memory. I never really understood that part of that patch.

I essentially used the same setup as you did, but rather than overriding using u-boot env, I used the control scheme suggested in the documents in the SDK: based on the U-boot of stderr, stdout, stdin, you can control whether the card will listen on the physical serial console in the top right corner, the PCIe console (oct-pci-console I believe?), etc. Once linux boots, it makes its own determinations about which consoles to listen to and at which baudrate.

Specifically, though, I set up the includes/snic10e.h in the SDK's u-boot tree so that it defaults stdin / stdout / stderr properly to the PCI console.

That all having been said -- there is some overriding going on with the aforementioned variables that I haven't gotten to the root of, so that's probably what you hit upon. I wonder if by setting the variables correctly in the envfile, you can force which console will work and how ...

1 Like

Scratch needing to send me any patches, I'll just pull from your remote and you can pull from mine.

The working tree now lives here: https://gitlab.com/stintel/openwrt/-/tree/snic10e-5.10

2 Likes

FYI: moved my tree to https://codeberg.org/stintel/openwrt/src/branch/snic10e-5.10.

2 Likes

Another FYI: people wanting to use the Octeon SDK on musl hosts can use my fixes branch.

Hi, i bought one of these boards (CN6640 cavium). When I try to execute this command "oct-remote-boot --envfile = env u-boot-octeon_snic10e.bin" I get this error:

Using environment from file: env
PCIE port 0
All cores in reset, skipping soft reset.
Using bootloader image: u-boot-octeon_snic10e.bin
N0.LMC0 Configuration Completed: 2048 MB
Initialized 2048 MBytes of DRAM
Unable to open file: No such file or directory
ERROR copying file u-boot-octeon_snic10e.bin to Octeon DRAM

Do you have any ideas on how to fix?

Hey where i can find the file "u-boot-octeon_snic10e.bin"?

Build it following these instructions.

Thank for the replay. i can't build the openwrt image (openwrt-octeon-snic10e-initramfs-kernel.bin) can you send a link to download it? I am new in openwrt. (if you dont want to send here , you can send at my reddit u/Shiny1712). Thanks.

Hello, so after all is this a good idea? To use these cavium cards with openwrt?

If you expect everything to just work, I would recommend you stay away from these cards. If you're willing to get your hands dirty, you can get them to work.

The NOR flash isn't large enough for an OpenWrt image, and afaik there is no working NAND driver (and some cards don't even have NAND), so the only way to actually use them is to load an initramfs via PCI. You can use the NOR flash as overlay to have persistent settings, but it's not ideal.

2 Likes

This thread is about running OpenWrt on CN6640-SNIC10E. Your question is about using different hardware as a NIC, so it's entirely unrelated. Please delete these posts here and open a new thread.