(WIP) Porting OpenWrt to Cavium CN6640-SNIC10E Octeon II NIC

I appreciate the offer to send a card, but I'm in Canada. If shipping isn't too bad I can shoot you some money. If not don't worry about it. I was just impressed with your project.

I'm in Vermont; if you're in eastern Canada I'd be happy sending you one.

Nova Scotia. I just signed up to this forum today for the purpose of saying you have a cool project, and I'm having a bit of trouble figuring out how to send you a private message.

Hi, i have cn6880 32core and 16GB ram with quad sfp+ slots and i would be happy to play with it and openwrt. do You have working sfp+ transceivers?

You will have working SFP+ transceivers. Some work was done to get CN6880's SFP ports to work, assuming you don't have a XAUI chip that needs drivers. Do you see any chips between your SoC and SFP ports, or cna you post an image?

C.f. this patchlist: https://lore.kernel.org/linux-mips/1439472106-7750-2-git-send-email-aaro.koskinen@nokia.com/

Do you know how to use the Octeon SDK to control this card over PCIe? If so, send me a photo of what you have and I'll spin you up a build.

Other notes: Appears that the VSC8490 PHY is used in the EdgeRouter Infinity. Their GPL sources show the use of a vsc8490.c on a newer kernel -- since these drivers were also written by Cavium they may give some insight into more sane applications of a Vitesse PHY not requiring a massive changeset.

I'll also add this -- I've been a bit of a fool. I thought for some reason that XAUI was the protocol from the "PHY" to the SFP transceiver's actual PHY in certain cases -- instead it's from the SoC to the "PHY"; SFI/XFI is from the "PHY" to the transceiver. This makes a lot more sense on e.g. a big board where you need to communicate with a large number of different SFP transceivers. In this sense it acts much more like an actual PHY.

XAUI is just a four-lane 8b/10b for longer interconnects; the VSC8488 can turn it into XFI or SFI as needed.

Hi, thanks to Your reply, i don't have physical access to my server until weekend, but i have a photo of my card

Do get me a photo of the underside too, please. Thanks.

I suspect the heatsink is thermal adhesive'd to the SoC, so no use trying to pull it open to get a look at what, if any, PHY there is under there.

Good to see that you have a serial connector on it already. You should be able to get into U-boot that way; do please report what comes up when you do load it up (remember to blacklist liquidio if you are using Linux).

Hello there,

I hope this is the right forum....

I have bought a "10GB Dual Port CAVIUM CN6640-SNIC10E-1.1-G PCI-E" on ebay. I guess this has been a mistake for my plan to upgrade my network to 10GB, because I cannot get it to work.

Should I trash it, or can I somehow get it to run under Win10 or Ubuntu 20.04?

Help appreciated. Thanks in advance.

It's not really the right forum, but let me explain the situation to you.

Under Linux, the liquidio driver written by Cavium is supposed to push the firmware /usr/lib/firmware/liquidio/lio_210sv_nic.bin to the card. This is a MIPS64 ELF executable built using Cavium's SDK's "Cavium Simple Executive"; it initializes any hardware that the U-boot bootloader hasn't already set up, and is supposed to talk to the liquidio Linux driver to create virtual interfaces.

However, on this card, the driver fails to load it properly. There are multiple issues; one is that the cards you will find on the internet don't have a U-boot version capable of receiving the firmware. Replacing U-boot allows the driver to get further -- see the Reddit thread above where I've described those issues in part. Still, I haven't gotten it to work with the driver and lack the knowledge, resources and time to fix that. Frankly, it's a lost cause. I haven't tracked out every issue to the point where I can get the firmware to start correctly.

My goal in this thread is to build OpenWrt properly on the device. In the far future this could possibly allow me to replace the firmware liquidio uses, so that the card can boot full Linux, more like /lib/firmware/liquidio/lio_23xx_vsw.bin, which uses a Linux kernel and a Simple Executive working cooperatively on a specific LiquidIO card; working cooperatively avoids the overhead of Linux system calls, but get the development advantages of Linux to e.g. reload stalled cores.

I don't ever plan to reproduce that cooperatively-run environment. I don't even know where I'd start. The Simple Executive and the documentation to build it aren't GPL-licensed. In addition, Cavium has so little interest in actually maintaining the drivers for their cards that their octeon-ethernet driver was briefly dropped from the Linux kernel's staging tree.

Thank you for the insights. I consider it a lost cause then. Will resell my adapter on ebay as "defective" or "expert only". Maybe I'll find someone who enjoys the hassle.

Totally a level-headed response. Best of luck!

First of all, let me thank you for the work you've done on this already. I have ordered a few of those cards on eBay as well, hoping one day they can replace my current HA router setup.

I took a look at your snic10e-ethernet branch, and based my own snic10e branch on it. Since master will move to kernel 5.10, I decided to move everything to 5.10 first.

I modified the VSC848x driver to compile with newer kernels, but some things were over my head, so those parts of the driver are commented out for now.

For those who want to try, here's how to boot OpenWrt on the device via the Octeon SDK.

  1. Prepare
git clone https://github.com/Hurricos/OCTEON-im8724-SDK.git
cd OCTEON-im8724-SDK.git
source ./env-setup OCTEON_CN63XX
  1. Build u-boot
cd bootloader/u-boot/
make octeon_snic10e_config
  1. Create u-boot env file

For some reason I'm unable to abort autoboot via the console. To work around this, create a file named env, with the following content:

bootdelay=-1

  1. Start u-boot via oct-remote-boot
export OCTEON_REMOTE_PROTOCOL=PCI:0
oct-remote-boot --envfile=env u-boot-octeon_snic10e.bin
  1. Load and boot the OpenWrt image built from my snic10e-5.10 branch
oct-remote-load 0x20000000 /home/stijn/openwrt-octeon-snic10e-initramfs-kernel.bin
oct-remote-bootcmd "bootoctlinux numcores=8"

Hope this helps other people who want to experiment with these cards.

1 Like

Looks like the fix for the hang on XAUI init is also relevant for the SNIC10E. Updated my staging tree.

Let me start by saying: Feel free to submit code into my trees; you may use https://git.laboratoryb.org/user/sign_up or just e-mail me patches.

Yes, Aaro Koskinen's patchset for the CN68XX series also contains one patch to prevent that hang on XAUI init. I believe there are some hardware queues that get initialized by that reset. Not sure. There are also some PKO / SSO differences between CN66XX and CN68XX. What that actually means, I do not know; I am just repeating the complaints in the response to that patchset.

If I find time I will pull your 5.10 branch and compile and take a look. Porting forward VSC848X was completely over my head: the use of the OF_MEMORY_ACCESSOR framework does not port forward easily as the concepts have been completely paved over with a different way of dealing with / mapping OpenFirmware / Device Tree objects into memory. I never really understood that part of that patch.

I essentially used the same setup as you did, but rather than overriding using u-boot env, I used the control scheme suggested in the documents in the SDK: based on the U-boot of stderr, stdout, stdin, you can control whether the card will listen on the physical serial console in the top right corner, the PCIe console (oct-pci-console I believe?), etc. Once linux boots, it makes its own determinations about which consoles to listen to and at which baudrate.

Specifically, though, I set up the includes/snic10e.h in the SDK's u-boot tree so that it defaults stdin / stdout / stderr properly to the PCI console.

That all having been said -- there is some overriding going on with the aforementioned variables that I haven't gotten to the root of, so that's probably what you hit upon. I wonder if by setting the variables correctly in the envfile, you can force which console will work and how ...

1 Like

Scratch needing to send me any patches, I'll just pull from your remote and you can pull from mine.

The working tree now lives here: https://gitlab.com/stintel/openwrt/-/tree/snic10e-5.10

2 Likes

FYI: moved my tree to https://codeberg.org/stintel/openwrt/src/branch/snic10e-5.10.