(WIP) Porting OpenWrt to Cavium CN6640-SNIC10E Octeon II NIC

For Developers
1

I don't have much backstory I want to repeat.

FAQ:

  • Yes, this board is a NIC. It's a PCIe card.
  • CN6640 (8-core / 1.6GHz MIPS64 Octeon2 processor).
  • 2GiB DDR3
  • 2x SFP+ ports
  • 8MiB NOR, most boards have 1GiB NAND
  • They're junk on eBay -- you can get them for ~$15 individually. I bought 50 for $78.
  • They're supposed to run a Cavium Simple Executive (/lib/firmware/liquidio/lio_210sv_nic.bin) as their firmware under the Linux/*BSD liquidio driver, but secondhand cards will have bootloaders in questionable state and I've never had them boot and work properly with liquidio from a Linux host system. Cavium is known for publishing stuff and not taking care that it's actually usable.
  • This board can [probably] be powered externally but to do so requires the 4-pin power header at the far right to be populated (it's 12V, I'm sure). You could probably then use them as a sick, high-powered 10Gb OpenWrt router if paired with a 10Gb managed switch.

Here are some posts for context. Note I was less informed then than I am now.

The state of OpenWrt:

  • https://git.laboratoryb.org/hurricos/openwrt/src/branch/snic10e-ethernet
    • Builds, boots (initramfs), brings up xaui0 and xaui1, but does not interact with SFP modules.
    • It's fast! Well, kind of. Core-for-core it's nearly as fast as a WR1900ACv1, but this doesn't count any of the crypto accelerators.
  • Use:
    • 6-pin serial in the top right (right-to-left: GND TX RX ? ? ?)
    • Use Cavium's 5.1 SDK to:
      • rebuild U-boot
      • interact with the board from an x86 host via PCIe
      • get a PCIe serial console
      • Get full access to the board's RAM

TODO:

  • [ ] Fixup XAUI in the Octeon ethernet driver
  • [ ] Write and test sysupgrade setup
    • These boards can vary - some do and some do not have a spare 1GB of NAND on the back; the main NOR flash is only 8MB, at least 1.5MB of which are U-boot.
  • [ ] Cleanup commits / squash / submit
8 Likes
2

Some work will need to be done to bring over code which has not made it to the mainline Linux kernel, from:

Examples:

  • Regarding the PHYs: the CN6640-SNIC10E board uses a Vitesse VSC8488 two-port "PHY" (really a XAUI transceiver); these drivers are not mainlined. They are available in a Yocto project tree here. It is no wonder that XAUI0 and XAUI1 do not come up -- they're not even there, despite the DTS telling the kernel they will be configured fine.
3

The VSC848X patch has been merged for testing, and an attempt at merging in the OF_MEMORY_ACCESSOR patch has been done as well. Let's see if it builds.

1 Like
4

No luck. https://paste.c-net.org/TippinStiffs

Re-run with full output: https://paste.c-net.org/TrailingLiberal

This is against https://git.laboratoryb.org/hurricos/openwrt/src/branch/snic10e-ethernet.

To be debugged tomorrow.

5

Breaking down this error list: https://paste.c-net.org/TrailingLiberal

For the struct memory_accessor patch, we have this:

commit bec3c11bad0e7ac05fb90f204d0ab6f79945822b
Author: Andrew Lunn <andrew@lunn.ch>
Date:   Fri Feb 26 20:59:24 2016 +0100

    misc: at24: replace memory_accessor with nvmem_device_read
    
    Now that the AT24 uses the NVMEM framework, replace the
    memory_accessor in the setup() callback with nvmem API calls.
    
    Signed-off-by: Andrew Lunn <andrew@lunn.ch>
    Acked-by: Srinivas Kandagatla <srinivas.kandagatla@linaro.org>
    Tested-by: Sekhar Nori <nsekhar@ti.com>
    Acked-by: Wolfram Sang <wsa@the-dreams.de>
    Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

diff --git a/include/linux/memory.h b/include/linux/memory.h
index 8b8d8d12348e..b723a686fc10 100644
--- a/include/linux/memory.h
+++ b/include/linux/memory.h
@@ -136,17 +136,6 @@ extern struct memory_block *find_memory_block(struct mem_section *);
 #define unregister_hotmemory_notifier(nb)  ({ (void)(nb); })
 #endif
 
-/*
- * 'struct memory_accessor' is a generic interface to provide
- * in-kernel access to persistent memory such as i2c or SPI EEPROMs
- */
-struct memory_accessor {
-       ssize_t (*read)(struct memory_accessor *, char *buf, off_t offset,
-                       size_t count);
-       ssize_t (*write)(struct memory_accessor *, const char *buf,
-                        off_t offset, size_t count);
-};
-
 /*
  * Kernel text modification mutex, used for code patching. Users of this lock
  * can sleep.

This means I will have to rewrite the patch to use the nvmem API instead.

This might explain the conflicting types error on this function signature.

Wow, is this why people hate mainlining drivers? :laughing:

6

I don't have anything constructive to say. I just spotted your effort to port OpenWRT to a Cavium CN6640 NIC, thought it was the coolest thing in the world, and wanted to give you a pat on the back. Keep up the good work. You're awesome!

I think I'll order one of these cards so I can play around. If I can get a terminal over SSH or something I'll be happy.

1 Like
7

Please don't buy one. I have fifty. Send me some contact information and if you're in the lower 48, I'll ship you one.

8

You can get a console over PCIe now. There's no networking -- the only standalone networking interfaces are SFP+.

There are two sets of serial pads (one is populated with female headers); you can always use PPP over the unused serial if you need networking access.

9

Actually, if the drivers had been mainlined, you wouldn't see this problem as the one pushing the API change would have to fix all affected in-tree ocurances of this issue before it hits you.

There are a couple of 'reasons' for not mainlining a driver:

  • patience, respectively the lack thereof. You will get change requests in the submission phase, ranging from cosmetic nitpicking to rather substantial ones (like using a different framework or refactoring some existing functionality into a more generic API)
    • too many hacks involved to get going
    • losing interest/ motivation for the particular hardware (next shiny thing or problems with the hardware making further efforts unreasonable)
    • $employer not granting the time needed for the necessary changes
  • unclear licensing or patent status
  • a wish to remain the sole point of contact (be that commercial interests (support contracts) or merely a need for constant affirmation)
10

I apologize, it was a bit of a joke. With any luck I'll be helping mainline these drivers.

11

…and I really appreciate that, these cards are interesting (o.k., availability in single quantities isn't that good).

12

I appreciate the offer to send a card, but I'm in Canada. If shipping isn't too bad I can shoot you some money. If not don't worry about it. I was just impressed with your project.

13

I'm in Vermont; if you're in eastern Canada I'd be happy sending you one.

14

Nova Scotia. I just signed up to this forum today for the purpose of saying you have a cool project, and I'm having a bit of trouble figuring out how to send you a private message.

16

Hi, i have cn6880 32core and 16GB ram with quad sfp+ slots and i would be happy to play with it and openwrt. do You have working sfp+ transceivers?

17

You will have working SFP+ transceivers. Some work was done to get CN6880's SFP ports to work, assuming you don't have a XAUI chip that needs drivers. Do you see any chips between your SoC and SFP ports, or cna you post an image?

18

C.f. this patchlist: https://lore.kernel.org/linux-mips/1439472106-7750-2-git-send-email-aaro.koskinen@nokia.com/

19

Do you know how to use the Octeon SDK to control this card over PCIe? If so, send me a photo of what you have and I'll spin you up a build.

20

Other notes: Appears that the VSC8490 PHY is used in the EdgeRouter Infinity. Their GPL sources show the use of a vsc8490.c on a newer kernel -- since these drivers were also written by Cavium they may give some insight into more sane applications of a Vitesse PHY not requiring a massive changeset.

I'll also add this -- I've been a bit of a fool. I thought for some reason that XAUI was the protocol from the "PHY" to the SFP transceiver's actual PHY in certain cases -- instead it's from the SoC to the "PHY"; SFI/XFI is from the "PHY" to the transceiver. This makes a lot more sense on e.g. a big board where you need to communicate with a large number of different SFP transceivers. In this sense it acts much more like an actual PHY.

XAUI is just a four-lane 8b/10b for longer interconnects; the VSC8488 can turn it into XFI or SFI as needed.

21

Hi, thanks to Your reply, i don't have physical access to my server until weekend, but i have a photo of my card

1024×768