Are Intel X540 NICs supported in OpenWrt?


#1

I tried installing the latest development snapshot (openwrt-x86-64-combined-ext4). It seems to runs fine, but it only sees the motherboard NIC, and doesn't recognize the X540-T2 (dual server NIC) addin card.
I wanted to compare the performance of OpenWRT on a PC with 10G NICs to an EdgeRouter Infinity, and to check the wireguard performance over a 10G connection.
Any help would be appreciated.


#2

Try installing kmod-ixgbe


#3

Thank you. That works.
Having it missing from the default binary makes installing a new version of OpenWRT harder (I have to have a console/screen attached.)


#4

Ok, it sort of works.
I think there's a problem with kmod-ixgbe.
With a single X540-T2 (dual 10G nic) card, only one of the interfaces works, as does the motherboard NIC.
So I add a second X540-T2 card (I don't have any single NIC Intel cards), and one of the interfaces from each card works, but the motherboard NIC stops working. One of the non-working 10G interfaces lights up, but OpenWRT doesn't see anything plugged into it, the other non-working 10G interface doesn't even light up and neither does the motherboard NIC.
I assume there's a bug in ixgbe. Is there someone who can look at this? - I'm happy to provide any logs or other diagnostic information.
Is it possible that OpenWRT can't handle more than two interfaces correctly?


#5

Not familiar with these cards at all. Do both interfaces show up under ifconfig?


#6

Yes, I get 5 interfaces under "ip addr", but only two seem to work.


#7

I think I've found my problem. More in a bit.


#8

The X540-T2's work fine.
If you don't set up the interfaces in /etc/config/network, you get strange results, but once the config file was set up correctly, everything worked fine.

With a fairly low end PC (under $200 motherboard+CPU+DRAM) plus a $100 dual Intel 10G NIC, I'm getting 8GB/second of routing throughput. I'm impressed, it's nearly as fast and flexible as an $1500 EdgeRouter Infinity.

Here's what I tried:
pfsense: great UI and performance, but no wireguard which was a deal killer for me.
vyos: no GUI, the CLI interface is not easy to use.
EdgeOS: not open source. Great otherwise.
OpenWRT seems to be the best. I wish the packages weren't so obsessively optimized for low resource routers, but they generally seem to work and be fairly complete. The UI is adequate if not great for a 10G router.
Next I'm going to measure the wireguard performance (from the PC to a EdgeRouter Infinity).


#9

OpenWrt is a distribution for embedded devices. Since drive space is not an issue, you could try something like Arch Linux.

The latest kernels should be faster than 8GB/s.


#10

Adding that package to the default image in the source is pretty easy, see this PR where I had them add broadcom Gigabit ethernet cards https://github.com/openwrt/openwrt/pull/1667
This is the actual commit https://github.com/openwrt/openwrt/pull/1667/commits/bc242f93c845cc7d821010b8e917f81bb3841b7f

The main point is that I don't know how well they will like adding drivers for 10G cards too in the default image.


#11

something like Arch Linux.

I like the router GUI that OpenWRT and other distributions have.


#12

Using Wireguard on a 2 cpu Intel chip connecting to an EdgeRouter Infinity, I'm getting about 1.5 Gbits/second over a 10G link. I'm going to try the same OpenWRT config moved to a high end Intel chip (probably about 4-5 times faster.)


#13

That may well be the source of some of your problems. Intel NICs are widely counterfeited and that price is even “too good to be true” for a dual GigE card.


#14

OEM cards (IBM/Lenovo for instance ) can be that cheap for a genuine Intel Dual Gbit card brand new :slight_smile:


#15

It seems a lot of datacenters are migrating to higher speed cards now (40G among others), you can find truckloads of genuine used 10G cards for 100$ or less on ebay.

And that's for the rj45 ones (normal ethernet connector). If you go with fiber or sfp cards it's as cheap as gigabit ethernet. It's completely insane.


#16

Damn, last time I looked at Chelsio 10G cards they were close to US$200 used!


#17

Thats why im planing on getting some sfp+ cards around summer once i will get a new pc and build a NAS. They cost here 20€/card, another 20€ for a multimode cable and about ~4€ for an sfp+ module.

(Check for connect 2 and connect 3 cards)


#18

So far, I'm quite pleased with the performance of OpenWRT on a middling PC with 10G nics.
The X540-T2 (10GE) and the Intel based SFP+ boards (eg Supermicro AOC-STGN-i2S) work very well.
I'm getting 9+ Gbps routing performance and are really cheap on Ebay (used.)
I need to find a decent 2U case. It's almost as functional as the high-end UBNT EdgeRouter Infinity.
Oddly, none of the "standard" linux or BSD router/firewall distributions met my needs. PFSense doesn't have wireguard, vyos doesn't have a GUI, etc. If there was a decent router GUI for Ubuntu (or other dist) that would be interesting. Having the OpenWRT GUI on ubuntu would be great (I realize that's pretty difficult.)