I tried installing the latest development snapshot (openwrt-x86-64-combined-ext4). It seems to runs fine, but it only sees the motherboard NIC, and doesn't recognize the X540-T2 (dual server NIC) addin card.
I wanted to compare the performance of OpenWRT on a PC with 10G NICs to an EdgeRouter Infinity, and to check the wireguard performance over a 10G connection.
Any help would be appreciated.
Try installing kmod-ixgbe
Thank you. That works.
Having it missing from the default binary makes installing a new version of OpenWRT harder (I have to have a console/screen attached.)
Ok, it sort of works.
I think there's a problem with kmod-ixgbe.
With a single X540-T2 (dual 10G nic) card, only one of the interfaces works, as does the motherboard NIC.
So I add a second X540-T2 card (I don't have any single NIC Intel cards), and one of the interfaces from each card works, but the motherboard NIC stops working. One of the non-working 10G interfaces lights up, but OpenWRT doesn't see anything plugged into it, the other non-working 10G interface doesn't even light up and neither does the motherboard NIC.
I assume there's a bug in ixgbe. Is there someone who can look at this? - I'm happy to provide any logs or other diagnostic information.
Is it possible that OpenWRT can't handle more than two interfaces correctly?
Not familiar with these cards at all. Do both interfaces show up under ifconfig?
Yes, I get 5 interfaces under "ip addr", but only two seem to work.
I think I've found my problem. More in a bit.
The X540-T2's work fine.
If you don't set up the interfaces in /etc/config/network, you get strange results, but once the config file was set up correctly, everything worked fine.
With a fairly low end PC (under $200 motherboard+CPU+DRAM) plus a $100 dual Intel 10G NIC, I'm getting 8GB/second of routing throughput. I'm impressed, it's nearly as fast and flexible as an $1500 EdgeRouter Infinity.
Here's what I tried:
pfsense: great UI and performance, but no wireguard which was a deal killer for me.
vyos: no GUI, the CLI interface is not easy to use.
EdgeOS: not open source. Great otherwise.
OpenWRT seems to be the best. I wish the packages weren't so obsessively optimized for low resource routers, but they generally seem to work and be fairly complete. The UI is adequate if not great for a 10G router.
Next I'm going to measure the wireguard performance (from the PC to a EdgeRouter Infinity).
OpenWrt is a distribution for embedded devices. Since drive space is not an issue, you could try something like Arch Linux.
The latest kernels should be faster than 8GB/s.
Adding that package to the default image in the source is pretty easy, see this PR where I had them add broadcom Gigabit ethernet cards https://github.com/openwrt/openwrt/pull/1667
This is the actual commit https://github.com/openwrt/openwrt/pull/1667/commits/bc242f93c845cc7d821010b8e917f81bb3841b7f
The main point is that I don't know how well they will like adding drivers for 10G cards too in the default image.
something like Arch Linux.
I like the router GUI that OpenWRT and other distributions have.
Using Wireguard on a 2 cpu Intel chip connecting to an EdgeRouter Infinity, I'm getting about 1.5 Gbits/second over a 10G link. I'm going to try the same OpenWRT config moved to a high end Intel chip (probably about 4-5 times faster.)
That may well be the source of some of your problems. Intel NICs are widely counterfeited and that price is even “too good to be true” for a dual GigE card.
OEM cards (IBM/Lenovo for instance ) can be that cheap for a genuine Intel Dual Gbit card brand new 
It seems a lot of datacenters are migrating to higher speed cards now (40G among others), you can find truckloads of genuine used 10G cards for 100$ or less on ebay.
And that's for the rj45 ones (normal ethernet connector). If you go with fiber or sfp cards it's as cheap as gigabit ethernet. It's completely insane.
2 Likes
Damn, last time I looked at Chelsio 10G cards they were close to US$200 used!
Thats why im planing on getting some sfp+ cards around summer once i will get a new pc and build a NAS. They cost here 20€/card, another 20€ for a multimode cable and about ~4€ for an sfp+ module.
(Check for connect 2 and connect 3 cards)
So far, I'm quite pleased with the performance of OpenWRT on a middling PC with 10G nics.
The X540-T2 (10GE) and the Intel based SFP+ boards (eg Supermicro AOC-STGN-i2S) work very well.
I'm getting 9+ Gbps routing performance and are really cheap on Ebay (used.)
I need to find a decent 2U case. It's almost as functional as the high-end UBNT EdgeRouter Infinity.
Oddly, none of the "standard" linux or BSD router/firewall distributions met my needs. PFSense doesn't have wireguard, vyos doesn't have a GUI, etc. If there was a decent router GUI for Ubuntu (or other dist) that would be interesting. Having the OpenWRT GUI on ubuntu would be great (I realize that's pretty difficult.)
Pay attention to one thing, sometimes there's a pfifio qdisc is applied to any interface on the router by default, sometimes this pfifo is set with small queue like 100p, this will cause a lot of drops and may degrade performance, you can disable pfifo if it's there.
You could most likely very easily create a package for wireguard based on the FreeBSD package, GUI is another story however. I can see why pfsense is a bit hesitant to include by default however