10G Router/High End?

So here is directly out of the department "bragging-rights":

https://www.init7.net/de/internet/fiber7/
Fiber7-X2

Glasfaser-Internet
Max: 25/25 Gbit/sec (download/upload)
Fix: CHF 64.75/Mt mit Jahresrechnung (CHF 777.-/Jahr)
Setupgebühr: einmalig CHF 333.-
Kostenlos dazu: TV7
(höhere Setupgebühr aufgrund teurerer Optik)

That is 25/25 Gbps active Ethernet over fiber for 777/12 = 64.75 CHF/month*, plus a one time 333CHF fee (and the cost for the 25G capable router)...

Okay, the uplink of the switches to the backbone is limited to 100 Gbps and the switches have up to 48 ports, but in the "rate/monetary unit" department that offer, while certainly a gimmick, seems the one to beat in Europe....

*) That is actually a fixed prices for 25, 10 or 1 Gbps....

1 Like

Used to have, but sold the 10Gbase-T switch I had, since it wasn't capable of running multi-gig, only 1 & 10.

Router's should still be capable of delivering 10gbit.

i'l be great if someone can make working openwrt for those cards https://www.ebay.pl/itm/184570473764?hash=item2af941ed24:g:lg4AAOSwRCZfzm0c , performance is awesome and they can be found dirt cheap (cn6xxx)

for X86 i am using some OpenWRT supported Mellanox Connect X3 cards

I also have a Dual Port QSFP+ (For 10GbE needs QSFP to SFP adapter)

Your can get them in various form factors

https://www.ebay.ie/itm/162990180516?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m1438.l2649

https://www.ebay.com/itm/233621305252?hash=item3664ea37a4:g:LzsAAOSwb65e8sb3

If you're after 10Base-T, dual port Dell W1GCR Broadcom 57810S' are often as low as $20 or so on ebay.

SFP+ model of the same card is called Y40PH.

Seems like the driver is upstream in linux kernel so it's just a matter of creating a package for it. So it should be relatively easy.
see

Do you have one of those cards to test?
Looks like a bit of a power pig though, what's the power consumption on that puppy?

Why upgrading all switches though. I think you can get away with a small 10Gbit switch for a few "VIP" devices and that's it. I assume you don't have 48 devices all need 10Gbit connection to the internet or to each other. Most will be basic users and security cams and wifi access points and similar.

In my home I don't even have 100 Mbit internet but I still have a "VIP network section" on a small 10Gbit switch for my home server cluster, the NAS and a desktop PC. Just for internal file transfer speeds.

You're right I could have a tiered network. I probably have 3 or 4 devices that could even utilize 10Gbps, the problem is that they're not all physically next to each other, they're spread around hanging off various switches. I could solve this by running more cat6 I guess. It's basically just not worth it. Although the NAS could probably push 2Gbps it couldn't push 10. So having the NAS hanging off a 10 switch would be pointless compared to just bonding two gig nics. Basically 10Gbps is a huge waste of money for me. I'd be better off upgrading my NAS and bonding a couple ports. 10Gbps to the internet would be truly a waste. Even when I'm running speed tests the speed test sites can't necessarily keep up with 1Gbps. So I'd just pass if it were offered here.

Ok I didn't consider that most of my 10Gb stuff on the VIP network is in the same room (with the desktop being in an adjacent room) so the longest cable length is like 3 meters.
So my home network is "easy mode".

Apart from the distance/ number of hops issues with distributing a VIP network, there is another problem. One of the expensive 10 GBit/s ports is by definition lost to the uplink towards the remainder of your 1 GBit/s network segments (two, if you think about LAG), typically leaving you with 3, maybe 4 ports, which would be covered pretty much from the get-go, without any free ports for future extension (get a new 10 GBit/s system, buy a new switch as well…). But for 'just' 3 VIP systems in close proximity, you don't really need an expensive switch to begin with (instead of peer-to-peer connections). At least to me, a 4-5 port switch which sells solidly in the three-figure-range simply doesn't make any sense (maybe 8 ports, for a tiered VIP segment, but that's the bare minimum to be viable - and I'm not even mumbling about 'managed', yet).

Looking into it closer, 2.5GBASE-T (and 5GBASE-T is even less common) isn't much cheaper than going all the way to 10GBASE-T (and probably SFP+ && Twinax DAC), but merely provides a relatively small speedup - for considerable money (a couple of months ago, I would have said ~800-1000 EUR/ USD for just 2-3 systems in cards, cables and switch - nowadays you might get away with 500-600 EUR/ USD, still a lot of money to speed up just 2-3 systems).

Sure, I want to speed up my LAN transfers (at least for the last 10 years already), I'd even take 2.5GBASE-T if it were available for a marginal markup (perhaps up to ~25 % more than 1GBASE-T), but not for 10-15 times of what 1GBASE-T would cost me - and I certainly won't buy a 4-5 port switch for big money and which is already full on day 1.

No, not necessarily. Since you are using a x86 router anyway you might as well add a card with gigabit ports in the router and bridge all these ports with the 10Gbit port. So that you are not wasting 10Gbit ports.
That device is powerful enough to NAT on 10Gbit so it is good enough to bridge a few ports at gigabit.

Or you could use Mikrotik switches (see below) that have a single 1Gb port you can use to connect them to the low speed network.

If it is 2 systems, yes, if it is 3 or more, I'd rather avoid it. Daisy chaining means that if one node is down the network is interrupted, while with a switch all nodes are independent.
There are "pass through" gigabit cards that will bridge the connection if the host is down, but I've not seen 10Gbit cards that do that at decent prices.

did you hear about our lord and saviour Mikrotik?
They make CRS305-1G-4S+IN which is a 4 port 10 Gbit SFP+ and one gigabit port for 120-140 euro and CRS309-1G-8S+IN which is a 8-port SFP+ and a gigabit port for around 250 euro. Both fanless, both can be powered by PoE, both managed switches. They can also boot RouterOS aka the firmware Mikrotik ships with their routers, but the CPU can't really NAT anywhere near 10Gbit so it's more to have a "smarter switch" with router features like DHCP, MAC filters, and running scripts and whatnot. (EDIT: it seems with RouterOS version 7.1 beta 5 it these things finally got hardware acceleration for routing and can now actually route at full speed between VLANS, didn't try this but sounds very nice)

Yes it is three figures but low end of three figures, for managed switches.

I've been using the 4-port one mostly because I need fast speed for a Proxmox cluster (inter-host communication to do live migration of VMs and Ceph cluster filesystem) and I tapped into that with my desktop PC as well.
(The cluster is 3 devices, my PC is the fourth, and the NAS is a VM inside the cluster so it does not use additional ports)

It depends on the assumptions and caveats.
I'm more in the 300-350 range to cover 4 systems actually, and using the 8 port switch instead it would scale up linearly to 600-ish for 8 systems, which isn't bad imho.
Buying used SFP+ cards, having most stuff in the same room, using fiber instead of Gbase-T for longer distances are what keeps most of the costs down if you want to make a VIP 10Gbit network.

As I said above most of that stuff in the same place so everything is connected with sfp+ copper cables and the cards are all used server surplus with a sfp+ port like the Mellanox mentioned above or Supermicro cards with double Sfp+ ports. So it's like 25 euro per 2m cable (i have 3), and 40 euro per card (I have 4), funnily enough the dual port cards cost the same as the single port cards, must be the lower demand for dual port cards.

The desktop PC in the adjacent room (I drilled a small hole in the wall to pull the cable through) is connected with fiber. 10Gbit optical transceivers are stupid cheap (like 20 euro per transceiver) because they are server surplus.

In my case the 10Gbit VIP network is fully independent and separated from the rest of the network for security/privacy reasons (besides all devices involved have gigabit ethernet ports anyway so I might as well use them), but I could hook it up if I wanted.

3 Likes

Prices are relative to each country. In Italy we have 2500/300 FTTH connection for 25 euros/month (if they have ran a proper fiber connection down your street).

that's a big IF. In many places the street boxes are overloaded so you don't get that

Also interested... What is the lowest power hardware you can get to run openwrt that can packet filter a 10gbps connection?

Probably something like this: https://www.newegg.com/supermicro-mbd-x10sdv-4c-tln2f-o-intel-xeon-processor-d-1521/p/N82E16813182973

I wonder how much less power the Raspberry Pi 4 uses - probably a lot less, eh? Considering the Pi 4 can pull off 3.2gbps, I'd like to try for something more power efficient than a Xeon. I don't care about wasted energy - just keeping my router online for hours with my UPS. An old school router could stay on for days in the event of a power failure!

Let's do some back of the envelope here... 10Gbps/(8*1500bits/pkt) = 833333 pkts/s

1e9 cycles/s /833333 PKs/s = 1200cycles/pkt

So there's on the order of a few thousand cycles to copy the packet to the kernel, decide on it's fate and send it to the NIC and that's assuming max sized packets. To make this work you will want quad core and 2GHz minimum, and something modern, probably you can do it with an i5 or so but not much less.

Thanks! Know if there are any ARM CPUs supported by openwrt that are faster than an i5?

There aren't that I know of.