Building a 10Gbps home network - advice on hardware

Hardware Questions and Recommendations
1

I recently moved in a new apartment, and I have a new internet provider, and a fiber connection which support of 10Gbps of download speed (GPON fiber).
I have a router/modem included in the deal, but it is not very configurable, and I'd like to replace it with my own device. And fiddle with all that network stuff, let's be honest.

So I have started to do some research, but there is still some things which I am not sure of, and I would appreciate some pointers.

Apartment Layout

                ▲ │       ┌─────────────────────────┐
                │ │       │                         │
                │ │       │                         │
                │ │       │                         │
                │ │       │                         │
                │ │       │       Office            │
                │ │       │                         │
                │ │                                 │
                │ │                                 │
                │ │       ┌─────────────────────────┤
                │ │       └─────────────────────────┤
                │ │                                 │
                │ │    ───┐                         │
X = Fiber       │ │     X │                         │
    arrival     │ │    ┌──┤                         │
                │ │    │  │       Bedroom           │
                │ │    │  │                         │
                │ │    │  │                         │
                │ │    │  │                         │
                │ │    │  └─────────────────────────┘
                │ │    │
                │ │    └────────────────────────────┐
                │ │                                 │
                │ │             Bathroom            │
                │ │    │                            │
                │ │    ├───────────┬────────────────┤
                │ │    │           │                │
                │ │    │           │                │
                │ │    │   WC      │     Kitchen    │
                │ │                │                │
                │ │                │                │
                │ │    ────────────┘                │
                │ │                                 │
                │ │                                 │
                │ │                                 │
        roughly │ │                                 │
        13m     │ │            Living Room          │
                │ │                                 │
                │ │                                 │
                │ │                                 │
                │ │                                 │
                │ │                                 │
                │ │                                 │
                ▼ └─────────────────────────────────┘

There is 5 cat6a cable going from the spot X into each room (1 office, 1 bedroom, 2 Living room, 1 kitchen).

What I'm looking for

Routing/Switching

  • SFP+ port for the wan
  • 5 (or more) 10Gbps ethernet
  • Can handle routing, NAT, firewall, traffic shapping (the latter is probably not a issue with 10G)
  • Allow me to define VLAN

Wifi AP

  • support several SSID and can VLAN tag (can you say that ?) accordingly (so I can for example have 1 VLAN for my personnal wifi and another for a Guest one)

The plan

I think the better solution is to have two devices : the router and the switch.
The router is connected to the net by the fiber, and to the switch, and the switch is connected to the cat6a cable which go in each room.

Router

Switch

I was thinking something like this : https://www.amazon.com/dp/B01GTWPTJY/ref=twister_B0939MK5HG?_encoding=UTF8&psc=1 . If I understand that stuff correctly, I could use a direct attach copper cable to connect the router and the switch.

Wifi AP

I don't really know what to use for that. I would rather build my own if it's possible. I'm going to need to research that part a bit more.

I'm not an expert on that (this woud me my first home networking tinkering) so I've come here to gather advices :

  • Do you know RAM / CPU requirements for the modem for my kind of usage ?
  • Do you think there is obvious flaws in that plan that I missed ?
  • Ideas ?

P.S. : I'm well aware that this not going to be cheap :slight_smile:
P.P.S : I'll probably use Archlinux on the router. I'm used to it, and having the latest kernel is a plus for hardware support, I think.

Thank you for reading my way too long post.

1 Like
2

Are you sure you providers device isn't a simple media converter from fiber to ethernet?

Or is the device perhaps bridgable?
If so, you could avoid the SFP "issue" all together.

3

You're going to need some really serious kit for this. It is going to require vast amounts of performance just to route, nevermind doing anything "extra" like SQM (traffic shaping), adblock, etc.

Your not just going to need separate devices, you're going to need a powerful router, a separate switch and a separate wireless AP. Bank on spending north of $1,000 for all the kit you need.

Nothing other than a powerful x86_64 with ample RAM is going to cut it for your router. I've used one of the variants of the boards linked below for some years now. Hugely powerful with low TDP for the grunt.

To get the 10Gbps ports, you'll need one of the 12 core variants or higher. You're probably looking at around $500 just for the motherboard/CPU. I couldn't recommend these boards more highly - they're bulletproof mini-ITX form factor. Here are two examples - look at their website as there are a bunch of different models with different port combinations.

https://www.supermicro.com/en/products/motherboard/A2SDi-TP8F
https://www.supermicro.com/en/products/motherboard/A2SDi-16C-TP8F

I use an 8-core variant of these boards for a 1Gbps internet link. RAM wise, Openwrt does not use much. My openwrt barely uses around 1GB of RAM on a good day, although I have much more installed since I could only get 8GB DIMMs and I figured on buying two in case one died.

For a 10Gbps link, your core count is more important. These boards have the ability to distribute your network interrupts across all cores, which is going to be vital if you're using a 10Gbps link. If I am using SQM on my 1Gbps link, it uses about 15% - 20% of every core across all 8 cores for a fully shaped gigabit flow. You have a link 10x as fast. You can do the math yourself. You need a 12 or 16 core CPU variant. The bonus is I can give you a fully customized Openwrt build for one of these boards....I've been running Openwrt since 2014 on successive versions of these boards

Then, for switches. The Mikrotik ones offer superb value for money - you will pay 2x or 3x the price for a 10gbps switch from any other vendor.

This is a 4 port SFP+ and a 8-port version

Wifi: you won't get an integrated router that will cope properly with gigabit speeds and give decent wifi performance, let alone one that will do 10Gbps, so don't even think in that direction. I'd suggest you look at proper enterprise class kit, such as Meraki or possibly Ubiquiti.

If you insist you can get a mini pci-e card to put in any router you build. I've used these cards before. They're enterprise class access point cards and the company that sells them has a vast array of choices, up to and including the latest high power wifi 6 cards. They're all Qualcomm Atheros based and just work in Openwrt. You wouldn't want to use a consumer card. These cards are not cheap but they're probably cheaper than going with enterprise class kit, although I personally think the enterprise class kit would be a better solution.

I suggest reading this thread as well: So you have 500Mbps-1Gbps fiber and need a router READ THIS FIRST

7 Likes
4

Think twice before buying a 10G setup. What do you plan to gain from this. My guess is with a high end x86 router and 10G switch and several access points the full cost is $2500 or something. The number of seconds per month when you will be transferring more than 1Gbps outside of speed tests is likely 1 or less.

The real case for 10G is for storage servers in a LAN or for data center interconnects etc

5 Likes
6

x86 would be cheapest option, take a look at dell r220 or supermicro (x10 or newer boards)1u servers plus intel x520 /x540 adapters plus nice switch with as many sfp+ ports as You need. Have You any experience with big router distros like opnsense /pfsense ?

7

I'm happy with my Unifi U6-LR access points, I get 700Mbps on my old 802.11ac Android phone. They don't do WPA3 yet, but they will (it's in beta) and one day they might be supported by OpenWRT natively.

They're based around MT7622 (quad core A53 arm) and MT7915 and there's several other devices using same chips already supported at head.

8

If read the specs properly, with these boards I get CPU, motherboard, and 2 10G ethernet + 2 10G sfp+. The package deal does seems interesting.

Is there

Do you use another fan/heatsink for the cpu, or just what comes with the boards ?

Could you elaborate on that ? What's the special feature of these boards that allow that (which would not work on mini-ITX + a Mellanox PCI for example) ?

Since I already have ethernet cables into the walls, should I not rather take something like this ?

I see that there is adpater SFP+ -> ethernet, but on the mikrotik website (and elsewhere, I expect) they are at 65 $, so it would be pricier to get a switch with sfp+ ports and use adapter right ?

I am going to ;). (think twice, I mean)
One of my use case will be to use HDMI over IP on the LAN to stream from my gaming/work PC (hum, once I'll get my hands on a graphic card, that is) to a TV screen, in differents rooms. If understand that technology correctly, it's uncompressed video and for 4k content, for example, it would push a 1G network a bit over its limits.).
Of course, the effort is not entirely justified. A big part of this is for myself and my interest rather that for any immediate and concrete improvement.

I do not, I've worked with Linux machines only. I do not need much in terms of interface anyway, I'll probably use ip /nftables/tc since I need practices on these one.

I'll take a look !

9

Buying a monster 8-12 core server board for routing just because they're the only ones currently shipping with onboard 10G nics is kind of silly. You won't need that kind of power for routing, not even close. It seems like picking something with an onboard 1G, which you can regard as a management port, and adding PCIe 10G nics will save you money, power and fan noise. Any middling spec Intel desktop or mini-ITX in recent memory can handle the interrupt rate and and routing for 10Gb, and still leave overhead for shaping and so forth. They don't spec them with big cores because of the 10G, they put 10G on them because they're meant for heavy server duty.

I've found that an 11-year old Sandy Bridge i5 with 1333Mhz DDR3 can route just under 40Gbps across virtual machines on different networks, with firewall rules. Allow for the interrupt-driven overhead of physical networking plus any additional processing for shaping and you're still talking about something that could handle 10Gbps, unless you're doing it for an entire enterprise or apartment complex.

10

You can do this with a single unmanaged 10Gbps switch (maybe $400) without having to set up a router that can route 10Gbps to and from the internet. Using an RPi4 you can still route more than a gigabit on the edge for less than $200 compared to $1500 for a 10G router

1 Like
11

That was my initial idea, using a dual SFP+ mellanox NIC which are kinda cheap. My problem is I don't know how to estimate/calculate what's needed (in terms of CPU) for all the routing part.

Would you have a specific model in mind, for a baseline ?

You're right, and as I said, the setup is not entirely justified. I do plan to self host a bunch of stuff, but that connection will probably not be maxed all the time. Which is also why I ask here, to have a better view of what can be done, and at what cost.

12

Yes, all included in the board.

That said, one of the comments above about not needing a board like this if you're just doing routing is correct. You did, however, specify that you wanted to do traffic shaping (ie., SQM), in which case you will need something very powerful to shape a 10gbps flow

Yes, I use a Noctua 80mm PWM on top of the CPU.

I'm not familiar with the other hardware, so I don't know if it can.

Yes, probably. You did specify SFP+ in your original post, hence why I linked that one

100% agreed. You're paying a lot of money for something which you're extremely unlikely to max out, if ever.

13

Not really, a lot depends on the loading you expect and how much real processing you want to do in addition to routing. VPN client/server, packet inspection, these things can be costly for high volumes, but do you really expect those volumes?

That said, a reasonably modern desktop CPU can do a LOT of work without breaking a sweat. My example was low end, but a mini-ATX with 3 slots (making sure they have the lanes for the cards you're going to put in) with a recent generation i3 or i5 at a frequency that doesn't need a serious cooling solution, will do a truly monster job as a router, since it's meant to do a lot more than that.

14

Ok, so the traffic shaping part raise the needed specs and the cost significantly. If I understand traffic shaping correctly, it's purpose is to keep acceptable latency (on interactive traffic) when the link is maxed. What are the consequence when hitting cpu bound ? Reduced throughput or increased latency ? (I don't find the info on the related openwrt page).

Well, I had thought about recycling my current desktop as a router (built mainly for gaming around 6y ago), but I did not know then than I would not be able to upgrade it until... some time in the future :sweat_smile: I'm not sure that the case would fit either. But depending on the evolution of the chips market, that could be a good option, I would just need to add that NIC to it.

15

I have an old 1285Lv4 in my "router".

Currently running 1/1gbit, you can't really see it doing any heavy work while routing at those speeds.

16

It depends, you can have either one or both. For SQM with HTB and fq_codel you will see reduced throughput and only mildly increased latency, with cake you will get less reduction in throughput with a bit more increase in latency....

17

i find it curious that while max was feverishly responding, he hasn't addressed the speculation as to whether his alleged 10Gbps per line is shared among all tenants or dedicated to his unit.

i think if he answers that question, he knows that much of our input is useless. i suspect his building has over 100 tenants with that throughput.

unless, of course, Max has the last name Power.

19

Good point !
As far as I know, the fiber is shared per building. There is two providers in my building and around 40 apartment, so it's shared between 10-30 apartment.
My provider has a one 1G offer two, and I don't know whether it's the same fiber.

I'm going to look at the consequence of variant bandwith on traffic shaping.

20

Well, the whole internet is shared, so the question is not so much "shared medium or not" but rather "shared from where on"... Sharing per se is not a bad thing (after all, your ISP will not have a backbone to transport gigabit plus speeds for all end-users concurrently, and they also do not have to, as not all users will be active concurrently).

Hard to tell, but ISPs try to get away with as little manual work as possible (as technicians are scare and requiring a tech slows down things), so it is not unlikely that the 10 and 1 Gbps plans are simply implemented with different traffic shaper settings at the ISP's end.

If the ISP is competent (and for a long time france's free has been quite competent on dsl, no idea whether that also holds for fiber, but I would assume competence) their traffic shaper and traffic sharing does not need to introduce bufferbloat/jitter, in which case you would not need to worry. If you need to instantiate your own shapers, things get dicier, traffic shaping @10Gbps requires serious hardware (but up to 1Gbps is not that hard, a raspberry pi4B might do), and local shaping does not work well wth variable speed links.

21

Also curious about this... Wondering which hardware is good enough to filter packets or potentially defend against a DDoS on a 10gb connection. Would like to get something as low-power as possible. Doesn't seem like a Raspberry Pi 4 would be enough, even if it had over 3.2gbps available.

23
1 Like