Looking for Gbit / multi-Gbit router available in the EU

harrydg · July 24, 2022, 7:50pm

Obviously i can elaborate. This is why I/we are here

Basically, my setup looks as follows:

What changed: i added the 100% similar setup for my opnsense install as i explained in my openwrt install. Shut down the openwrt, so my full network runs on opnsense. Literally EVERYTHING works, traffic shaping to eliminatie bufferbloat is implemented with queues. My wireshark etc also works fine. I do video calls, my wife too... So low latency is never an issue. Now... For some strange reason, when, in the evening, i play PUBG or CSGO, i get sometimes "lag detected" issues. Nothing shows on the VM, no errors in proxmox/linux, no mentions on opnsense console... I have changed virtual drivers/offload/... I stopped all IDS and IPS functionality, added (virtual) cpu, mem, ... Nothing helps!
So for the moment, i am (re)running my openwrt 19.blabla version for my gaming rig alone. How? Disabled all DHCP services and manually route my gaming rig through the openwrt VM. The house runs on opnsense/uses dhcp, even the static ips are given by dhcp.
My openwrt never gave any issue... Ever! So:
Once the new openwrt implementation with nftables goes.stable, i will 100% switch to openwrt again.

If anyone can help debug the opnsense issue, i would be happy to help, but this is not the right forum for that

If you want to dig deeper/more help, i am on holidays right now, so i type everything on the phone.
Meaning, when i get home, i can do proper testing/explaining.

moeller0 · July 24, 2022, 8:28pm

Well, the BSDs have a different approach to traffic shaping than Linux. Sure there is an fq_codel version for *BSD, I believe it is competently implemented (I hold BSD developers in high esteem), but it might not be identical to Linux's implementation. Also the *Senses tend to focus on other things beyond and above mere traffic shaping and AQM, so I believe what sqm-scripts does or what sch_cake does almost single-handedly for Linux is simply not a top priority for PF-/OPN-sense.
So what I want to say, your issue might not be amendable to debugging, because it might not be a bug, but simply the consequence of having a different focus.

Caveat: not everybody needs/wants to accept the trade-offs involved in maintaining lower latency (mainly lower throughput), so OpenWrt/sqm/cake are neither required nor without alternatives, but IMHO a pretty decent package pretty much leading the pack (it is debatable by how much).

harrydg · July 24, 2022, 9:50pm

Fully agree. And completely aware. Thats why i played with everything to try to figure out where things go "wrong". I failed up to now. Believe me: Nothing frustrates me more than failing on network issue debugging. All help/guidance is appreciated, but pollutes the current topic

BigG · July 24, 2022, 10:10pm

That OpenWRT does only aim at low end devices is only partly true. Some defaults are not very optimal for big x86 setups but I can run OpenWRT as a router VM with NICs passed through and it does happily route 25 Gbps between vlans. So if your hardware is fast enough nothing stops you to use OpenWRT in busy setups with fast uplinks, you may just have to tune some things to fit your needs but that you will have to do on *sense as well.

diizzy · July 24, 2022, 10:14pm

No one said only but by "tune some things" does involve a lot of you want to get decent (expected) performance out of non "low cost" hardware as moeller0 phrased it.

BigG · July 24, 2022, 10:21pm

Well then let's make it easy.
Here my settings for 25 Gbps throughput. With a Intel 9900k and 16 GB RAM

#optimizations
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.ipv4.tcp_rmem = 4096 87380 33554432 
net.ipv4.tcp_wmem = 4096 65536 33554432  
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_fin_timeout = 4 
vm.min_free_kbytes = 65536
net.ipv4.netfilter.ip_conntrack_max = 196608
net.netfilter.nf_conntrack_tcp_timeout_established = 7200
net.netfilter.nf_conntrack_checksum = 0
net.netfilter.nf_conntrack_max = 196608 
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 15
net.nf_conntrack_max = 196608
net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.ip_local_port_range = 1025 65530
net.core.somaxconn = 20480
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_timestamps = 0

moeller0 · July 24, 2022, 10:33pm

You might notice these are basic Linux capabilities, OpenWrt however is a distribution not just the kernel. And IMHO that distribution is biased towards storage starved devices you find marketed as customer premise equipment much more than for network big iron...

So I think we are not really disagreeing we just have not found a common basis yet.

ed8 · July 25, 2022, 6:11am

possibly interesting router-only options:

1Gbit: Raspberry Pi CM4 + DFRobot Routerboard, see also link - however Raspberry Pi hardware is almost everywhere out of stock (https://rpilocator.com/)
2.5Gbit: NanoPi R5S, under active development, see here

harrydg · July 25, 2022, 9:30am

Would it be possible/a good idea to create a vmdk and/or full disk installer image of openwrt?

Why virtual disk image? For easy high availability/failover to other devices.

frollic · July 26, 2022, 7:48pm

Currently hard to find used, but check out the NetGate 6100.

2x SFP+
4x 2.5 gbe
Atom C3558

shdf · July 27, 2022, 5:56am

i don't know if you're using an external ONT to connect to your router, but they usualy have a 1Gb output on LAN side, and this is a problem when you are on a 2Gb offer.
i just found an ONT with 2.5Gb output here, the blue RJ45 is the 2.5Gb port, the yellow one is 1Gb :

it can be very useful with routers that have 2.5gbps ports

moeller0 · July 27, 2022, 6:50am

Some ISPs go that route especially those that offer > 1 Gbps plans over GPON. E.g. Deutsche Telekom's (DT) recent ONT (Glasfasermodem2, the only ONT they currently market to residential customers) offers a 2.5 Gbps ethernet port, supposedly as preparation for 2Gbps over GPON. Given GPON's hard limt at ~2.4/1.2 Gbps it is questionable how many ISPs will actually provision more than 1 Gbps, e.g. DT currently only offers 1 Gbps, but provisions a gross rate above 1Gbps, so users can actually achieve >= 1 Gbps in speedtests. Whether these 6~10% higher throughput justifiy the cost to deploy a >1 Gbps network in one's home, is a question for each network to decide individually.

However, GPON isnot the technology ISPs seem to prefer for >1 Gbps plans, they rather seem to flock to XGSPON, which offers a nominal 10/10 Gbps per segment (if FEC is used this reduces to ~8.6/8.6), so I am not sure whether a GPON ONT with 2.5 Gbps ethernet port will see much use above 1 Gbps....

All of that said, there are markets like switzerland, where ISPs are willing/permitted to market the full but shared segment capacity to each customer (e.g. 10 Gbps plans on an XGSPON segment with up to 32 users), if one's ISP does that for GPON, then 2.5 Gbps ethernet might be the optimal choice to occasionally get the maximum rates.

frollic · July 27, 2022, 8:13am

Yeah, my local ISP offers the Zyxel AX7501 to their 10/10gbit customers.

moeller0 · July 27, 2022, 9:49am

Which in all likelihood only achieves acceptable performance by using accelerators instead of in-kernel networking. I am not saying that because that is inherently "bad", but only to indicate that I expect if the AX7501 should ever get an OpenWrt firmware, the performance (due to lack of Linux-supported accelerators) to stay well below 10/10 Gbps...

frollic · July 27, 2022, 10:14am

Yeah, I wasn't going to use it, unless they forced me to (not jumping on the 10/10 train any time soon), got a dual 10/10gbe NIC in my server/router, might needed an ONT though.

frigg · July 31, 2022, 2:52pm

I've been running OpenWRT on a rpi4 (2GB RAM although 1GB is more than enough) since 21.02rc1 and it is working fantastic. I use the TP-Link UE300 USB Ethernet adapter as secondary Ethernet interface and my old Archer C7 as wireless AP.
I made some WAN/LAN throughput measurements (using this: https://github.com/mmeisner/sshopenwrt) and it maxes out the 1Gbit/s with CPU usage below 10%. With Wireguard it loses, I think it was less than 5% throughput, which is probably protocol overhead. I don't remember the CPU usage but nothing alarming.
I never measured with OpenVPN as I think it is pretty much obsoleted by Wireguard - unless you have some legacy installations to cater for.
BTW, I used this repo: https://github.com/mmeisner/openwrt-image-buildomatic to build an image for the rpi. It creates an image with a lot of useful pre-installed packages.

IvoK · September 8, 2022, 12:46pm

Fully agree with you. Even for a lesser price, the WRT3200ACM would still be a poor choice for the lack of modern technology (e.g. WiFi 6). I'm just wondering about the two models you mention - neither appears on https://firmware-selector.openwrt.org/. How do yoy instal OpenWRT on those?

shdf · September 8, 2022, 12:54pm

it was just to show that at a much lower price we could have better performing equipment.
The WAX206 and the QHora-301W are still in Work In Progress.

IvoK · September 8, 2022, 12:59pm

I see. What would you recommend as the best options of equipment that runs OpenWRT?

shdf · September 8, 2022, 1:07pm

there is not only 1 answer to this question, it will depend on several parameters. The budget, the speed of your internet connection, what you want to do with your network, if you want an all-in-one router/wifi, or if you just want a router and manage the wifi separately.
We need more details so we can answer.