with 2 x 10Gbit ports there is also the Asus GT-AXE16000 but it will never be supported by openwrt.
That's a strange conclusion to draw . The better conclusion would be you need beefier hardware to compensate for vendor specific proprietary hardware acceleration etc.
The correct conclusion would be Linux, not
Openwrt, since the 2nd only supports what the 1st provides.
I do think that OpenWrt with its relative light weight (but also somwhat lacking in features) opkg and focuss on updates by flashing a new firmware is taylored at devices with limited storage/price. That is not the class of whitebox servers used for more potent routers,and more the class of cheap 'plastic' all-in-one consumer routers, no?
We are at an interesting time with access links in the 1 to ~10 Gbpas are becoming available, so the definition of what a cheap all-in-one needs to deliver is shifting somewhat... though as @Borromini wrote most of the heavy lifting is going to end up in acellerators.
Interesting, could you elaborate on that a little?
I also run a bunch of other containers, contexts and VMs
I have a separate gaming-PC-turned-NAS that I run my other services on, I like to keep critical services (like a router) on separate silicon.
Obviously i can elaborate. This is why I/we are here
Basically, my setup looks as follows:
What changed: i added the 100% similar setup for my opnsense install as i explained in my openwrt install. Shut down the openwrt, so my full network runs on opnsense. Literally EVERYTHING works, traffic shaping to eliminatie bufferbloat is implemented with queues. My wireshark etc also works fine. I do video calls, my wife too... So low latency is never an issue. Now... For some strange reason, when, in the evening, i play PUBG or CSGO, i get sometimes "lag detected" issues. Nothing shows on the VM, no errors in proxmox/linux, no mentions on opnsense console... I have changed virtual drivers/offload/... I stopped all IDS and IPS functionality, added (virtual) cpu, mem, ... Nothing helps!
So for the moment, i am (re)running my openwrt 19.blabla version for my gaming rig alone. How? Disabled all DHCP services and manually route my gaming rig through the openwrt VM. The house runs on opnsense/uses dhcp, even the static ips are given by dhcp.
My openwrt never gave any issue... Ever! So:
Once the new openwrt implementation with nftables goes.stable, i will 100% switch to openwrt again.
If anyone can help debug the opnsense issue, i would be happy to help, but this is not the right forum for that
If you want to dig deeper/more help, i am on holidays right now, so i type everything on the phone.
Meaning, when i get home, i can do proper testing/explaining.
Well, the BSDs have a different approach to traffic shaping than Linux. Sure there is an fq_codel version for *BSD, I believe it is competently implemented (I hold BSD developers in high esteem), but it might not be identical to Linux's implementation. Also the *Senses tend to focus on other things beyond and above mere traffic shaping and AQM, so I believe what sqm-scripts does or what sch_cake does almost single-handedly for Linux is simply not a top priority for PF-/OPN-sense.
So what I want to say, your issue might not be amendable to debugging, because it might not be a bug, but simply the consequence of having a different focus.
Caveat: not everybody needs/wants to accept the trade-offs involved in maintaining lower latency (mainly lower throughput), so OpenWrt/sqm/cake are neither required nor without alternatives, but IMHO a pretty decent package pretty much leading the pack (it is debatable by how much).
Fully agree. And completely aware. Thats why i played with everything to try to figure out where things go "wrong". I failed up to now. Believe me: Nothing frustrates me more than failing on network issue debugging. All help/guidance is appreciated, but pollutes the current topic
That OpenWRT does only aim at low end devices is only partly true. Some defaults are not very optimal for big x86 setups but I can run OpenWRT as a router VM with NICs passed through and it does happily route 25 Gbps between vlans. So if your hardware is fast enough nothing stops you to use OpenWRT in busy setups with fast uplinks, you may just have to tune some things to fit your needs but that you will have to do on *sense as well.
No one said only but by "tune some things" does involve a lot of you want to get decent (expected) performance out of non "low cost" hardware as moeller0 phrased it.
Well then let's make it easy.
Here my settings for 25 Gbps throughput. With a Intel 9900k and 16 GB RAM
#optimizations net.core.rmem_max = 33554432 net.core.wmem_max = 33554432 net.ipv4.tcp_rmem = 4096 87380 33554432 net.ipv4.tcp_wmem = 4096 65536 33554432 net.core.netdev_max_backlog = 262144 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_max_orphans = 262144 net.ipv4.tcp_max_syn_backlog = 262144 net.ipv4.tcp_fin_timeout = 4 vm.min_free_kbytes = 65536 net.ipv4.netfilter.ip_conntrack_max = 196608 net.netfilter.nf_conntrack_tcp_timeout_established = 7200 net.netfilter.nf_conntrack_checksum = 0 net.netfilter.nf_conntrack_max = 196608 net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 15 net.nf_conntrack_max = 196608 net.ipv4.tcp_keepalive_time = 60 net.ipv4.tcp_keepalive_intvl = 10 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.ip_local_port_range = 1025 65530 net.core.somaxconn = 20480 net.ipv4.tcp_max_tw_buckets = 2000000 net.ipv4.tcp_timestamps = 0
You might notice these are basic Linux capabilities, OpenWrt however is a distribution not just the kernel. And IMHO that distribution is biased towards storage starved devices you find marketed as customer premise equipment much more than for network big iron...
So I think we are not really disagreeing we just have not found a common basis yet.
possibly interesting router-only options:
Would it be possible/a good idea to create a vmdk and/or full disk installer image of openwrt?
Why virtual disk image? For easy high availability/failover to other devices.
Currently hard to find used, but check out the NetGate 6100.
4x 2.5 gbe
i don't know if you're using an external ONT to connect to your router, but they usualy have a 1Gb output on LAN side, and this is a problem when you are on a 2Gb offer.
i just found an ONT with 2.5Gb output here, the blue RJ45 is the 2.5Gb port, the yellow one is 1Gb :
it can be very useful with routers that have 2.5gbps ports
Some ISPs go that route especially those that offer > 1 Gbps plans over GPON. E.g. Deutsche Telekom's (DT) recent ONT (Glasfasermodem2, the only ONT they currently market to residential customers) offers a 2.5 Gbps ethernet port, supposedly as preparation for 2Gbps over GPON. Given GPON's hard limt at ~2.4/1.2 Gbps it is questionable how many ISPs will actually provision more than 1 Gbps, e.g. DT currently only offers 1 Gbps, but provisions a gross rate above 1Gbps, so users can actually achieve >= 1 Gbps in speedtests. Whether these 6~10% higher throughput justifiy the cost to deploy a >1 Gbps network in one's home, is a question for each network to decide individually.
However, GPON isnot the technology ISPs seem to prefer for >1 Gbps plans, they rather seem to flock to XGSPON, which offers a nominal 10/10 Gbps per segment (if FEC is used this reduces to ~8.6/8.6), so I am not sure whether a GPON ONT with 2.5 Gbps ethernet port will see much use above 1 Gbps....
All of that said, there are markets like switzerland, where ISPs are willing/permitted to market the full but shared segment capacity to each customer (e.g. 10 Gbps plans on an XGSPON segment with up to 32 users), if one's ISP does that for GPON, then 2.5 Gbps ethernet might be the optimal choice to occasionally get the maximum rates.
Yeah, my local ISP offers the Zyxel AX7501 to their 10/10gbit customers.
Which in all likelihood only achieves acceptable performance by using accelerators instead of in-kernel networking. I am not saying that because that is inherently "bad", but only to indicate that I expect if the AX7501 should ever get an OpenWrt firmware, the performance (due to lack of Linux-supported accelerators) to stay well below 10/10 Gbps...
Yeah, I wasn't going to use it, unless they forced me to (not jumping on the 10/10 train any time soon), got a dual 10/10gbe NIC in my server/router, might needed an ONT though.