10G NAT/Firewall performance problems

I have a 10Gbps fiber connection. Currently, I use a UBNT EdgeRouter Infinity router/firewall, which works quite well. I get 4Gbps down, 5.5 Gbps up with speedtest, which seems about right.
I'm testing other firewalls, specifically OpenWRT, OPNSense, and PFSense on some medium to good performance PCs (i3-8100 and i7-7xxx) with 32GB of memory.
For straight routing (no firewall no NAT), all of the firewalls give me well over 6 Gbps - probably limited by the test rather than the router - on all of the PCs I've tested. This is fine.
When I use Speedtest (i.e., using NAT and a firewall), I get very odd results:
OpenWRT 18.06.2 ~100 Mbps down, > 4 Gbps up
OPNSense 19.1 (and 18.7) - ~100 Mbps down, > 4 Gbps up
PfSense CD 2.4.4r1 - 4 Gbps down, > 4 Gbps up
These are fresh installs, with no tweaks or tuning. The numbers are consistent across different PCs. I'm using an Intel X540-T2 for all tests and I've tested 3 PCs with every firewall, so it's not hardware dependent.
Clearly, there is a problem with my use of OPNSense and OpenWRT, but I don't what to try to fix this.
Why does PFSense work out-of-the-box, and nothing else?
Any suggestions?

From OPNSense forum, it was suggested I set mss to 1300 on the LAN.
I set mss_clamping in the LAN firewall zone, and I'm 20x faster, i.e., >2Gbps Speedtest download.
So, OpenWRT, OPNSense and PFSense on a decent PC all seem to be capable for routing at interesting gigabits per second.
Now to lookup mss so I understand what difference this made.

this is a common problem and means that your isp connection does not support the full/default packet size (1500).
by forcing tcp connections going over your router to reduce their packet size (mss_clamping) you avoid running into packet fragmentation and thus dont cripple performance.
this is probably because the mechanisms designed to "figure it out" (path mtu discovery) failed.

increase this mss value to find the greatest value before peformance degrades again.

1 Like

routing at these speeds you likely run into cpu exhaustion.
for further optimizing it might be sensible to cut down on firewall rules, running services and maybe powersaving stuff.
there is also flow_offloading in the firewall that might help.

Out of curiosity is this a home or business connection, and where and from what ISP?

Atherton, CA home connection.
2500 homes. I own the fiber plant, there are 6 ISPs available, we split the monthly fees roughly 50/50.
1 Gbps is $60-100/month using a GPON.
10 Gbps is $200/mo, but you have to pay $12,000 upfront for a "dedicated" fiber, which means an individual fiber "owned" by the homeowner from the home to the CO. There's one home that is getting 100Gbps connection (I think for bragging rights.) We have 1G and 10G equipment in the CO, but if you buy a dedicated fiber and are will to go to the effort of setting it up, we'll let you have any bandwidth you want. I'm hoping for 1 terabit in 10-15 years, although I have no idea what to do with it.
We'll might expand into neighboring cities (Menlo Park, Palo Alto, etc.) in the future.

I'm not 100% sure but I'm fairly sure that Chelsio nics are to be preferred at least on FreeBSD when reaching such speeds. I think you'll find these sites much valuable.

https://github.com/ocochard/netbenches (a bit old but you get an idea, scroll down)

Also, pfsense is heavily tuned (and tested) for network performance. OpenWrt doesn't aim for best performance possible, instead going for size.

1 Like