Hey there.

Here's the reason why. Don't read if you don't care.

I just payed a little with a Banana Pi.

My test scenario currently is
* a single NIC (that's what the Pi has)
* providing two VLANs tagged
* going into an ESX
* where one VM (sender) connects to one VLAN and another VM (receiver) to the second VLAN.
* The routing OpenWRT involved doesn't NAT.

I get something between 400 and 600MBit from one sender to receiver going through the PI.

When using my TP-Link 4900 with OpenWRT, I get something bewteen 350 and 400MBit.

Replacing the Banana Pi with a virtual OpenWRT on Virtualbox on my desktop computer (but still wired through a single tagged vlan cable), I get 950MBit from one VM to the other.

So I'm pretty sure I get nearly 1GBit full duplex through a single NIC into and out of the ESX at the same time.

I don't *need* a new router, I'm just curious and want to play a little. The BananaPi does provide enough performance at the moment. But since things can always get better if you throw more money at them ...

Here's the topic

I'm looking for a "routing device" (doesn't necessarily be titled as a router by its vendor) that can route something between 800 and 1000MBit from one vlan to another, ideally through a single NIC full duplex.

I don't want an OEM router with onboard memory but something that directly boots from either USB or SD-Card where I can just replace the router but keep the OS in place, or just "DD" the entire storage as backup.

I'm currently thinking about either the PC-Engines apu3a4 or the Turris Omnia.

I don't need Wifi (since I have enough APs spread around the house for that purpose), so I'm comparing the No-Wifi versions of both.

The Turris Omnia has 5 LAN and an SFP. The LAN is clearly a plus I could use, the SFP is currently of no use for me at all.

The Turris has only 1GB of memory, the APU has 4.

What do you think about the CPU involved? Can they handle 1GBit routing?
What about the NICs of both devices? Can some offloading be enabled to reduce CPU load?

Any thoughts are welcome, even if you suggest a completely different device.

I'm asking here because I want OpenWRT on it, no matter which device it will be.

Regards,
Stephan.

These should start shipping soon.

I think J1800 based motherboard will get you there.

J1900 is overkill for openwrt, specially that openwrt doesnt use pf (like what freebsd based dsitro like pfsense, monowall etc...) instead uses iptables/netfilter which is not a multi-core friendly.

But there's a Qotom J1900 mini pc there, that has 4 Intel NIC on it, its a little bit expensive since (compare to APU based system) you still need to buy a separate RAM for it and SSD (m-sata).

(Last edited by remlei on 11 Apr 2017, 23:17)

Mmh, if you route as VLAN over the same NIC, would your performance ceiling not be half the interface rate if yu test with full duplex saturating load, as each packet will need to go in and out through the same NIC?

Thank you for the answers.

Damn, some more device that I'd really like to get my hands on and test .

The UP could really be a thing. I think I'll wait for it to be not on pre-order bot regularly available and then give it a shot.

J1800 and J1900 was something I didn't even know to search for. But of course, really makes sense to search for the CPU generation.
And they aren't necessarily more expensive then the APU boards.

• A PC Engines bundle with board, case, msata and power cord is available for ~190€ here.

• The Qotom Q190G4N has board, case, msata and RAM just like the PC Engines bundle (it even has 64GB msata compared to the 16GB my APU3A4 comes with) is available for 180€.

Hmm. I guess I'll give the Qotom a shot. There are of course other vendors providing J1800 or J1900, but the mentioned Qotom is a direct hit with 4 NICs on board, and the price range is right on the spot as well. So really really thank you.

@moeller0:

Nope, that's exactly what "full dupplex" means. 1GBit full duplex of a certain NIC means the very same NIC can handle 1GB in and 1GB out at the same time.

In contrast, there is "half duplex", which means data can "virtually" go in and out at the same time (as on "for the application it appears as they are simultaneous), but the actual wire switches fast between in and out and thus can only make the given bandwidth as a combination of in and out.

If course, in reality there's always some overlap. If you go TCP, you will get an ACK for every package sent, so sending data in one direction means also receiving confirmation.
When I do "iperf -s" on one side and "iperf -c $sender -n 1G" on the other side (which means sending 1GB of data via TCP) over the internet (which means no jumbo frames but MTU in the area of 1500) I have ~2% of ACK traffic, which means "receiving 100MBit requires me to send 2MBit at the same time". But that's TCP (OSI layer 4) and has nothing to do with the full duplex capability of my NIC (OSI layer 1/2).

[edit]

The thing that goes saturated is sirq, which is not the NIC being at max but the CPU being not capable of handling more interrupts.

(Last edited by golialive on 12 Apr 2017, 10:06)

Let's go through this again, slowly:

your sender sends over VLAN1 to the receiver, but the data first flows though the Router NIC's ingress side, where the VLAN is terinated and it is packed into VLAN2 and send on the egress side of the NIC, so this packet occupied both ingress and egress of NIC. Now let's assume that both VMs send at full blast to the other; still with me?
Now, if VM 1 sends at 1Gbps* it will fully saturate the NICs ingress AND egress bandwidth, leaving no space/time for the second VMs packets. So if both send at "full blast" each is only expected to get 500Mbps. If you had a half-duplex NIC you would obviously only achieve 250Mbps.
Now, I give you that for one sender you should be able to see throughput close to 1Gbps (baring all other obstacles), if you still disagree, please elucidate...

Best Regards
        M.

Got it. So to clear things up:

• I don't do full duplex data traffic, transmitting from VM1 to VM2 and from VM2 to VM1 with 1GBit at the same time and expect that to reach 1GBit.

• I only do traffic from VM1 to VM2. VM1 goes vial VLAN1  to the router, the router translates it to VLAN2 and sends the data to VM2.

This of course should saturate both directions of the NIC of the router at 100% level as long as no other traffic is involved. And this is the situation where my Banana Pi caps at 400 to 600MBit (depending on settings) and my TP-Link at 350 to 400. Both limited by CPU.

So we're clearly on the same page, and probably were from the beginning.

Sorry for the confusion. I didn't even think about my explanation could be interpreted like that .

Regards,
Stephan.