Hi all
Perhaps somebody here can advise me on if OpenWrt/LEDE is a good starting point for my project.
I want a single low-cost device that is a switch, a firewall, and with dumpcap or tcpdump captures some part of the frames arriving at all ports before or after the firewall rules are applied.
It should securely forward the captured data to another host.
For example, I might want it to forward just the destinations of traffic from a particular host.
It will be sufficient to SSH to the device to make configuration changes via CLI.
Is this possible with just configuration of OpenWrt/LEDE, or will I need to add functionality?
I have some C programming experience on Linux and a Ubiquiti ER-X that I could use, but I am happy to buy something more appropriate for this project.
You were OK until "[pcap] after firewall rules are applied".
- low-cost -- check, if that means under $100 for a few hundred mbps, and under $200 for closer to 1 Gbps. You need the CPU power to handle the interfaces, run pcap, and encrypt the data for the ssh connection
- switch -- check, most can "mirror" a port to another port (such as an interface on the SoC) -- make sure the device has at least two interfaces on the SoC (not just 5 ports on the back!)
- firewall -- check, iptables "by default" and nftables can be installed from packages as an alternative
- tcpdump -- check, tcpdump-mini package is probably sufficient
- ssh -- check
Since pcap runs on an interface, that either needs to be as the packet is coming in an interface, or going out. If you want to look at packets in the firewall, you'd have to hack your firewall rules to replicate the packet to a pseudo-interface of some sort. If you really only need packets entering the box and leaving it, then OpenWRT should handle it at moderate throughput rates. For anything over a few hundred mbps, at least a multi-core ARM device is likely needed. Approaching 1 Gbps I'd recommend arm64 / x86_64 with my personal preference being for FreeBSD for network-intensive tasks, Debian if you require or insist on Linux.
Thanks Jeff
Broadly my thoughts were to start with as light an OS as possible, like OpenWrt/LEDE, capturing from a transparent bridge with combinations of ingress and egress firewall rules on the bridged interfaces in promiscuous mode. I have not thought it all through and done all the necessary tests at this stage to see if this would all hang together. Still looking for a best overall scheme before throwing a load of time and some money at it.
Low cost is because I want to deploy a device behind a few hosts, the less they cost the more I can deploy.
You are correct to consider the CPU. The main cost I think is dissecting frames. In my experiments running dumpcap on âCentOS 7 minimalâ, a Celeron N2940 (x86_64 four single thread cores from 1.8 GHz to 2.25 GHz with TDP at 7.5 W) can be stressed long before a single 1 GBE hits capacity, and the whole instance uses about 850 MB RAM. So, itâs very important to reduce the traffic as much as possible before dissecting it, and running in a fuller Linux distro is going to require more RAM pushing up the price.
Do you mean to ensure the ports are not on one hub?
I did not know about the tcpdump-mini package, might be important.
With device cost as a primary driver, a router running OpenWRT is likely a good choice.
I don't think there's going to be a big difference between OpenWRT and another Linux-based distro running the same amount of processes. "Full" distros tend to run a lot of things you don't need, with the more consumer-focused distros being more challenging to strip down than the ones that tend to be server oriented. My "little" FreeBSD 11 firewall / router is consuming around 800 MB of RAM on a 4-thread, 2 GHz, Celeron J1900. If you're going to go to PC-compatible hardware, I find the environment of a server distro much easier to work, especially on an interactive basis.
You might also want to consider some of the "Pi-like" devices that aren't crippled by the way the Raspberry Pi designers connect the Ethernet interface. One such is the ODRIOD-XU4. There are many more. A USB 3.0 GigE adapter, on a USB 3.0 capable device that doesn't conflict with the onboard Ethernet can make a single-port device into a two-port device for under US$20 additional. I'd look for something with at least 2 GB of RAM.
On the ports, older SoCs had a single Ethernet phy that was connected to the switch. VLAN tagging was used to split the "WAN" from the "LAN", so all traffic flowed through a single phy. Newer SoCs have two phys, one typically used for the WAN and the other for the LAN. In your case, I'd set it up so that one would be the "monitor" port and the one to feed the packets out to your post-processing machine over ssh.
You make a sage observation. The limiting factor other than price is processing power. In fact, processing power is such a dominant factor that like you say the OS choice is not significant.
The OpenWrt switch functionality would be useful for a device to handle a few hosts, but on reflection if I add a switch to the network I am changing how the network behaves. Without care traffic might for example end up in the wrong VLAN. Avoiding such issues would make deploying them a lot more work. So I think an L2 transparent device is the way to go.
Do you have any advice on how to avoid the pitfalls of poorly connected Ethernet ports? Its the sort of information that is not easy to find - I have never seen that declared in a data-sheet.
FYI, I just turned off everything I did't need on Fedora 28 Server running on a NUC and it used just 325 MB RAM. I read that Tiny Core Linux can run in 16 MB, but I would no doubt need to add a lot to it.
I'm not sure on the "Pi-like" devices how to tell if the Ethernet is "choked" by the SoC connection or not. For Raspberry Pi devices, as far as I know, all suffer from the malady, even the Pi 3B+ with its GigE phy. As I recall, the Raspberry Pi designs use USB 2.0 connectivity to the phy, so are limited to on the order of 200 Mbps. There's a good explanation and examination at https://www.jeffgeerling.com/blogs/jeff-geerling/getting-gigabit-networking
If the throughput is noted as limited to significantly less than 1000 Mbps, then it almost certainly is choked. If it advertises that it avoids the Raspberry Pi choking, then it probably is connected in a higher-throughput manner, but confirm with people that have the device or testing on throughput that you can find.
On SOHO routers, checking the OpenWRT wiki to see if it has two phys in the "switch configuration" section is helpful. You might have problems creating a transparent bridge with them as the connections from the physical ports to the phys are through the switch chip. As such, you might have to configure every VLAN that is active on the network segment you're examining.
My current choice for inexpensive (under US$150), low-power (7 W, typical), x86 hardware is the PCEngines APU2 series. Quad-core, 1 GHz AMD SoC, 3 or 4 Intel NICs, mSATA and SATA support (can run mirrored ZFS, for example), serial port, and more. I have several running both FreeBSD and Debian without having to jump through any hoops (past Linux's challenges with ZFS, due to GPL).
Actually the 3B+ is even worse, as the previous 100 MBit/s phys acted as a natural rate limiter, while their current choice really chokes the USB 2.0 bus by providing more data than the SOC can process.
Thanks again Jeff
The PCEngine APU2 series is new to me and looks interesting, especially if as you say it plays well with Linux. Will examine that more closely as a hardware option.
I like the look of the Odroid-XU4, and in fact I have an Odroid C2. If possible though I would like to go with passive cooling as I have experienced a chip failure in the past when a fan stopped.
Currently I donât understand enough about problems with creating a bridge between Ethernet ports on the same PHY. So thanks for the note, I will look into it more.
Have used a few RPi 2B and 3B+. They are great for low power and light processing jobs, missing a few features for some of my use cases, like WOL, eMMC storage, and a power button. I donât want to be fitting all sorts of extra bits because it raises the price, takes time, and sometimes throws up reliability and compatibility issues. Anyway, for this project they will not have enough processing power.
Nonetheless, as I use them a fair bit and since you mention the bandwidth sharing of the RPi Ethernet and USB, which I was vaguely aware of, I looked again at the specifications. The RPi3 B+ claims âGigabit Ethernet over USB 2.0â. I know roughly nothing about electronics design, but I take it that this means the GbE port uses a USB 2.0 bus: 480 Mbps PHY limited to 280 Mbps after PHY management overhead so in use it will be a little less. That makes sense of the 222Mbps and 214 Mbps in the tests you linked.
I looked a bit deeper and found this statement âThe Raspberry Pi Model B is equipped with two USB2.0 ports. These are connected to the LAN9512 combo hub/Ethernet chip IC3, which is itself a USB device connected to the single upstream USB port on BCM2835â and âOn the Model A, the single USB2.0 port is directly wired to BCM2835â. So, on the Pi B I take this to mean the Fast Ethernet port and two USB port shared a single USB 2.0 bus while on the Pi A the single USB 2.0 had a separate bus from the Fast Ethernet. This would make USB and Ethernet IO on the Pi A faster, as long as it did not become CPU bound obviously. I think the RPi 3 USB-Ethernet arrangement is the same as the RPi 2, although the LAN9512 chip is replaced with the LAN7515. The takeaway from all this is that a single USB 2.0 bus is shared by all Ethernet and USB traffic on every RPi except the original model A â if I have interpreted all this correctly. In a sense slh is correct i.e. the use of a GbE port allows it to swamp the bus, although if you are not using the USB much that can still be a good thing. However, itâs important to understand that high IO in one can affect the other.
Thanks slh, my comments about your point are in my last reply to Jeff as they fitted well there.
Every RPi variant so far had only a single USB 2.0 port on the SOC, all onboard and external USB peripherials need to share this single port - for most variants this fact is just being 'hidden' by an onboard USB hub (and yes, this is mostly a combo-chip acting as ethernet card and hub).
The XU4 is an interesting unit. I bought the passively cooled unit through a US distributor and was moderately happy with it. I'd definitely put a little fan on it if I pushed it hard, like a 40 mm Noctua, 5 V unit. With its onboard GigE and an AX88197-based USB-to-GigE adapter from a reputable manufacturer, I couldn't get the bandwidth and processing I wanted with a Linux-based OS.
Rather than try to beat the XU4 into submission, I switched to an APU2C4 for that specific application. The APU's amd64/x86_64 architecture seems well supported with both FreeBSD and most Linux-based distros (likely including OpenWRT). The only "tricky" thing is that the SoC has onboard video, but no exposed video port, so you have to configure your console as serial else you have some install-time challenges. PCEngines sells decent, reasonably priced mSATA drives as well. If running Linux on a multi-core processor for this type of application, I'd make sure that you've got irqbalance up and running to spread the load a bit. FreeBSD seems to handle this without additional software.
The challenge with a transparent bridge is that there isn't a cable port physically attached to the phy itself. If you had two phys, each with their own direct-attached port, you could simply bridge them in the CPU and monitor them to your heart's content. With the chip-based switches, there are typically 7 or more ports. Two are attached to the phys, the other 5 are attached to cable ports. Internally they get "wired" up with VLANs. Not from any real device, but one arrangement might look like (in a "normal" configuration):
0 -- attached to phy0, VLAN 1, untagged
1 -- attached to phy1, VLAN 2, untagged
2 -- attached to WAN, VLAN 1, untagged
3 -- attached to LAN1, VLAN 2, untagged
4 -- attached to LAN2, VLAN 2, untagged
...
To transparently bridge LAN1 to LAN2 and monitor on phy1, you'd need to assign every VLAN in use on your network as tagged to LAN1, LAN2, or tagged traffic would not be bridged between the LAN ports. You would either need to also assign every VLAN to the phy1 port as well, or use "port monitoring" if the switch chip and its driver support it. A warning about these consumer-grade switch chips in "sensitive" applications is that they usually come up in a "wide open" configuration on power-up. They can "leak" between ports until the switch is reconfigured during the boot process.
Yes, it seems like you've got the gist of the USB bottleneck in the Raspberry Pi series.
Certainly there is bus sharing on the original model B and RPi 2 and 3. As I am unlikely to use an original model A again I can't bring myself to check that in greater detail. Bus sharing could be a significant factor for some projects, so its good to understand it.
RPi have their uses, ultimately enabling many uses cases where higher cost compute units would not be tolerated. I often see them in use in businesses for a whole range of purposes, but they have a low price for a reason, so we should not expect too much of them. I am glad they came to market, as they got me to look seriously at low power compute units.
I have never used ZFS, but you have got me interested. I will be looking for use cases for that from now, although as you say there are issues reported around using it that stop me using it until I understand it and its issues better.
Performance comparison shows the AMD GX-412HC at 1.2 GHz (canât find numbers on the 1 GHz GX-412TC used in the APU2C2) is quite close to the Celeron N2920 and probably fast enough. Performance to price ratio is very similar too. However, I can get a Chinese J1900 device (so circa 22% faster CPU mark over the GX-412HC and N2920) with 4 Intel NIC 2 GB RAM 32 GB eMMC (but no circuit design so not sure what I am getting) for very similar money.
Glad you mentioned irqbalance, not used it. Docs on it says its for balancing across processors rather than cores. So I will experiment a bit with it.
Great detail on the bridging requirement. This sounds like it will be the biggest challenge as I clearly donât understand it well enough yet.
Also Interested to hear about the cross-port leaking before boot is complete. In fact, a few weeks back I looked into similar behaviour I have detected multiple times over the years in different in running commercial switches. The more I looked the more special cases I found. When I got to ten I stopped. Broadly, in some special cases a switch may not be certain of trafficâs destination port, so rather than just drop it broadcasts the traffic so that it will arrive where it should, even it arrives where it shouldnât too. I have no doubt that a network exploit using this will be created, if it has not already.
Thanks for all the help; you know a lot.
The "issues" are very straightforward -- Linux and many Linux-based distros are licensed under the very restrictive GPL-family, ZFS is licensed under a much more open license. As such, you can't distribute the two of them together in binary form. You just have to build the ZFS modules from source during install, then on your target machine. It's straightforward. See for example https://github.com/zfsonlinux/zfs/wiki/Debian-Stretch-Root-on-ZFS
Nor the quality or authenticity of the components on the board or the build quality. Take a gander at, for example, the pfSense forums for some enlightening opinions about the cheap imports. In a similar vein, there are lots of new "Intel" NICs out there selling at "surprisingly reasonable" prices that are certainly couterfeit. See, for example, https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/
On the consumer-grade switch chips "leaking", basically they come up with a "stock" configuration at power-on. In contrast to a good SOHO- or enterprise-grade switch, the ports are not "held off" until the switch is configured. So you get whatever that switch chip has as its power-on configuration for the minute or so until the OS configures it. This can mean that, for example, all the LAN ports are bridged together for that period of time.
Yes, I looked at ZFS for CentOS and Fedora, it seems DKMS is the answer to rebuild the modules on upgrading the kernel. I can see ZFS being useful for typical servers, but it sounds like it will be a bit costly on performance for this project.
I suspect I will not find the right compute unit first time on this project, as it is hard to estimate the actual performance cost.
Agree re cheap Chinese copies. I remember one chip maker complaining that some firm over there copied one of their chip designs so closely it included the designers name! I have also spoken someone that had his equipment made in China, turned up at a trade fair with it to find them selling it under their name.
I only buy from suppliers in my country, so if its rubbish I can send it back, but its hard to tell if something contains fake parts, and harder to convince the seller.
ZFS is for a lot more than servers. I run every system I can using it because of its ability to perform instant, "no cost" snapshots and cloned filesystems, no need to pre-allocate file systems, and a host of other great features. On-the-fly compression and de-dupe can significantly increase useful drive capacity (1.5-2x in my experience). Mirrored drives add redundancy with ease. The speed is comparable or better than other current filesystems. Throw in some cheap SSDs for L2ARC and ZIL and slow, spinning-platter drive for backing store become blisteringly fast. I've been using it for many years for FreeBSD and finally Linux has a working implementation (though no well-supported beadm that I've found yet).
I agree with @jeff here, I'd also go for a x86 (64-bit) solution running FreeBSD or possibly pfsense although FreeBSD will probably be a better choice in this case.
Depending on network (throughput) speed I'd be very careful going for a rather old/slow GX-412HC solution.
I'd recommend at least Intel Gemini Lake or newer
ASRock J4105M should be a pretty good starting point if you have a few parts around otherwise https://up-shop.org/28-up-squared might be of interest, Realtek NICs aren't great but given the price it's not a bad deal.
Interesting boards -- I've bookmarked them.
I agree that the GX-412HC aren't screamingly fast and may have problems with high-bandwidth connections. I could test what one can bridge and either tee to the third interface or tcpdump and send over ssh with FreeBSD. I know they're fine at 300 mbps and would guess that 500 mbps wouldn't be a problem.
The Realtek NICs are reasonably decent these days (10 years ago they really, really sucked under load), though I'll swap in a real Intel PCIe NIC (lots of counterfeits on the secondary market) if I'm going to depend on "100%" performance.
If you do go with an embedded amd64 / x86_64 board, I'd go with 4 GB or more. While you can run ZFS with only 2 GB, you lose some of the advantages such as prefetch. It's generally a small up-charge that can either improve performance now, or extend the useful lifetime of the board.
You have me convinced on ZFS. I will try it today on a Fedora host I know has a lot of unused SSD space. The bit I have read makes it look like I have a lot to learn about it though. I was under the impression from some of what I read that ZFS required more resources than many file systems, and extra admin to get good performance from it. Have you ever tried to convert another a file system to ZFS, or perhaps more to the point would you consider doing that on a live system?
If I have a âseriousâ data requirement on a server or workstation I always go for hardware RAID 1. My experience of software RAID has not been good. Having said that, storage space is cheap, so if ZFS is doing software mirroring on hosts where I would not normally install hardware RAID 1 then I suppose it is an advantage, if it comes at reasonable admin effort. It would be extra useful if the mirroring could be across hosts, say to SAN/NAS.
Thanks diizzy
I only have BSD through a pfSense instance. To take on a new OS I would need to find a significant advantage in it over the two I have converged on to now: CentOS and Fedora. It sounds like support for ZFS is better integrated, so that might be the trigger if I find I canât live without it.
If I can justify the cost, my low power processors of choice are Pentium Silver N5000 with TDP of only 6 W and circa 50% faster than J1900, or the J5005 with TDP of 10 W like the j1900 but circa 60% faster than it. The problem at the moment is finding them in any SFF device. As far as I can see only the ASRock J5005-ITX and the Gigabyte GB-BLPD-5005 have the J5005, and an MSI device has the N5000. Either way, by the time you add it all together they come out noticeably more expensive than a J1900 device, as I suppose they should. The J4105 is circa 46% faster than the J1900 also with TDP of only 6 W and is also a fair bit cheaper than the other two, so I agree itâs probably the best choice if the J1900 cannot do the work. It is also available in more devices, like the Gigabyte GB-BLCE-4105C and some ASRock boards. The other problem is they all have only 1 NIC, so I would have use a USB 3 to NIC adapter like Jeff said. I have had problems with these not being recognised on CentOS, but Fedora has more current drivers, so I may have to use that. Thanks for the link, they have dual NIC devices, but I am in the UK so I suspect the postage will be high.