Have 500Mbps+ Internet? My 2 cents

Hardware Questions and Recommendations
1

Do you use an x86 or other high powered machinery to run your home automation, media systems, plex, etc? Do you use VM's? Have you accepted the poor state of OSS WiFi (as host/server, not client) with regards to performance, reliability, stability, features (no fault of the community to be clear) at this time yet? (As of writing this for anything approximately post atheros N era, some limited promise from mediatek to get this back again)

Use OpenWrt as an NVA (network virtual appliance), and enjoy tremendous benefits. Send your wireless through ubuquiti, or the other cheap commercial alternatives or if you don't need advanced features (multi ssid control, vlans etc), run consumer grade in WAP only mode. same goes for switching to some extent (I vlan my wans in through vswitch, aka only one nic required)

Never worry about being dropped support for (x86, 64 bit especially is likely never to be dropped as a platform)

Don't have to bother with drivers, as most virtual platforms are in base x86 built in. Even if you run bare metal, driver management is pretty trivial with the wonderful build system.

Have a couple systems clustered? Literally live migrate your router/firewall :slight_smile: ( I do this on a mixed hw cluster even )

Run as fast as your line rate will take you on even moderate hw from the past 10 years. (I can max a 20Gb hyper-v lacp vswitch with ~15% cpu usage on a i7 3820)

Handle all sorts of heavy filtering, ipsec, wireguard, whatever you want with so much hw offloading it will make you wonder why you didn't do this sooner.

"Why dont you use pfsense/other appliance?"

Cause OpenWrt is awesome, extremely light (boots in 5 seconds, upgrades only take 8 seconds (internet down time)), its proper Linux, not BSD, features are available extremely quickly. There is a reason most consumer router manufacturers use owrt as a base, its awesome and flexible (albeit I don't think those manufacturers use that power in the greatest way, aka hacking old builds into oblivion, proprietary drivers, etc, etc).

Anyways my rant is over, I have used this solution for 5 years in a fairly complex home lab, 6 months of which I had 0 physical access to and it ran solid, no crashes. Do with this information what you will. This project is incredible and I hope everyone involved knows their work is greatly appreciated :slight_smile:

1 Like
2

I see your 2 cents and I raise it by 2 cents.

tl;dr I'm running it on x86 bare metal and I agree on using that for anything requiring performance, my main argument is against using a VM because that means anything happens to your virtualization your internet goes down.

I'm assuming you are talking about opensource without firmware, because Ath10k or whatever mediatek has with wifi ac is far from poor with opensource drivers with wifi firmware loaded.

And watch how your virtualization infrastructure becomes critical since any issue in the virtualization software stack or hardware that breaks the VM networking will break your network.
I can deal with my NAS and buildbot and password manager sync and website being down, but if I lose network connection on everything apart from my phone it's a big issue since I usually need that to search info and fix whatever issue I'm having.
I mean yeah I can tether my phone and whatever, but why adding a failure point like that.

And enjoy using outdated and vulnerable wifi drivers on the APs because the manufacturers don't care to push firmware updates in years if ever.
Or in the case of Ubiquity, enjoy seeing features you have used for years get removed in a firmware update because they decided it's too good for you (they actually did this).

One of OpenWrt's stronger use cases is on dedicated wifi APs, because being able to send a firmware update to patch whatever is the wifi vunlerability of the month within a week or so is very nice.

That said yeah, separating the wireless AP hardware from the routing hardware makes a lot of sense for better wifi coverage, upgradeability, price/performance and so on.

Dropping support is a thing that is going to be less and less common in the future, since the main reason is firmware size and RAM requirement growth. But that is a more or less linear increase, while RAM and flash size increase in newer devices is exponential, they go up by powers of 2.
Devices nowadays come with 256 or even 512 MB of RAM and OpenWrt is still just dropping support for 32MB devices after like a decade of its life as a project.
In a few years devices with 1GB of RAM will be the norm, while OpenWrt will still be using less than 64MB for itself (wifi drivers can increase this number)

Drivers for their own hardware are always built-in in all supported devices.

1 Like
3

This opens another issue as well, as you have to duplicate a lot of your network configuration in two places now, once on the host (to selectively forward interface to the VM running OpenWrt) and the actual VM doing the routing as well, which is both fragile to failures and exposes the host (which is rarely updated as eagerly as it should be, as doing so involves downtimes) to the internet.

So yes, I fully agree with this sentiment - OpenWrt on bare iron x86_64 works well. Using virtualization for testing or intra-VM routing can be very useful as well, but I wouldn't want this for my connection to the internet.

4

some clarifications:

It's not "duplicating a lot of your network configuration".
The VM's virtual network device is added to a bridge with the physical interface and has its own IP and acts like an independent device on the same network, or the card is creating a virtual network port with SR-IOV, or you are passing through the whole network controller to the router VM.
This is trivial to do in dedicated virtualization hosts like Proxmox, XCP-ng+XenOrchestra and VMWare's vsphere/esxi/whatever.

Also most virtualization software like Virt-Manager/KVM, Virtualbox, VMWare's player/workstation/fusion and Parallels can do the first one (the bridging with physical interface).

Yes it is an additional point of failure, but it's not necessarily exposing the host to the internet, nor really fragile per-se.

For example if you have more than one network interface you can dedicate the second one for the router's WAN and just leave it unconfigured for the host (unconfigured in the sense of it's just bridged but has no IP nor dhcp).
If the network card is using SR-IOV or is in full passthrough you also have full isolation.

What I've seen people do is use a single port for everything (also with out virtualization, the OpenWrt/pfSense/whatever is installed on a device that has a single ethernet port) and set up VLANs, which means that traffic is isolated and the host is secure, but now your managed switch is connected to the Internet, which is not a good idea unless it is running OpenWrt or a similar opensource OS/firmware where you can be sure nothing is listening on the "wan" port.

Modern dedicated virtualization hosts like Proxmox, XCP-ng+XenOrchestra and VMWare's vsphere/esxi/whatever (if you pay the licenses) natively support clustering, and live migration of VMs between hosts.
This means that when you need to do maintenance on a host you migrate the VMs to the other hosts and then you are free to reboot and do whatever without downtime for the VMs.

Since now even relatively old small form factor office workstations can run 32GB or even 64GB of RAM, many home labs are using multiple smaller hosts running as a cluster so they can do maintenance and updates on the hosts without downtimes (since the VMs are migrated online to another host in the cluster), also because that's what the "big boys" do in real businness environments.

You need to have the same network setup in all your hosts of course or the VMs will not be able to reach the same networks.

Fun fact: this feature is very expensive in VMWare (for a home user anyway) because you need to pay advanced licenses for the esxi hosts, and a license for the vCenter appliance that coordinates the cluster. So anyone that is doing this in his home lab on VMWare is most likely a filthy pirate.

5

Its like everything else in life, opinions are plentiful. It ain't nothing to setup both cost and time (again can very significantly depending on how you do things), but I maintain some of the benefits over the long term are hard to ignore. Never having to reconfigure the firewall itself even with host hardware changes as the physical is abstracted in another layer was another big one for me. No argument from me about adding reliance on another stack, I just happen to need most of my other stuff running on it up all the time too. Knowing that even if my hw cluster fails, I can with very little effort (apply switch port config to another port/device), move or migrate or restore (depending on failure type) my fw vm to any other metal I have lying around the house is plenty for me (I accept and respect those that don't feel the same way). I think one of the strongest cases to using off the shelf owrt compatible hw is for breakglass, something where having a great feature set helps when you need it, but performance isn't paramount. I can wire up something like that with a cellular modem, or another IP if the ISP supports it, and have a way in if my virtual stack dies. On the switch front, if my switch dies, everything kicks it anyways. I appreciate all in one devices running openwrt alleviate some of this, but a process crash due to resource exhaustion, or a driver fault that doesn't recover, or any other cause for a random reboot or lockup, 1 either doesn't recover itself (I admit those are rare), or 2 take a decent amount of time to start back up (I don't have any personal experience with a device that doesn't at least take 1 minute).

Also on switches, I am super stoked that there are some owrt options now that have a high port count :slight_smile:

On the wireless front, I just haven't found openwrt devices that hit anywhere near what the proprietary similars, or equivalents (running stock, vs owrt) provide in terms of performance and stability in recent years. No contest on the security front, that I will wholeheartedly admit. Though not as good as the old days with full OSS wifi stack, with no blobs. Then again I treat anything over wireless as close to untrusted internet as possible. Wanna crack my SSID? Have at it, but the worst you'll do is put up some nyan cat on my tv :stuck_out_tongue:

In all reality, most folks can be ddos'd simply enough with interference/invalid connection requests over wifi, or simply bombarding their internet address with enough traffic they run out of bandwidth.