Hey tapper,
You're inscribing yourself into a fairly narrow circle; you require a combination of performance, small size, and passive cooling. Here's what I would look at (in no particular order):
- A tiny PC with dual Ethernet sockets (Intel NUC and clones, Zotac Zbox, etc.) This will be router only and will require a passive switch and/or access point to make the network functional.
- An industrial PC. Personally, I am partial to Lanner FW-74xx and FW-75xx models. They are a bit bigger than a typical home router, about 10.5 x 5.5 x 1.5 in (27 x 14 x 4 cm), have distinctive ribbed tops for cooling, and can come with as many as six Ethernet ports, so you may be able to combine routing and switching in a single box. Wireless will still require a separate access point. When new, these things cost about seven metric tons of money (they are rugged and dependable), but occasionally, you can snag a used one for a reasonable price.
- A quasi-industrial PC. These are usually not ruggedized per se, but borrow the industrial PC's aesthetics (ribbed cases) for the sake of passive cooling. There are some Chinese companies in this space that crank out decent product; look at Qotom and Vnopn (funny names, but in the age of search engine they are distinctive and thus easily findable). Depending on the model, you get anywhere between two and six Ethernet ports. As for built-in Wi-Fi, I would suggest not getting it; the Wi-Fi cards those manufacturers use often do not support the access point mode. On the luxury end of this space, there's Protectli; they have a range of passively cooled products with two, four, or six ports and processors that go up to Core i7.
- A branch router. One obvious name in this space is Sophos; they have a range of boxes which periodically go out of support, so their corporate customers constantly upgrade to the new ones, and the fully functional old ones end up in the secondary market. Note, however, that Sophos has four product lines (maybe more, but I can think only of four right now): RED, SG, UTM, and XG. RED boxes are not x86-compatible, so stay away from them. UTM boxes are actively cooled (although very quiet even when the fan is on). SG and XG, on the other hand, check all the boxes (pun intended). Sophos distinguishes them because they ship with different software; hardware-wise, an SG105 is hard to tell apart from an XG105. Depending on the model, you get between four and eight Ethernet ports and may or may not get built-in Wi-Fi. If you do get Wi-Fi, it may be an N card or an AC card. Specific models I personally worked with include XG85(w), SG105(w), SG115(w), and SG125(w); the "w", if present. indicates wireless capability. Because of their deployment model, you can sometimes get amazing deals on these. I once got an SG105w in a factory box with all accessories for well under USD 100 (I suspect it was a spare kept on a shelf in a large IT department and rarely, if ever, used until it had to be retired because Sophos said so). This is where my Sophos SG105w HOWTO came from...
Passively cooled models come with Intel Atom of some kind (typically, E3xxx). RAM-wise, the older ones have a single 2 GB stick; newer ones may have a 4 GB one. Note the "stick"; it is standard PC memory, which may be upgraded if needed...
As to installing OpenWrt on an x86 box, it's generally on the easy side compared to other platforms. There are, generally speaking, three steps:
Step One: Installation Proper
There are at least two ways to go about it. You can either (1) boot the box off a USB stick with OpenWrt on it and copy OpenWrt onto the internal drive using dd (this is described in the above-mentioned HOWTO) or (2) take out the internal drive and write OpenWrt onto it from your computer, then put it back in and boot.
Step Two: Configuration
Out of the box, OpenWrt may or may not recognize all hardware components, so some discovery may be needed. To date, I have not seen an x86 box on which OpenWrt could not be configured to work properly. With older boxes, there may be tricks involved, but anything 2017 and newer usually fits within the script I outlined in the SG105w HOWTO (also, check out the Check Point U-5 HOWTO; I spent a little more time on hardware discovery there).
Step Three: Expanding the Root Partition
When OpenWrt first runs, it uses only 120 MB or so of disk space, regardless of the actual drive size. It is possible to expand the root partition and/or create additional partitions to take advantage of the entire drive. There is a page on the OpenWrt wiki that deals with this (among other topics):
https://openwrt.org/docs/guide-user/installation/openwrt_x86
There's also my humble effort to make it more easily readable for the new user:
[HOWTO] Resizing root partition on x86
Note, however, that the tail end of the procedure for dealing with SquashFS systems is out of date; the wiki has a more up-to date guidance. I need to put out a new edition, I suppose...