Normally I'd use OpenWRT in a VM, but I also do things on an old all-in-one I found to learn about how it'd work, learn about the whole single-interface-shared-among-many-ports thing, and other "quirks" from these devices.
I learned quickly it has basically no resources for anything; multi-homing the device even if it's not the gateway of anything seemed like a bad idea. It was already running out of memory with just a handful of interfaces. I still needed it to know the VLANs though, fortunately they could be added in the switch section, and sure enough, memory was freed. Not a lot, but still.
For all intents and purposes, the device as I set it up, appears to work OK. The docs are all over the place but using my own experience I connected the dots and I got this much; the CPU port for VLAN0 (as in VMware, otherwise known as the native VLAN untagged--1 in this case) must be tagged, but in the switchport it should go like it should go (untagged). I don't know how to put it into words, but tagging the native VLAN internally makes all the sense to me. Done, done.
When I SSH into the device, I can reach it over VLAN0, or in my regular VLAN, 9; where I added a second interface. There's one other but it's still "unfinished" so to speak.
My workstation is also multi-homed, it has interfaces on those two VLANs, and here's where it gets odd: if I disable its interface (workstation's) on VLAN 1, I can still reach the tp (OpenWRT…it's a TP-Link device), the connection doesn't drop since theoretically I should be reaching it through the workstation's default gateway which then would route it to subnet/VLAN1. Right?
The workstation has several more interfaces other than VLANs 1 and 9, though. So I start a ping to tp's VLAN1 interface, it's echoes back. My workstation at this point only shares VLAN9 with tp, ws' VLAN1 has been shut down. I then kill ws' VLAN9: tp stops echoing.
The tp still has both of its interfaces up, both have gateways. The ws no longer has interfaces on VLANs 1 or 9, but it can still route to them through other interfaces, the echoes shouldn't have been interrupted. I restarted the ping in case it stopped due to failing to reroute, ARP caching, something. But still no response.
It seems like when I was contacting tp's VLAN 1 interface, the traffic was getting there through its VLAN9 interface which it shared with the workstation. I assume the workstation went with it because tp sent it ICMP redirects or something along those lines. Once it had no L2 access though, that was it for both devices. There are a few other issues I can't quite explain that as I said, it looks like things are OK, but they aren't quite. For instance, it appears to be be sending like really hardcore router advertisements because some hosts are unable to connect through IPv6 despite these having not only static IPv6 addresses, they already have a default gateway set. I'm not sure how it managed to steal the route.
And by the way, the tp can connect to the Internet, meaning it must be aware of its gateways A gateway at least, right? Therefore it should be able to route back regardless of where is the other host pinging from. The firewall service is disabled on the tp.
I'd love to hear some comments on this 'cause I'm a little puzzled about it. I multi-home routers all the time and it usually works…of course they tend to have discrete interfaces or a proper 8-0-something-dot1q* trunk.
*
It doesn't matter how much I try, I'll never be able to memorize the last numbers or or never not get them mixed with wireless. In my defense though, I know the std's letter is correct.