SOHO Network Topology and the like; care to share?

Done that.

When you configure the interfaces from the CLI on OPNsense (and this is similar to other distributions) you have the option to assign LAN and WAN to an interface. I only have one physical interface. So I assigned LAN to the actual physical interface em0 and WAN to the VLAN interface em0_vlan10 that tagged for Vlan 10 (WAN). This is where I was not sure, if my grasp of physical vs logical was on target. But I believe this should be right?

Ah, I think I did Vlan 10 tagged for this port. I will change that to untagged. Vlan 1 remains untagged for this port as well?

Thanks a bunch!

No, you can only have one untagged VLAN per port, because it has to decide when it receives an untagged frame which vlan it should be in, and it needs to be unambiguous. Every frame from Tomato needs to be a part of vlan 10, and since you aren't making tomato tag its frames, when the zyxel receives the frame it needs to know to tag it with 10, so hence untagged frames are tagged 10 (that's what the port/untagged membership means).

I've been separating all my "untrusted" devices onto their own SSIDs for years, each with their own VLANs (as well as guest networks). Not only does this help in the event of a compromise with the device, but also keeps them from snooping more than just their own kind. On their own VLAN, you can not only restrict devices from "phone home" actions (for example, TP-Link home automation devices connect to the TP-Link cloud even though you "turned it off"), but also provide locally managed "custom" DNS. I run unbound for DNS and kea for DHCP on a non-LEDE/OpenWRT host rather than dnsmasq for both clearer and more flexible configuration, as well as a belief that the security of those two products is greater than dnsmasq.

The more complicated your drawing, the harder it will be to understand "what if" questions.

Firewalls are sort of the opposite. The fancier the GUI, the less likely it is that you can understand the rules they produce.

You may have problems with WDS and VLANs, as VLANs are an Ethernet concept and, at least last I checked, weren't supported as an 802.11 standard. You'll probably need a Level 2 encapsulation to get out to that remote group if you have more than a couple VLANs to bridge. You'll see recommendations of using OpenVPN or the like, but the throughput when running on home-grade WiFi platforms is pretty poor. If only a couple, and your WiFi drivers support it, you can use multiple SSIDs, one for each VLAN you want to bridge.

As for myself, I have always assumed that WiFi is not secure and that it was the responsibility of the attached devices to encrypt, so tunneling without encryption is sufficient for me (GRE, for example). Hence one of the reasons my fireplace switch is still manual; it would annoy me if someone turned on the lights in the middle of the night, but at least it wouldn't burn down my house.

Going along with this, I don't "trust" any WiFi-connected devices to have direct access to my internal networks, even if they are "known" devices. You might want to consider requiring authenticated VPN access.

I'm not sure how you're managing the DHCP from the modem, either for IPv4 or for IPv6 with the /60 that many Comcast head ends will supply. Also, you need to be able to access the modem's management interface. The way that I handle it is with a "box" that is directly connected to the modem that isolates the rest of the network from potential DHCP changes, as well as allowing my infrastructure-level management VLAN to access the modem's interface. My main firewall always knows it's outside interface address is fixed, simplifying the rules.

I separate my network devices' management interfaces from those for service hosts as I very seldom need to manage the network infrastructure. Your VLAN configuration needs to be protected as strongly as your firewall rules. VLANs are pretty much "free" with decent hardware. There is some risk of a rogue device coming online and "guessing" the right VLAN to get more access than they should. To help mitigate this, all segments require the correct VLAN tagging for the port; nothing is assumed at the switch.

My setup is a minimal Ubuntu system running on PCEngines hardware in front of the modem that handles DHCP from Comcast and provides one level of firewall with netfilter (not iptables). DHCP is handled by dhcpcd, as unpatched ISC DHCP doesn't handle both the interface and the /60 PD.

Everything is handled by Cisco SG300-series switches. The Ubuntu "talks" to my main firewall, running FreeBSD and "raw" ipfw. LEDE/OpenWRT handles the WiFi portion of the links, associating SSIDs with VLANs and either routing them over wired or over 802.11s backhaul between the units to the main firewall. With a single point of connection between the various network segments, it's easier to visualize and rationalize the paths and what data flows where.

DHCP is supplied by kea and the Cisco DHCP relays built into the switches. These Cisco switches can restrict the ports on which DHCP servers can be located, "preventing" a rogue DHCP server from impacting your entire network. Local DNS is provided by unbound which goes "straight to the source" rather than trusting Comcast's servers. DNS is custom for the network served, providing local knowledge "on a need to know basis". NTP is local. Service hosts are typically run within FreeBSD jails, with full virtualized network stacks in each jail.

File servers run FreeBSD on mirrored ZFS. Rolling snapshots provide at least a little short-term hope against ransomware. Backups are the fall-back. I do not run any SMB services.

If you don't fully understand IPv6, you might want to get things working with IPv4 first, then slowly introduce IPv6. Just remember that assuming that NAT magically protects your network is asking for trouble.

1 Like

That's all I do right now as far as segregation is concerned. I have all phones, camera, media devices in separate Vlan with its own SSID.

Hopefully that's something I will moving towards to. More segregation, more SSIDs. All cloud cameras in one Vlan, all Androids and iOS devices in separate one, etc. Seems to me like it could easily get too complex fast.

You might be right. If so, I will need to redesign my whole approach. I though I saw some options for Vlan settings on the Engenius that I will use. I'll see.

When you say internal network, you mean trusted network part of your internal network or are you talking about the known devices on your untrusted network that are on a separate Vlan? You would want to use VPN for that?

Could you elaborate on that? How do you enforce that through the Vlan settings?

Using Vlans, correct?

Your setup is probably too complex for me. i am already struggling with just getting the Vlan porting figured out. :stuck_out_tongue_winking_eye:Thanks for sharing though! i can tell you put a lot of thought into it. Curious; is this for business or home?

Ok, I now got OPNsense (OPNs) to pick up an IP from 172.16.1.x.; after I reconfigured the Zyxel as you suggested; and after I blew away the interface configurations on OPNs and assigned the physical interface em0 as WAN. I haven’t tested anything else yet. That probably will have to wait until the weekend.

On the Zyxel
port 1 is Vlan 10 untagged; Vlan 1 tagged
port 2 is Vlan 1 untagged; Vlan 10 tagged

I wasn’t sure, if port 1 Vlan 1 tagged is correct?

My understanding of Vlans feels fragmented. I've re-read your instructions and explanations above and they make sense and I watched some more videos; but I don’t feel any more confident. Maybe that’s partly because I don’t find the Vlan portion of the Zyxel to be very intuitive. Although the device is sold and was a great recommendation.

The concept of all of this seems straightforward enough, but every next step I take seems like a brand new puzzle.

For example, Zyxel allows for more than one Vlan to be selected as untagged for one port; or maybe that’s unique to the native Vlan 1; or its an idiosyncrasy of the GUI?

Under Port settings, should I leave everything for now as it is, in particular Accepted Type and Ingress Filtering, or will this come into play later? What port needs to be a Vlan trunk? I thought the port on the Zyxel that the OPNs is connected to should be a trunk and each LEDE? Now I am not so sure anymore. Argh.

Do I need to mess with Membership options such as Forbidden and Excluded for each port / Vlan?

The management Vlan confuses me as well because on the Zyxel it defaults to the native Vlan 1. Do I need to create a separate Vlan to securely manage my devices (OPNs, LEDE, etc.) with 802.1x as we discussed?

Mon Dieu! :scream:

I think I severely lack an understanding of what logical steps to take when designing the architecture of a Vlan topology and how that translates then into what settings to use. There’s a disconnect. I think my drafts are garbage. It’s a mesh of physical and logical topology.

Do you know of any good resources that explain Vlan methodology or approach? Like short 10-15 minute graphics driven tutorials that just throw out a couple of simple examples, but emphasize the best practices approach, like first you need to think about this, then that, etc. What I’ve found thus far is either overly complex or too simplistic. The Zyxel videos on this were utterly useless.

Sorry for these ramblings. Feel free to ignore me. Ha, ha...

Once you get the hang of VLANs, it is deciding on topology and numbering (including your IPv4 and IPv6 subnetting) that is the challenging part, not the implementation. I'm old school and just think about them as Ethernet cables when doing my planning.

For carrying VLANs over WiFi, you'll either need each on their own SSID, or a tunnel. There is a good write-up of using GRE at Vlan trunk over wlan Be aware that busybox doesn't implement ip tunnel as configured in at least the TP-Link Archer C7 images and a custom build is required. There isn't an easy way to replace busybox on a running system https://bugs.lede-project.org/index.php?do=details&task_id=1380 but it can be done very carefully by extracting the self-build package yourself.

I consider every WiFi device to be untrusted. For a WiFi-connected device to get access to resources on any of my internal nets, it needs to be via a secure and authenticated path; ssh or VPN.

I don't know the workings of your switch, but on the Cisco SG300-series, you can require all packets coming in to be VLAN tagged on a port-by-port basis. Combined with the VLANs associated with the port, this means that you have restricted to only the VLANs you intend to be accessed through the port (or switch-to-switch trunk, for that matter).

It's a home network and it's not all that crazy to work through. I start by thinking about resource type (UPS/power management, switches, access points, service hosts, end-users, printers,...), what services they need (DHCP, DNS, NTP,...), and how and from where they should be accessed (management interfaces and service interfaces both).

On implementing VLANs on a switch, finding the screen or CLI command to give you the information you need can be challenging. For any connected device that is VLAN-aware, you generally set that VLAN as tagged on the port. If a connected device is sending untagged packets to the switch and expecting the same in return, you need to set a PVID (default VLAN) for the port, and generally allow untagged packets for the PVID on that port. When I know the remote device only sends VLAN-tagged packets, I block untagged ingress and egress from the switch.

Maybe some ASCII art will help?
Switch port set for PVID of 100
Configured VLANS on switch are 100, 101, 102, 103

There is physically only one "wire" but, in my head, I sometimes think of them as individual wires when reasoning out how things will work.

"Dumb" remote device

  • VLAN 100 is untagged
  • VLANs 101-103 not enabled
    Any untagged packets on the wire become VLAN 100 internally
    All tagged packets are rejected (assuming you have set it that way for the port)
VLAN 100 internal <-> port <-> (untagged) on wire <-> printer
VLAN 101 internal  X  port <-  VLAN 101   on wire <-  rogue
VLAN 102 internal  X  port <-  VLAN 102   on wire <-  rogue
VLAN 103 internal  X  port <-  VLAN 103   on wire <-  rogue

"Smart" remote device, only uses VLAN 101 and 102

  • VLANs 100, 103 is not enabled
  • VLANs 101-102 tagged
    Edit: Even when only using tagged, you generally need to set the PVID. Some switches require that it be a VLAN configured for the port.
VLAN 100 internal  X port <-   (untagged) on wire <-  rogue
VLAN 101 internal <-> port <-> VLAN 101   on wire <-> remote server
VLAN 102 internal <-> port <-> VLAN 102   on wire <-> remote server
VLAN 103 internal  X port <-   VLAN 103   on wire <-  rogue

"Mixed" remote device, untagged packets, as well as uses VLAN 101 and 102

  • VLAN 100 is untagged
  • VLANs 101-102 tagged
  • VLAN 103 is not enabled
VLAN 100 internal <-> port <-> (untagged) on wire <-> remote server
VLAN 101 internal <-> port <-> VLAN 101   on wire <-> remote server
VLAN 102 internal <-> port <-> VLAN 102   on wire <-> remote server
VLAN 103 internal  X  port <-  VLAN 103   on wire <-  rogue

While I've shown the "rogue" on a non-operational VLAN, there is nothing to prevent them from snooping the wire and seeing that VLAN 101 and VLAN 102 are "valid" VLANs and configuring their own interface for one or both of those VLANs. This is where something like 802.1X port authentication comes in, if you really think you need it. For many people, password/keyed authentication to their devices is the weak link; a compromise of a "trusted" device is much more likely than an untrusted device appearing on a trusted VLAN.

For switch-to-switch trunking, I set things up with everything on VLANs, no untagged packets. In general, the more explicit you are with things, the less chances for surprise.

Speaking of surprises, make sure you know how to factory-reset your switch before you think about changing the management VLAN from its default (which you probably should do). I've messed up more than once on that myself!

1 Like

To get VLANs, remember how they are implemented. Each packet is either an untagged packet, or a "tagged" packet with a particular VLAN number. They travel down the wire and are received by devices that understand VLANs and interpreted as if they were coming on a "different wire"

So, you can set a PVID for a particular port, and then when an untagged frame is received it will be "internally tagged" inside the switch.

If a tagged frame is received, the switch understands the tag and it becomes a part of that particular VLAN, so it can only go out on ports that belong to that VLAN...

From a security perspective you set up each port so that it only accepts the minimum set of VLANs that are needed to perform the tasks your network requires.

So for example, coming from your Tomato/modem you'll be receiving untagged frames at the switch, you want them to go into vlan 10, so you set up PVID 10 for that port, and this port ONLY belongs to vlan 10.

Trunking is only needed when you want unknown vlans to pass through your switch. In your situation I don't think it will ever be needed to have trunking set up.

1 Like

Ha, ha... I can definitely appreciate that.

Actually that's what got me confused thinking of them as cables, whereas I believe it might work better for me to first think about the logical layout and how the data should flow and then translate that to the physical topology. Listening to couple networking professionals vent their own frustrations, put things in perspective for me. Apparently terminology may vary between vendors and sometimes even between models by the same vendor (Cisco). So that was interesting.

I took a look at my docs last night. The Engenius should support VLANs in WDS Bridge mode, but I will still need to confirm.

So, if you have a trusted network, untrusted network and maybe a guest network, you require each user to connect to that subnet which is on its own VLAN SSID via OpenVPN for example? So each device needs to have a client installed. Don't you guests hate you? Just thinking out loud. :face_with_raised_eyebrow:

Yes, I think that's possible. Like I said, I think some of the terminology that some of the vendors use differs and somehow the Zyzel has the VLAN configuration laid out doesn't gel with me. That's probably just me. Probably just need some more hands-on time with this.

I think that was my first little epiphany. When I first started thinking about the layout, I went top to bottom (devices farthest away); but I think it much better the other way around when it comes to VLANs.

That makes sense. Once I get my network flow working, I will review each port and be as restrictive by ingress and egress as possible. Is that the right approach?

Thanks for this. It help clear up a few points.

Well, my switch comes with it and Daniel thinks 802.1x is at least supported one way to the LEDE devices I am going to use. So what the heck. Unless I've run completely out of patience by the end of this project. :laughing:

Not sure I follow?

So the uplink ports between each switch need to be marked as 'Trunk' in addition you need those same ports on each swith to be tagged for all VLANs that the uplink is supposed to carry?

Oh, yes! Very wise words indeed!

Thanks for your time helping me out with this. It's appreciated.

Cheers

No, at least in the Zyxel, their notion of "trunk" is that the switch will pass vlans set up by other switches that it doesn't know about. So for example you have Cisco switch somewhere with VLANs 99 100 101 coming out some port that then connects to your Zyxel port 1, and another Cisco switch with same vlans connected to port 2...

The zyxel is not configured to use 99 100 101 at all, so if you make port 1 and 2 a trunk, it will just happily accept any vlan tag there and pass it along back and forth between port 1 and 2...

In your scenario all the VLANs that are valid on your network should be ones that the Zyxel knows about and has explicitly enabled on each port that they can appear on. So nothing should be "trunk"

1 Like

Thanks man!

Ok, so trunks behave the same in a VLAN scenario as they do when connecting (trunk, uplink) two physical switches with no VLANs. I thinks some of the fog is lifting. :grinning:

Thanks Daniel!

Specific advice to get you going:

Keep one port "open" and accepting all VLANs and accepting untagged packets that go direct into the management vlan, while you configure the other ports

To see which VLANs your ports should accept, just look at your diagram. Each color is a VLAN. So what I see is:

Tomato: red, vlan 10 untagged with ingress filtering only untagged frames allowed

LEDE box with blue/green at bottom: only blue and green vlans, make them both tagged on that port. Make the LEDE box tag its frames.

OPNSense: needs to communicate to all the vlans so its port should belong to red, green, blue, purple and orange, make them all tagged

DMZ: you're connecting to a "dumb" switch here, easiest is to make this port belong to only orange vlan untagged

LEDE Box at the top: make its port belong to purple and blue vlans tagged

on each port in the zyxel you'll enable ingress filtering so that it can only accept VLANs that the port belongs to.

1 Like

Thanks for these specifics. I will be testing some of this out with the equipment that I have on hand this weekend.

No, guests log in to the guest WiFi with access to the outside world -- they have no need to access anything else.

If you have an unauthorized user or application on a "trusted" system, your firewalls, network separation, 802.1X authentication, ... are not going to help you. Personally, I'd spend significant time there before going on to "niceties" like 802.1X.

It depends on your switch's terminology. I have to look at the docs every time I adjust the configuration of my Cicso switches. I never remember clearly the difference between, as I recall, "access", "general", and "trunk" including which options each allow.

:+1::relaxed:

I've perhaps assumed that you have wireshark installed. If not, get it up and going! It's a lot easier to "see" the traffic than with tcpdump

Right 802.1x requires a device to prove it knows a secret. However if someone compromises the device... Then they can do malicious things because the device knows the secret. The value of 802.1x is it thwarts someone plugging into a port and doing whatever they want. If you have strong physical security on ports it's less relevant.

I know this is something like a zombie thread but I'm think about my network topology too and I'm Really curious, why you recommend to put a managed switch in front of the ipfire. Why would you do that, isn't the switch a security risk too, if it's directly connected to the modem? You can't control what the software of the switch does. So you rely on a device with closed source software, to really send all packets from the WAN to your firewall device (ipfire).

I'd be happy if you share your thoughts and explanations.

I had recommended bonding the interfaces for redundancy, you need a managed switch to handle the bonding. If your threat model is that you're concerned about the switch being compromised then sure put it entirely behind the firewall. It just depends on your concerns. For this matter, how do you trust your NICs? couldn't they receive a special packet from the NSA that activates the secret management core known to be in Intel CPUs? It seems plausible. If the switch is operating properly, packets stay on their VLAN, if you are concerned about compromise and have multiple VLANs then having the switch on your LAN exposes you to guests compromise as well... You have to assess probabilities, and it seems that bonding reduces the risk of cable or hardware port failure taking you offline. That happens pretty regularly. Remote compromises of switches from non management vlans I've never heard of, because switches are pretty dumb devices.

Ok I understand. I was only asking, no offense... Thank you for your explanation.

Not offended at all. It's a totally legit question, I just think that the risks from a properly configured switch are much less than many other real world risks.

1 Like

I'd personally add the caveat, "... properly configured managed switch from a reputable manufacturer" as I've seen some unexpected behavior from VLAN-enabled, [not very] "smart" switches, even from Netgear several years ago. With a reputable manufacturer and good control over VLANs and how the switch handles tagged and untagged packets that don't "match" the assigned VLANs for a port, the risks are reasonable, in comparison to the benefits gained.

1 Like