Difference between DSA and driver-level VLANs in the context of RPi4 (no switch)?

@richb-hanover-priv Thanks Rich.

@darksky Let's start by explaining a bit of how things work.

DSA only sets up the switch ports. E.g. interfaces called wan, lan1, lan2, etc.

To configure VLANs on a switch using DSA framework, you'd have to put all of the interfaces (wan, lan1, lan2, etc.) into one bridge (e.g. br0). Then, create a subinterface of the bridge interface (e.g. br0.5 which is based off of br0 and set to use VLAN ID 5) to send & receive frames with the specified VLAN. Check the Converting to DSA page I wrote for more details.

These interfaces (br0.5 in this example) are also called subinterfaces. This just means that they're based off of another interface. E.g. if the br0.5 is based off of br0, it's a subinterface of br0.

The name of your interface actually doesn't matter. You could name your interface whatever you want and still make it based off of, for example, br0 and set it to use VLAN 5.
Here's a screenshot where I call my interface name "whatever", it's still based off of br0 and set to use VLAN 5:


To do the same thing on the command line is this command: ip link add link br0 name whatever type vlan id 5

You can see VLAN and other information of the interfaces on the command line with this command: ip -d link (you need ip-full package on OpenWrt to use the "-d" option). E.g. the output of the "whatever" interface I created:

6: whatever@br0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether a0:b0:c0:d0:e0:f0 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535
    vlan protocol 802.1Q id 5 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 62780 gso_max_segs 65535

It's more convenient to name your subinterface e.g. br0.5 so you can easily figure out which interface it's based off of and what VLAN ID it uses by just looking at the name of the subinterface.

OpenWrt makes things more easier. If you just add an interface (it's incorrectly called "device" on LuCI) called br0.5 on one of the networks on the "Interfaces" page, OpenWrt will automatically create a subinterface of br0 and set the interface to use VLAN 5.

When you have a subinterface called br0.5 based off of br0 with VLAN 5, this is how frames are forwarded:
Any untagged frames going to br0.5 are forwarded out of br0 as tagged frames with VLAN 5.
Any tagged frames with VLAN 5 going to br0 will be forwarded to br0.5 as untagged frames.

VLAN filtering needs to be enabled on the bridge. The Linux kernel will then forward frames per the VLAN configuration.


Back to your case

On devices without a switch, the ethernet port(s) - mostly singular - is directly connected to the CPU. Therefore the interface of the ethernet port is usually called something like eth0, eth1, etc. You don't use DSA to initialise these ports in this case.

You only have one port so you don't need to put it under a bridge (if you don't want to include wireless interfaces on the VLANs. If you intend to, the wiki page I linked above should keep you covered).

Let's say you have 3 networks. On "Interfaces" page, for the first network, specify the interface as eth0.10.
For the second, specify eth0.20. For the third eth0.30.

And then you can set up the router to route between these networks.

I hope this was inclusive enough. Let me know if you have any questions.

2 Likes

Thank you very much for the detailed reply. So DSA for my is not needed, although it is working. I will try reconfiguring as you said as I have time and update the thread. Thank you.

I think I got it working. The difficult part was figuring out the LXC I'm running.

/etc/config/network
config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1a:184b:b879::/48'
	option packet_steering '1'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'eth0.1'
	option type '8021q'
	option ifname 'eth0'
	option vid '1'
	option ipv6 '0'

config device
	option name 'eth0.3'
	option type '8021q'
	option ifname 'eth0'
	option vid '3'
	option ipv6 '0'

config device
	option name 'eth0.4'
	option type '8021q'
	option ifname 'eth0'
	option vid '4'
	option ipv6 '0'

config device
	option name 'eth0.5'
	option type '8021q'
	option ifname 'eth0'
	option vid '5'
	option ipv6 '0'

config device
	option name 'br-lxc.4'
	option type 'bridge'
	option ipv6 '0'
	list ports 'eth0.4'

config device
	option name 'wg0'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lan'
	option device 'eth0.1'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'

config interface 'guest'
	option device 'eth0.3'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'

config interface 'lxc'
	option device 'br-lxc.4'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'

config interface 'iot'
	option device 'eth0.5'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'

config interface 'wg0'
	option proto 'wireguard'
...
lxc network section config
lxc.net.0.type = veth
lxc.net.0.link = br-lxc.4
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.250/24
lxc.net.0.ipv4.gateway = 10.0.4.1

Note that if I didn't create br-lxc.4 and if I tried using a macvlan device, when I reboot my dumb AP, the contain's interface would mysteriously but consistently go down. I would then need to restart the lxc which is not optimal. So that bridge was my solution. Bug maybe. The bridge is more robust.

Thanks again!

1 Like

I did a slight edit to correct misinformation about how VLAN filtering works in Linux and how DSA comes into play, please read it. Also, because of your situation with LXC, you should put the eth0 into bridge (e.g. br0) and configure VLAN filtering there. Here is an example according to your set up:


And then use br0.1, br0.3, br0.4 & br0.5 on the networks.

You can simply attach any wireless interfaces to any of the networks this way. For example, if you attach one to the network using br0.5, it will be in the bridge and in the specified VLAN.

That is similar to my original setup. What is wrong with my current one? Why pull Bridge VLAN filtering into the mix again?

I think your current setup will work, I just don't find it as a proper solution. Either do VLAN filtering for eth0 on the bridge interface or do it as subinterfaces of eth0.

I made the changes to /etc/config/network to use VLAN filtering on eth0/bridging setup. But I am unable to start the container.

When I try:

# lxc-start -n pihole -F
lxc-start: pihole: network.c: lxc_ovs_attach_bridge: 2781 Failed to attach "vethUaru4x" to openvswitch bridge "br0.4": lxc-start: pihole:
lxc-start: pihole: network.c: netdev_configure_server_veth: 715 Operation not permitted - Failed to attach "vethUaru4x" to bridge "br0.4"
lxc-start: pihole: network.c: lxc_create_network_priv: 3419 Operation not permitted - Failed to create network device
lxc-start: pihole: start.c: lxc_spawn: 1826 Failed to create the network
lxc-start: pihole: start.c: __lxc_start: 2053 Failed to spawn container "pihole"
lxc-start: pihole: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: pihole: tools/lxc_start.c: main: 313 Additional information can be obtained by setting the --logfile and --logpriority options
/etc/config/network (using br0+VLAN filtering)
config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1a:184b:b879::/48'
	option packet_steering '1'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'wg0'
	option ipv6 '0'

config device
	option name 'br0.1'
	option type '8021q'
	option ifname 'br0'
	option vid '1'
	option ipv6 '0'

config device
	option name 'br0.3'
	option type '8021q'
	option ifname 'br0'
	option vid '3'
	option ipv6 '0'

config device
	option name 'br0.4'
	option type '8021q'
	option ifname 'br0'
	option vid '4'
	option ipv6 '0'

config device
	option name 'br0.5'
	option type '8021q'
	option ifname 'br0'
	option vid '5'
	option ipv6 '0'

config device
	option type 'bridge'
	option name 'br0'
	list ports 'eth0'
	option ipv6 '0'

config bridge-vlan
	option device 'br0'
	option vlan '1'
	list ports 'eth0:t'
	option ipv6 '0'

config bridge-vlan
	option device 'br0'
	option vlan '3'
	list ports 'eth0:t'
	option ipv6 '0'

config bridge-vlan
	option device 'br0'
	option vlan '4'
	list ports 'eth0:t'
	option ipv6 '0'

config bridge-vlan
	option device 'br0'
	option vlan '5'
	list ports 'eth0:t'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lan'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'
	option device 'br0.1'

config interface 'guest'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'
	option device 'br0.3'

config interface 'lxc'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'
	option device 'br0.4'

config interface 'iot'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'
	option device 'br0.5'

config interface 'wg0'
	option proto 'wireguard'
...
LXC config
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = aarch64

# Container specific configuration
lxc.rootfs.path = dir:/srv/lxc/pihole/rootfs
lxc.uts.name = pihole

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br0.4
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.250/24
lxc.net.0.ipv4.gateway = 10.0.4.1

Either do VLAN filtering for eth0 on the bridge interface or do it as subinterfaces of eth0.

The other option you mentioned might be appealing, but what configuration in LXC would you recommend to do so?

Ok, now I see what you're trying to do. Change interface name of lxc to anything but br0 & eth0 (to avoid confusion), for example: lxc.net.0.link = lxc

Add lxc interface to the bridge and set up VLAN 4 tagged for this interface. Use br0.4 on the network configuration at the Interfaces page.

Edit: Actually, use lxc interface on the network at the Interfaces page.
Set up both eth0 & lxc interface VLAN 4 tagged.

This way, anything at VLAN 4 connecting to your Pi (over eth0) and your router itself can reach lxc.

Can you tell what do you use lxc for?
Do you need it to be in the same VLAN as something connecting to your Pi? Or do you just need to route to lxc?

I run pi-hole in the container. It needs to serve all of my zones except for iot (no WAN access allowed on that zone). So it's lan, guest, and lxc.

Perhaps I am making the setup more complex by adding a VLAN 4 and a dedicated interface for the lxc. I am happy to have it exist with my guest zone on the guest interface.

What if I simplify it, for interfaces on the Pi, remove LXC all together and some how get the container on the GUEST interface. Thoughts?

Next step is me bring-up the bridge and the lxc interface when I am having trouble doing. But first, what do you think about my simplification idea?

Do you have anything connecting to your Pi on VLAN 4? If this was made just for lxc, you can get rid of it.

Makes sense, here's how I would do it. Just let lxc create the interface and IP configuration. Get rid of this:

config interface 'lxc'
	option device 'br-lxc.4'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'

Then, create a new firewall zone and include the lxc interface e.g.:


Let's see what happens when you don't specify a gateway. Also change the address to 10.0.4.1/24. This way, your devices can reach pi-hole over 10.0.4.1.

Does this all make sense?

Backing up, are all these changes using the subinterfaces of eth0 setup? If so, here the state I am at currently (not using VLAN filtering and bridges).

/etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1a:184b:b879::/48'
	option packet_steering '1'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'eth0.1'
	option type '8021q'
	option ifname 'eth0'
	option vid '1'
	option ipv6 '0'

config device
	option name 'eth0.3'
	option type '8021q'
	option ifname 'eth0'
	option vid '3'
	option ipv6 '0'

config device
	option name 'eth0.4'
	option type '8021q'
	option ifname 'eth0'
	option vid '4'
	option ipv6 '0'

config device
	option name 'eth0.5'
	option type '8021q'
	option ifname 'eth0'
	option vid '5'
	option ipv6 '0'

config device
	option name 'br-lxc.4'
	option type 'bridge'
	option ipv6 '0'
	list ports 'eth0.4'

config device
	option name 'wg0'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lan'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'
	option device 'eth0.1'

config interface 'guest'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'
	option device 'eth0.3'

config interface 'lxc'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'
	option device 'br-lxc.4'

config interface 'iot'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'
	option device 'eth0.5'

config interface 'wg0'
	option proto 'wireguard'

No we're doing routing now. We just create an interface for lxc and route other subnets there. Get rid of:

config device
	option name 'br-lxc.4'
	option type 'bridge'
	option ipv6 '0'
	list ports 'eth0.4'
config interface 'lxc'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'
	option device 'br-lxc.4'

Lxc should create an interface which won't appear on Interfaces page. Then set up firewall like my reply above.

Sorry, we covered a few setups. I am asking if I should revert to the one not using VLAN filtering and br0. I think so but want to be clear. You asked me to get rid of several sections from the configuration based on subinterfaces. Does my question make sense?

If so, I removed those two sections and now have this:

inprogress network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1a:184b:b879::/48'
	option packet_steering '1'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'eth0.1'
	option type '8021q'
	option ifname 'eth0'
	option vid '1'
	option ipv6 '0'

config device
	option name 'eth0.3'
	option type '8021q'
	option ifname 'eth0'
	option vid '3'
	option ipv6 '0'

config device
	option name 'eth0.4'
	option type '8021q'
	option ifname 'eth0'
	option vid '4'
	option ipv6 '0'

config device
	option name 'eth0.5'
	option type '8021q'
	option ifname 'eth0'
	option vid '5'
	option ipv6 '0'

config device
	option name 'wg0'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lan'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'
	option device 'eth0.1'

config interface 'guest'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'
	option device 'eth0.3'

config interface 'iot'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'
	option device 'eth0.5'

config interface 'wg0'
	option proto 'wireguard'
...

I'm leaving the choice to you, whether do VLAN filtering on bridge or just create subinterfaces of eth0.

What I pointed to remove are related to lxc, we're going to take a different way at it.

You can get rid of this too if you don't use VLAN 4 for any devices connecting to the Pi.

config device
	option name 'eth0.4'
	option type '8021q'
	option ifname 'eth0'
	option vid '4'
	option ipv6 '0'

OK. I'd like to work off the subinterfaces configuration. I removed the 3 sections you called out. What do you suggest I use for the lxc configuration itself? Your suggestion was to omit the gateway therein and change the IP address assigned. Beyond that, still use the veth setup and link to what?

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = 
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.1/24

Put lxc-test there (as the name of the interface at lxc.net.0.link =) and start lxc. See if it runs fine, then check if lxc-test interface is created by running ip link command.

OK. Here is the current config:

/etc/config/network

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1a:184b:b879::/48'
	option packet_steering '1'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'eth0.1'
	option type '8021q'
	option ifname 'eth0'
	option vid '1'
	option ipv6 '0'

config device
	option name 'eth0.3'
	option type '8021q'
	option ifname 'eth0'
	option vid '3'
	option ipv6 '0'

config device
	option name 'eth0.5'
	option type '8021q'
	option ifname 'eth0'
	option vid '5'
	option ipv6 '0'

config device
	option name 'wg0'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lan'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'
	option device 'eth0.1'

config interface 'guest'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'
	option device 'eth0.3'

config interface 'iot'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'
	option device 'eth0.5'

config interface 'wg0'
	option proto 'wireguard'

Here is the lxc config:

lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = aarch64
lxc.rootfs.path = dir:/srv/lxc/pihole/rootfs
lxc.uts.name = pihole

lxc.net.0.type = veth
lxc.net.0.link = lxc-test
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.1/24

When I try to start it, it fails:

# lxc-start -n pihole -F
lxc-start: pihole: network.c: netdev_configure_server_veth: 708 No such file or directory - Failed to attach "vethU6Vom5" to bridge "lxc-test", bridge interface doesn't exist
lxc-start: pihole: network.c: lxc_create_network_priv: 3419 No such file or directory - Failed to create network device
lxc-start: pihole: start.c: lxc_spawn: 1826 Failed to create the network
lxc-start: pihole: start.c: __lxc_start: 2053 Failed to spawn container "pihole"
lxc-start: pihole: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: pihole: tools/lxc_start.c: main: 313 Additional information can be obtained by setting the --logfile and --logpriority options

I'm thinking the type of the network needs to be something else?

Can you rename lxc.net.0.link = lxc-test to lxc.net.0.name = lxc-test?

Done. Now it starts.

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether dc:a6:32:02:c1:c1 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc cake state UP qlen 1000
    link/ether 60:a4:b7:59:24:af brd ff:ff:ff:ff:ff:ff
4: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether dc:a6:32:02:c1:c2 brd ff:ff:ff:ff:ff:ff
131: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether dc:a6:32:02:c1:c1 brd ff:ff:ff:ff:ff:ff
132: eth0.3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether dc:a6:32:02:c1:c1 brd ff:ff:ff:ff:ff:ff
133: eth0.5@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether dc:a6:32:02:c1:c1 brd ff:ff:ff:ff:ff:ff
134: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN qlen 1000
    link/[65534] 
137: ifb4eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc cake state UNKNOWN qlen 32
    link/ether 72:27:93:1b:43:a7 brd ff:ff:ff:ff:ff:ff
142: veth1ZgkoO@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether fe:64:43:d4:bd:dc brd ff:ff:ff:ff:ff:ff

I cannot ping it. If I attach to it lxc-attach -n pihole I cannot ping outside of it.

Ok so let me quote this from linuxcontainers.org

veth: a virtual ethernet pair device is created with one side assigned to the container and the other side on the host. lxc.net.[i].veth.mode specifies the mode the veth parent will use on the host. The accepted modes are bridge and router. The mode defaults to bridge if not specified. In bridge mode the host side is attached to a bridge specified by the lxc.net.[i].link option. If the bridge link is not specified, then the veth pair device will be created but not attached to any bridge.

Getting rid of lxc.net.0.link made lxc switch to routing mode but it did not give the name we specified for some reason. Apparently, I can't even read something I quote.
Add lxc.net.0.veth.mode = router. Keep lxc.net.0.name = lxc-test as is

Anyway, set up firewall for lxc-test as I stated above, it should start working. Also check the IP configuration of lxc-test by ip a and check whether route to 10.0.4.0/24 is created by ip route