Difference between DSA and driver-level VLANs in the context of RPi4 (no switch)?

darksky · September 16, 2021, 10:56am

Yes, I too found that bit about lxc.net.0.veth.pair in the name page. So the name is consistent now and what I used in the firewall zone.

After starting the container, still no ability ping, but after I ran the command you suggested, I can ping it.

# ip addr add 10.0.4.1/24 dev lxc-test
# ping 10.0.4.1
PING 10.0.4.1 (10.0.4.1): 56 data bytes
64 bytes from 10.0.4.1: seq=0 ttl=64 time=0.133 ms
64 bytes from 10.0.4.1: seq=1 ttl=64 time=0.249 ms

And

# ip route
default via xxx.xxx.xxx.1 dev eth1  src xxx.xxx.xxx.34 
10.0.4.0/24 dev lxc-test scope link  src 10.0.4.1 
10.0.4.1 dev lxc-test scope link 
10.9.5.0/24 dev eth0.5 scope link  src 10.9.5.1 
10.9.7.0/24 dev eth0.3 scope link  src 10.9.7.1 
10.9.8.0/24 dev eth0.1 scope link  src 10.9.8.1 
10.200.200.0/24 dev wg0 scope link  src 10.200.200.200 
10.200.200.201 dev wg0 scope link 
10.200.200.202 dev wg0 scope link 
10.200.200.203 dev wg0 scope link 
xxx.xxx.xxx.0/22 dev eth1 scope link  src xxx.xxx.xxx.34

And:

# ip a
...
137: ifb4eth1: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc cake state UNKNOWN qlen 32
    link/ether 72:27:93:1b:43:a7 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7027:93ff:fe1b:43a7/64 scope link 
       valid_lft forever preferred_lft forever
152: lxc-test@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether fe:d8:6e:bc:44:f3 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.1/24 scope global lxc-test
       valid_lft forever preferred_lft forever
    inet6 fe80::fcd8:6eff:febc:44f3/64 scope link 
       valid_lft forever preferred_lft forever

arinc9 · September 16, 2021, 11:01am

Ok, sounds good. There must be an option to automatically specify the IP on the interface when lxc creates it. Moving on for now.

Can the container connect to the internet right now? Does everything work?

darksky · September 16, 2021, 11:02am

No, I cannot ssh into the container from the router. If I attach with lxc-attach I have no connectivity inside it either.

arinc9 · September 16, 2021, 11:03am

OK, the lxc.net.[i].ipv4.gateway option specifies gateway to use on the inside of the container, can you try putting 10.0.4.1 there?

P.S. I'm getting this feeling that just bridging to an interface which we can control with LuCI looks like a better idea in terms of automation and ease of control. Anywho, this was quite entertaining.

Just create a bridge e.g. lxcbr0.
Set up lxc to use lxcbr0 as the link. Set up IP 10.0.4.250/24 gateway 10.0.4.1.
Create network using lxcbr0 interface. Set up IP 10.0.4.1/24.
Create firewall zone for this network.

Voila?

The difference is traffic will be routed to lxcbr0 instead of lxc-test and we have control over LuCI since the network with lxcbr0 appears on Interfaces page & /etc/config/network.

darksky · September 16, 2021, 11:25am

I redefined the IP to the higher number and added the gateway:

lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = aarch64

# Container specific configuration
lxc.rootfs.path = dir:/srv/lxc/pihole/rootfs
lxc.uts.name = pihole

lxc.net.0.type = veth
lxc.net.0.veth.pair = lxc-test
lxc.net.0.veth.mode = router
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.250/24
lxc.net.0.ipv4.gateway = 10.0.4.1

It is now partially working.

There is no DNS resolution in the container
I cannot get DNS resolution working outside of the container from the lan zone
I can ssh to the container from the lan zone
I can connect to pihole's web interface from the lan zone

Here is the firewall forwarding page:

Interestingly, the 'lxc' firewall zone I created seems empty.
If I edit the 'lan' zone for example:

arinc9 · September 16, 2021, 11:28am

That's just the visuals of LuCI. This is because it only shows networks defined on this zone. Since we define an interface, it shows empty.

What do you think about this? You oughta remove lxc.net.0.veth.mode = router since we go back to bridging. And add lxc.net.0.link = lxcbr0

darksky · September 16, 2021, 11:32am

I see now that you edited your reply. I created lxcbr0 in luci and I bridged it to eth0 like this:

config device
  option type 'bridge'
  option name 'lxcbr0'
  list ports 'eth0'
  option ipv6 '0'

Is that correct?

My lxc config is now the following but it does not start:

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.250/24
lxc.net.0.ipv4.gateway = 10.0.4.1

Yet:

# lxc-start -n pihole -F
lxc-start: pihole: network.c: netdev_configure_server_veth: 708 No such file or directory - Failed to attach "veth0ZzBNB" to bridge "lxcbr0", bridge interface doesn't exist
lxc-start: pihole: network.c: lxc_create_network_priv: 3419 No such file or directory - Failed to create network device
lxc-start: pihole: start.c: lxc_spawn: 1826 Failed to create the network
lxc-start: pihole: start.c: __lxc_start: 2053 Failed to spawn container "pihole"
lxc-start: pihole: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: pihole: tools/lxc_start.c: main: 313 Additional information can be obtained by setting the --logfile and --logpriority options

I must also need to create an interface in LuCI for the bridge to use?

arinc9 · September 16, 2021, 11:35am

Don't add eth0. Just an empty bridge.

darksky:

My lxc config is now the following but it does not start:

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.250/24
lxc.net.0.ipv4.gateway = 10.0.4.1

This looks good.
Did you do this step below?

That's a network, not an interface. I suppose you mean what I just quoted above, so do that.

darksky · September 16, 2021, 11:40am

I removed the list ports so it now reads:

config device
  option type 'bridge'
  option name 'lxcbr0'
  option ipv6 '0'
  option bridge_empty '1'

I'm not sure what you mean when you say "create networking using lxcbr0.."

In LuCi, Network>Interfaces has a button "Add new interface..."
In LuCi, Network>Devices has a button "Add device configuration..."

EDIT: Ah! I am calling that an interface due to the naming in the button!

arinc9 · September 16, 2021, 11:43am

I know that LuCI represents confusing networking terms. I'm actually in the middle of writing a mail to correct them. For now, think of "Interfaces" as "Networks", and "Devices" as "Interfaces".

So,

That's "network", not "networking".

darksky · September 16, 2021, 11:56am

That is confusing. I think I have it setup as you recommend. The container runs, I can ssh into it, but DNS resolution is not working.

/etc/config/network

config interface 'loopback'
  option device 'lo'
  option proto 'static'
  option ipaddr '127.0.0.1'
  option netmask '255.0.0.0'

config globals 'globals'
  option ula_prefix 'fd1a:184b:b879::/48'
  option packet_steering '1'

config device
  option name 'eth0'
  option ipv6 '0'

config device
  option name 'eth1'
  option ipv6 '0'

config device
  option type 'bridge'
  option name 'lxcbr0'
  option ipv6 '0'
  option bridge_empty '1'

config device
  option name 'wg0'
  option ipv6 '0'

config device
  option name 'eth0.1'
  option type '8021q'
  option ifname 'eth0'
  option vid '1'
  option ipv6 '0'

config device
  option name 'eth0.3'
  option type '8021q'
  option ifname 'eth0'
  option vid '3'
  option ipv6 '0'

config device
  option name 'eth0.5'
  option type '8021q'
  option ifname 'eth0'
  option vid '5'
  option ipv6 '0'

config interface 'wan'
  option device 'eth1'
  option proto 'dhcp'
  option peerdns '0'
  option delegate '0'
  list dns '1.1.1.1'
  list dns '1.0.0.1'

config interface 'lxc'
  option proto 'static'
  option device 'lxcbr0'
  option ipaddr '10.0.4.1'
  option netmask '255.255.255.0'

config interface 'lan'
  option proto 'static'
  option ipaddr '10.9.8.1'
  option netmask '255.255.255.0'
  option device 'eth0.1'

config interface 'guest'
  option proto 'static'
  option ipaddr '10.9.7.1'
  option netmask '255.255.255.0'
  option device 'eth0.3'

config interface 'iot'
  option proto 'static'
  option ipaddr '10.9.5.1'
  option netmask '255.255.255.0'
  option device 'eth0.5'

config interface 'wg0'
  option proto 'wireguard'
...

/etc/config/firewall


config defaults
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option synflood_protect '1'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fc00::/6'
	option dest_ip 'fc00::/6'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule 'wg'
	option name 'Allow-WireGuard'
	option proto 'udp'
	option target 'ACCEPT'
	option src 'wan'
	option dest_port '4500'

config include
	option path '/etc/firewall.user'

config zone 'lan'
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lan'

config zone
	option name 'guest'
	option output 'ACCEPT'
	option input 'REJECT'
	option forward 'REJECT'
	list network 'guest'
	list network 'wg0'

config zone
	option name 'lxc'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lxc'

config zone 'wan'
	option name 'wan'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	option input 'DROP'
	option forward 'DROP'
	list network 'wan'

config rule
	option src 'guest'
	option target 'ACCEPT'
	option name 'guest dhcp and dns'
	list proto 'tcp'
	list proto 'udp'
	option dest_port '53 67 68'

config forwarding 'lan_wan'
	option src 'lan'
	option dest 'wan'

config forwarding
	option src 'guest'
	option dest 'wan'

config rule
	option src 'iot'
	option target 'ACCEPT'
	list proto 'tcp'
	list proto 'udp'
	option dest_port '53 67 68'
	option name 'iot dhcp and dns'

config rule
	list proto 'udp'
	option src 'lxc'
	option dest_port '53'
	option target 'ACCEPT'
	option name 'pi-hole-dns lxc to input'

config rule
	list proto 'udp'
	option src 'guest'
	option dest 'lxc'
	option dest_port '53'
	option target 'ACCEPT'
	option name 'pi-hole-dns guest to lxc'

config zone
	option name 'iot'
	option output 'ACCEPT'
	option forward 'REJECT'
	option input 'REJECT'
	list network 'iot'

config forwarding
	option src 'lan'
	option dest 'guest'

config forwarding
	option src 'lan'
	option dest 'iot'

config forwarding
	option src 'lxc'
	option dest 'guest'

config forwarding
	option src 'guest'
	option dest 'lxc'

config forwarding
	option src 'lan'
	option dest 'lxc'

I have to be missing something simple.

arinc9 · September 16, 2021, 12:07pm

The network configuration looks good to me. It's kind of hard to read the firewall configuration though. That's about all I can do. And make sure /etc/resolv.conf inside the container has a correct nameserver defined.

darksky · September 16, 2021, 12:12pm

Within the container /etc/resolv.conf is good:

nameserver 1.1.1.1
nameserver 1.0.0.1

Here is a screenshot of the firewall forwarding page. Seems right to me.

No idea why DNS resolution is failing inside the container.

arinc9 · September 16, 2021, 12:15pm

If you can ping 1.1.1.1 from inside the container, there must be something wrong with the container itself or firewall rules allowing UDP/TCP port 53 for the lxc zone.

darksky · September 16, 2021, 12:16pm

I cannot ping external addresses from within the container, just the router and other devices behind it. Very confusing.

From container:

% ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
From 10.0.4.1 icmp_seq=1 Destination Port Unreachable

% ping 10.9.8.1
PING 10.9.8.1 (10.9.8.1) 56(84) bytes of data.
64 bytes from 10.9.8.1: icmp_seq=1 ttl=64 time=0.121 ms

arinc9 · September 16, 2021, 12:16pm

AH! You're also supposed to allow forwarding to wan zone (or wireguard if you use it for internet access) for lxc zone. Or did you only allow port 53 to be forwarded?

darksky · September 16, 2021, 12:40pm

I think I've been focused on this for too long... losing my common sense and objectivity. You are of course 100% correct.

I have a few rules to allow the lxc to get to the router and to the guest zone and I think it's function:

Deep and sincere THANK YOU for all the time you spent with me on this. Once my head clears, I think I might write-up this for others.

arinc9 · September 16, 2021, 12:44pm

You're very welcome. It was entertaining, to say the least. Good luck with your future endeavors!

darksky · September 17, 2021, 1:17am

Acknowledgement to @arinc9. For others wanting this setup, see below.

RPi4 primary router/firewall using VLANs (my setup has WiFi provided by a dumb AP and uses these VLANs to maintain isolation between three networks but that is outside the scope of this) via subinterfaces
LXC running pi-hole in my case but can be anything obviously (great post on LXC setup)
Dumb AP (VLAN-aware) connected for WiFi (outside of scope of this but mentioned for clarity

/etc/config/network


config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd1a:184b:b879::/48'
	option packet_steering '1'

config device
	option name 'eth0'
	option ipv6 '0'

config device
	option name 'eth1'
	option ipv6 '0'

config device
	option name 'wg0'
	option ipv6 '0'

config device
	option type 'bridge'
	option name 'lxcbr0'
	option ipv6 '0'
	option bridge_empty '1'

config device
	option name 'eth0.1'
	option type '8021q'
	option ifname 'eth0'
	option vid '1'
	option ipv6 '0'

config device
	option name 'eth0.3'
	option type '8021q'
	option ifname 'eth0'
	option vid '3'
	option ipv6 '0'

config device
	option name 'eth0.5'
	option type '8021q'
	option ifname 'eth0'
	option vid '5'
	option ipv6 '0'

config interface 'wan'
	option device 'eth1'
	option proto 'dhcp'
	option peerdns '0'
	option delegate '0'
	list dns '1.1.1.1'
	list dns '1.0.0.1'

config interface 'lxc'
	option device 'lxcbr0'
	option proto 'static'
	option ipaddr '10.0.4.1'
	option netmask '255.255.255.0'

config interface 'lan'
	option device 'eth0.1'
	option proto 'static'
	option ipaddr '10.9.8.1'
	option netmask '255.255.255.0'

config interface 'guest'
	option device 'eth0.3'
	option proto 'static'
	option ipaddr '10.9.7.1'
	option netmask '255.255.255.0'

config interface 'iot'
	option device 'eth0.5'
	option proto 'static'
	option ipaddr '10.9.5.1'
	option netmask '255.255.255.0'

config interface 'wg0'
	option proto 'wireguard'
...
# wireguard specifics omitted for privacy

/etc/config/firewall

# standard firewall rules omitted
config rule 'wg'
	option name 'Allow-WireGuard'
	option proto 'udp'
	option target 'ACCEPT'
	option src 'wan'
	option dest_port '4500'

config include
	option path '/etc/firewall.user'

config zone 'lan'
	option name 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'lan'

config zone
	option name 'guest'
	option output 'ACCEPT'
	option forward 'REJECT'
	list network 'guest'
	list network 'wg0'
	option input 'REJECT'

config zone
	option name 'lxc'
	option output 'ACCEPT'
	list network 'lxc'
	option forward 'ACCEPT'
	option input 'REJECT'

config zone
	option name 'iot'
	option output 'ACCEPT'
	option forward 'REJECT'
	option input 'REJECT'
	list network 'iot'

config zone 'wan'
	option name 'wan'
	option output 'ACCEPT'
	option masq '1'
	option mtu_fix '1'
	option input 'DROP'
	option forward 'DROP'
	list network 'wan'

config rule
	option src 'guest'
	option target 'ACCEPT'
	option name 'guest dhcp and dns'
	list proto 'tcp'
	list proto 'udp'
	option dest_port '53 67 68'

config forwarding 'lan_wan'
	option src 'lan'
	option dest 'wan'

config forwarding
	option src 'guest'
	option dest 'wan'

config rule
	option src 'iot'
	option target 'ACCEPT'
	list proto 'tcp'
	list proto 'udp'
	option dest_port '53 67 68'
	option name 'iot dhcp and dns'

config rule
	list proto 'udp'
	option src 'lxc'
	option dest_port '53'
	option target 'ACCEPT'
	option name 'pi-hole-dns lxc to input'

config rule
	list proto 'udp'
	option src 'guest'
	option dest 'lxc'
	option dest_port '53'
	option target 'ACCEPT'
	option name 'pi-hole-dns guest to lxc'

config forwarding
	option src 'lan'
	option dest 'guest'

config forwarding
	option src 'lan'
	option dest 'iot'

config forwarding
	option src 'lan'
	option dest 'lxc'

config forwarding
	option src 'lxc'
	option dest 'wan'

/srv/lxc/pihole/config

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist archlinux --release current
# Template script checksum (SHA-1): 8dff53d9a72ba3c071585c5762e2d14c57943cfa
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = aarch64

# Container specific configuration
lxc.rootfs.path = dir:/srv/lxc/pihole/rootfs
lxc.uts.name = pihole

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.ipv4.address = 10.0.4.250/24
lxc.net.0.ipv4.gateway = 10.0.4.1

Networks a.k.a. Interfaces and Interfaces a.k.a. Devices:

Firewall zones and manually added traffic rules

system · September 27, 2021, 1:18am

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.