VLAN on different bridge can't get internet access

MathijsNL · December 23, 2022, 12:31am

The problem I am facing is that I have a VLAN on port 1, 2, 4 and another VLAN on port 3 of the router.
Port 4 is connected to a modem with a 1GBPS connection.

Firmware version and router:
OpenWRT version: openwrt-22.03
Router: Netgear Nighthawk X4S R7800

Here is a schematic overview:

Vlan 1 is getting their IP addresses from the DHCP from the modem (192.168.1.1/24). Vlan 3 is getting their IP addresses on another subnet from the R7800 router (192.168.3.1/24).

The router has internet, VLAN 1 has internet, VLAN 3 is getting DHCP addresses in the correct range. But no internet for Vlan3. WAN port is not being used.

This is the setup for the VLANs (only 1 image allowed ):

VLAN ID	CPU (eth0)	CPU (eth1)	LAN 1	LAN 2	LAN 3	LAN 4	WAN
1	t	t	u	u	off	u	off
2	t	t	off	off	off	off	u
3	t	t	off	off	u	off	off

I am unsure how to debug what is missing right now. Maybe I have to do something with VLAN filtering in the bridges, but I believe that is if you just toss all interfaces in 1 bridge.

So it is probably some routing / firewall issue. I tried to add some similar rules that were already in place for lan, enabling masquerading but so far no luck.

In my previous setup using DD-WRT this IPTABLES entry did the trick:

iptables -t nat -I POSTROUTING -o `get_wanface` -j SNAT --to `nvram get wan_ipaddr`

If someone could point me in the right direction that would be great.

psherman · December 23, 2022, 12:42am

I think the solution will be fairly simple, but it is best for us to see the config files:

Please copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

cat /etc/config/network
cat /etc/config/firewall

MathijsNL · December 23, 2022, 1:30am

root@OpenWrt:~# cat /etc/config/network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdbe:127f:3be9::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1.1'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option gateway '192.168.1.254'

config interface 'wan'
        option device 'eth0.2'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth0.2'
        option proto 'dhcpv6'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option vid '1'
        option ports '0t 6t 4 3 2 1'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option vid '2'
        option ports '0t 6t 5'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option vid '3'
        option ports '0t 6t 2'

config device
        option type 'bridge'
        option name 'br3'
        option ipv6 '0'
        list ports 'eth1.3'
        option mtu '1500'

config bridge-vlan
        option device 'br3'
        option vlan '3'
        list ports 'eth1.3:t'

config interface 'switchmat'
        option proto 'static'
        option ipaddr '192.168.3.1'
        option netmask '255.255.255.0'
        option broadcast '192.168.3.255'
        option gateway '192.168.1.1'
        list dns '8.8.8.8'
        list dns '8.8.4.4'
        option device 'eth1.3'

config device
        option name 'br3.3'
        option type '8021q'

And



config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'

config zone
        option name 'wan'
        list network 'wan'
        list network 'wan6'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config zone
        option name 'switchmat'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'switchmat'
        option masq '1'

config rule
        option name 'Switchmat DNS'
        option src 'switchmat'
        option target 'ACCEPT'
        list proto 'all'

config rule
        option name 'Switchmat dhcp'
        option src 'switchmat'
        option dest_port '67'
        option target 'ACCEPT'

config forwarding
        option src 'switchmat'
        option dest 'wan'

config forwarding
        option src 'switchmat'
        option dest 'lan'

From mobile, forgive me if formatting went wrong.

psherman · December 23, 2022, 2:39am

Formatting is perfect. Thanks!

Let's start with the LAN configuration...
The gateway (192.168.1.254) should correspond to the modem's LAN address. Please confirm.

MathijsNL:

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option gateway '192.168.1.254'

You do not have DNS specified for the lan network interface (although it is possible you have this defined in the DHCP file -- I didn't ask for that, but probably should have). I'd recommend either putting DNS servers in the lan interface definition or in the dnsmasq resolvers in the DHCP config file.

Now, on to the VLAN...

I suspect that 0 and 6 are your CPU. If that's correct, you don't actually have any physical ports connected with VLAN 3. Logical port 5 (in your VLAN 2 definition)is likely the WAN port, and logical ports 1-4 (from the VLAN 1 config) are probably your 4 physical LAN ports.

You need to remove a port from VLAN 1 and put that port into VLAN 3... I'll recommend something below, but I cannot guarantee the logical <-> physical port mapping.

You only need either 0t or 6t in each of the VLANs, not both. These would correspond to eth0 and eth1 (I'm not sure how they map, but I would guess 0t -> eth0 and 6t ->eth1, but that could be reversed).

Will you be using VLAN3 with wired + wireless, or just wired? If wired only, delete the section below.

This stanza below doesn't belong (you're mixing DSA and swconfig syntax)... remove it.

And remove this:

Next, we'll deal with the switchmat network... the DNS entries here don't do anything. Delete them. If you want these DNS servers advertised via DHCP for hosts that use this network, it is done by specifying option 6 in the DHCP options for this network.

Also remove the gateway from this stanza as well.

MathijsNL:

config interface 'switchmat'
        option proto 'static'
        option ipaddr '192.168.3.1'
        option netmask '255.255.255.0'
        option broadcast '192.168.3.255'
        option gateway '192.168.1.1'
        list dns '8.8.8.8'
        list dns '8.8.4.4'
        option device 'eth1.3'

As for the device you used here, if you are wired only, this is fine. If you're wireless, you need to use 'br3' instead of 'eth1.3'

Below is how I would recommend setting up the switch for the VLANs. As stated above, there is a risk that the logical ports don't map to the ports in use here. (I am making a guess that logical port 4 = physical port 4, and that eth0 = logical port 0, and eth1 = logical port 6).

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option vid '1'
        option ports '6t 3 2 1'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option vid '2'
        option ports '0t 5'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option vid '3'
        option ports '0t 4'

Let's take care of all of this, then we'll do the firewall. When you're done with the network config file, please repost the latest copy of it for review.

MathijsNL · December 23, 2022, 4:55pm

Yes. That is correct.

Oops, I fiddled with it yesterday and that's why it showed up wrong in the config. My bad. Fixed it + updated it in the previous post. The mapping is 0,6 CPU, 5 WAN, 1234 for Ports but in reverse order.

For now wired, but once everything works there should be a wireless network added as well. So I will keep it with a bridge I guess.

Interesting, I don't know why this got added. I am sure I didn't add it manually. This was probably caused by the gui setting for Bridge Vlan filtering. I removed it.

Did you remap the logical ports to spread the CPU load here? It sure looks better this way.

Everything is changed, I will grab the latest configs after reboot and post them in a new reply.

MathijsNL · December 23, 2022, 5:10pm

Updated config files, added dhcp config as well. I added the dhcp option as well now to the switchmat interface.

root@OpenWrt:/etc/config# cat dhcp

config dnsmasq
        option domainneeded '1'
        option boguspriv '1'
        option filterwin2k '0'
        option localise_queries '1'
        option rebind_protection '1'
        option rebind_localhost '1'
        option local '/lan/'
        option domain 'lan'
        option expandhosts '1'
        option nonegcache '0'
        option authoritative '1'
        option readethers '1'
        option leasefile '/tmp/dhcp.leases'
        option resolvfile '/tmp/resolv.conf.d/resolv.conf.auto'
        option nonwildcard '1'
        option localservice '1'
        option ednspacket_max '1232'

config dhcp 'lan'
        option interface 'lan'
        option start '100'
        option limit '150'
        option leasetime '12h'
        option dhcpv4 'server'
        option dhcpv6 'server'
        option ra 'server'
        list ra_flags 'managed-config'
        list ra_flags 'other-config'
        option ignore '1'

config dhcp 'wan'
        option interface 'wan'
        option ignore '1'

config odhcpd 'odhcpd'
        option maindhcp '0'
        option leasefile '/tmp/hosts/odhcpd'
        option leasetrigger '/usr/sbin/odhcpd-update'
        option loglevel '4'

config dhcp 'switchmat'
        option interface 'switchmat'
        option start '100'
        option limit '150'
        option leasetime '12h'
        list dhcp_option '6,8.8.8.8,8.8.4.4'

root@OpenWrt:/etc/config# cat network

config interface 'loopback'
        option device 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fdbe:127f:3be9::/48'

config device
        option name 'br-lan'
        option type 'bridge'
        list ports 'eth1.1'

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option gateway '192.168.1.254'
        list dns '8.8.8.8'
        list dns '8.8.4.4'

config interface 'wan'
        option device 'eth0.2'
        option proto 'dhcp'

config interface 'wan6'
        option device 'eth0.2'
        option proto 'dhcpv6'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option vid '1'
        option ports '6t 4 3 1'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option vid '2'
        option ports '0t 5'

config switch_vlan
        option device 'switch0'
        option vlan '3'
        option vid '3'
        option ports '0t 2'

config device
        option type 'bridge'
        option name 'br3'
        option ipv6 '0'
        option mtu '1500'
        list ports 'eth0.3'

config interface 'switchmat'
        option proto 'static'
        option ipaddr '192.168.3.1'
        option netmask '255.255.255.0'
        option broadcast '192.168.3.255'
        option device 'br3'

psherman · December 23, 2022, 5:19pm

Ok... these files look good.

Next up is the firewall...

enable masquerading on the lan firewall zone:

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'
        option masq '1'

Turn off masquerading on the switchmat zone.

config zone
        option name 'switchmat'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        list network 'switchmat'

If you want to allow devices on the switchmat network to access the OpenWrt router itself, the reset of the rules for the zone are correct. However, if you don't want that (i.e. switchmat cannot administer the router), set input = reject.

remove this -- it doesn't do anything useful.

this is only necessary if you have switchmat input = reject. If input = accept, this rule becomes unnecessary.

And you can remove this rule because it is unused (no wan connection)

This should make it work, but doesn't isolate the switchmat from the upstream network... do you want to prohibit those connections?

MathijsNL · December 23, 2022, 6:51pm

Tested the connectivity. switchmat is able to connect to the router only, nothing else.

I will configure it this way and see what it does, but the end goal (which I should have mentioned in my first post) is that I want switchmat isolated from everything else for security reasons.

psherman · December 23, 2022, 6:54pm

this would likely be the case if the switchmat firewall zone does not have forwarding allowed to the lan zone. Did you add that?

EDIT: I realize I forgot to specify that... here:

config forwarding
        option src 'switchmat'
        option dest 'lan'

Ok... let's get everything else working first, then we'll isolate.

MathijsNL · December 23, 2022, 7:11pm

Sorry my phrasing was not clear. I tested the base configuration without any firewall adjustments, and they seemed to be working as expected. It is from there that I started adding the firewall rules. Also the forwarding thing was already in there.

So yes, everything is working as expected right now.

psherman · December 23, 2022, 7:12pm

Great! Is everything done now, or do you still need add the blocking rule?

MathijsNL · December 23, 2022, 7:19pm

I would still like to isolate the switchmat interface from the rest

psherman · December 23, 2022, 7:25pm

Add This:

config rule
        option name 'BlockLANfromSwitchmat'
        list proto 'all'
        option src 'switchmat'
        option dest 'lan'
        list dest_ip '192.168.1.0/24'
        option target 'DROP'

MathijsNL · December 23, 2022, 7:39pm

Brilliant! Everything works as I wanted it. I did switch src and dest so nothing from lan can reach switchmat.

The rest of the setup should be doable to work out from here. Thanks a lot for your help, I learned a lot from it and it gave me a working setup.

psherman · December 23, 2022, 7:42pm

Actually, this isn't necessary because the upstream network doesn't probably doesn't have a static route for the switchmat -- it doesn't know how to reach switchmat in general, and then the firewall on OpenWrt both has masquerading on the lan zone and further doesn't allow forwarding from lan > switchmat.

If you'd like a final review of your config files, feel free to post them here. But sounds like everything is working, so that's great!

MathijsNL · December 23, 2022, 7:50pm

I agree. I tested it and the devices can't reach eachother. But now I can access the upstream modem at 192.168.1.254, and that doesn't work the other way around.

I appreciate the offer for the review of the config files, but as everything is working and tested I think it is not needed at this point.

psherman · December 23, 2022, 7:59pm

The level of and direction of isolation between your networks comes down to the goals.

By default, the main LAN will not be able to reach switchmat network, no additional rules are required.

Devices on the switchmat network will be able to reach the main router at 192.168.1.254 (and anything on the 192.168.1.0/24 network) by default. If you used the rule as I provided earlier, that would prevent the switchmat devices from connecting to that network, but the OpenWrt router itself will still be able to reach both networks, of course.

MathijsNL · December 23, 2022, 9:30pm

Yeah switchmat is where my switch is attached, and I just want everything else isolated from that switch. If it is isolated by default, would this rule won't be needed right, and it doesn't have to be reverted?

psherman:

Add This:

config rule
        option name 'BlockLANfromSwitchmat'
        list proto 'all'
        option src 'switchmat'
        option dest 'lan'
        list dest_ip '192.168.1.0/24'
        option target 'DROP'

And it looks like you are right about the main router, I can access it without problems.

psherman · December 23, 2022, 9:34pm

The isolation should be thought of in the context of initiating connections (if a connection can be initiated by one side, the other side will be allowed to respond).

With that said, the isolation can be uni-directional or bi-directional. The default configuration of OpenWrt (and what we created in your case) is uni-directional isolation in that the upstream network (192.168.1.0/24) cannot reach the downstream network (192.168.3.0/24). The rule I suggested blocks the other direction. If you don't need or want to block 192.168.3.0/24 devices from accessing the 192.168.1.0/24, you can simply delete/omit the rule.

MathijsNL · December 23, 2022, 9:47pm

Just one last question about my specific setup. I have read in some places not to use the WAN port for performance reasons. I am missing out on some +- 200 mbit when on the switchmat vlan. Would it make a difference if the openwrt router and main router are connected LAN <> WAN instead of LAN <> LAN?