I struggle with the firewall settings. I'm used to write my rules. There's probably something wrong in the settings which I can't detect / understand.
Maybe I have missed it, but what hardware do you use for the managed switches running Openwrt? (I am asking because afaik there is still no support for routing on the ASIC and everything needs to go through the really weak CPU. On my dlink switches with Openwrt I only get 50 Mbit when routing. Switching on Vlan is fine but routing should be done on a router or on a device with a strong CPU)
And I want to ask you to get rid of using route. Please use ip route and the ip util in general. https://openwrt.org/docs/guide-developer/networking/network.interfaces contains a good overview for the "modern" tools.
It's a Zyxel GS1900-24
As this is a rtl838x I would argue to rethink your setup if you want to have more then 50 Mbit/s.
I have Openwrt on 2 switches too but only as an access switch for various vlans. Like I said, the driver still has no support to use the layer 3 features of the chip so every packet needs to go through the weak CPU and this just does not scale. Besides that is having Openwrt on these switches pretty nice.
Just to be (more-) specific, rtl838x doesn't even have that many L3 features in hardware, those only started to appear in rtl839x and rtl93xx (and yes, there driver support to actually make use of them is rather at the beginning).
You're right. I just tested the throughput and it's very bad, approx 30mbits.
Is there a way to define VLAN and plug devices directly to the switch, without defining an interface on the managed OpenWRT switch ?
Yes... of course. If you define all of the networks on the actual routing devices (ideally R3 based on your diagram), the rest can be handled by switching. You won't need to define any static routes if all of the networks are configured on the single router. You'll use unmanaged interfaces to enable the switching on the switch (and potentially other devices) of the VLANs that are just 'passing through' (only one network needs to be used on each device -- the management network, so that you can talk to the infrastructure devices for configuration).
If you're routing in the managed switch you should call it a router. Although, as noted it is not a very good one due to its weak CPU.
For access both forward and backward between the various LANs, symmetric routing should be used. Turn off the masquerade in all routers except the one that actually leads to the Internet (R3). Then each router needs routes to all the LANs on site, with the gateway being the next router in line toward that LAN. Finally the firewall needs to allow forwarding in both directions between lan and wan.
I changed the network configuration a bit. Here's what it looks like now :
Since the Zyxel GS1900-24 is just a managed switch, I didn't define any real interface on it, and try to pass the traffic from WAN to LAN only using VLANs.
I can't reach R5 from R3 or the managed switch.
On the switch, when I try to reach R5, the traffic goes to R3, and doesn't go through port #10 :
traceroute to 192.168.240.2 (192.168.240.2), 20 hops max, 46 byte packets
1 *
2 *
3 *
4 192.168.114.63 104.405 ms !H
ip route
default via 192.168.114.1 dev switch.1 proto static
192.168.114.0/24 dev switch.1 proto kernel scope link src 192.168.114.63
192.168.240.0/24 dev switch.240 proto static scope link offload
type or paste code here
I spent several hours trying to get it to work, but it's not OK.
Here's the /etc/config/network file contents :
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix '(info removed)'
config device 'switch'
option name 'switch'
option type 'bridge'
option macaddr '(info removed)'
option ipv6 '0'
option bridge_empty '1'
list ports 'eth0'
list ports 'lan1'
list ports 'lan2'
list ports 'lan3'
list ports 'lan4'
list ports 'lan5'
list ports 'lan6'
list ports 'lan7'
list ports 'lan8'
list ports 'lan9'
list ports 'lan10'
list ports 'lan11'
list ports 'lan12'
list ports 'lan13'
list ports 'lan14'
list ports 'lan15'
list ports 'lan16'
list ports 'lan17'
list ports 'lan18'
list ports 'lan19'
list ports 'lan20'
list ports 'lan21'
list ports 'lan22'
list ports 'lan23'
list ports 'lan24'
list ports 'lan25'
list ports 'lan26'
config bridge-vlan 'lan_vlan'
option device 'switch'
option vlan '1'
list ports 'lan1'
list ports 'lan2'
list ports 'lan3'
list ports 'lan4'
list ports 'lan5'
list ports 'lan6'
list ports 'lan7'
list ports 'lan8'
config device
option name 'switch.1'
option macaddr '(info removed)'
option ipv6 '0'
config interface 'lan'
option device 'switch.1'
option proto 'static'
option ip6assign '60'
option gateway '192.168.114.1'
list dns '192.168.114.1'
option netmask '255.255.255.0'
option ipaddr '192.168.114.63'
option delegate '0'
config bridge-vlan
option device 'switch'
option vlan '245'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan9'
config bridge-vlan
option device 'switch'
option vlan '114'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t'
config bridge-vlan
option device 'switch'
option vlan '243'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t'
config bridge-vlan
option device 'switch'
option vlan '242'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t'
config bridge-vlan
option device 'switch'
option vlan '241'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t'
config bridge-vlan
option device 'switch'
option vlan '244'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t'
config bridge-vlan
option device 'switch'
option vlan '249'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t'
config device
option name 'switch.114'
option type '8021q'
option ifname 'switch'
option vid '114'
option ipv6 '0'
config device
option name 'switch.241'
option type '8021q'
option ifname 'switch'
option vid '241'
option ipv6 '0'
config device
option name 'switch.242'
option type '8021q'
option ifname 'switch'
option vid '242'
option ipv6 '0'
config device
option name 'switch.243'
option type '8021q'
option ifname 'switch'
option vid '243'
option ipv6 '0'
config device
option name 'switch.244'
option type '8021q'
option ifname 'switch'
option vid '244'
option ipv6 '0'
config device
option name 'switch.245'
option type '8021q'
option ifname 'switch'
option vid '245'
option ipv6 '0'
config device
option name 'switch.249'
option type '8021q'
option ifname 'switch'
option vid '249'
option ipv6 '0'
config bridge-vlan
option device 'switch'
option vlan '240'
list ports 'eth0:t'
list ports 'lan1:t'
list ports 'lan10:t*'
config interface 'ITF240'
option proto 'none'
option device 'switch.240'
config route
option interface 'ITF240'
option target '192.168.240.0/24'
config device
option name 'switch.240'
option type '8021q'
option ifname 'switch'
option vid '240'
What's wrong ? I tried to create an unmanaged interface on the switch, but that doesn't seem to work.
I think the problem is in the router I try to access: R5.
I installed tcpdump on it, to see what happens on the WAN interface of R5 while I try to ping it from R3.
root@R5:~# tcpdump -i wan
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wan, link-type EN10MB (Ethernet), capture size 262144 bytes
13:15:42.459999 IP 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6636, seq 1596, length 64
13:15:43.460476 IP 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6636, seq 1597, length 64
13:15:44.460945 IP 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6636, seq 1598, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel
root@R5:~# tcpdump -i wan.240
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wan.240, link-type EN10MB (Ethernet), capture size 262144 bytes
13:15:48.462969 IP 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6636, seq 1602, length 64
13:15:49.463461 IP 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6636, seq 1603, length 64
13:15:50.463960 IP 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6636, seq 1604, length 64
Here are the conf files of R5 :
/etc/config/network :
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option packet_steering '1'
option ula_prefix '(info removed)'
config device
option name 'br-lan'
option type 'bridge'
list ports 'br-lan.240'
list ports 'eth0'
list ports 'eth1'
list ports 'eth2'
list ports 'lan1'
list ports 'lan2'
list ports 'lan3'
list ports 'lan4'
list ports 'wan'
list ports 'wan.240'
config device
option name 'wan'
option macaddr '(info removed)'
option ipv6 '0'
config interface 'wan'
option proto 'static'
option ipaddr '192.168.240.2'
option netmask '255.255.255.0'
option gateway '192.168.240.1'
option device 'wan.240'
config interface 'LAN1'
option proto 'static'
option device 'lan1'
option ipaddr '192.168.114.65'
option netmask '255.255.255.0'
config interface 'LAN4'
option proto 'static'
option device 'lan4'
option ipaddr '192.168.243.2'
option netmask '255.255.255.0'
config interface 'LAN3'
option proto 'static'
option device 'lan3'
option ipaddr '192.168.242.2'
option netmask '255.255.255.0'
config interface 'LAN2'
option proto 'static'
option device 'lan2'
option ipaddr '192.168.241.2'
option netmask '255.255.255.0'
config device
option type 'bridge'
option name 'brvlan114'
list ports 'lan1'
list ports 'wan'
option bridge_empty '1'
config bridge-vlan
option device 'brvlan114'
option vlan '114'
list ports 'lan1'
list ports 'wan:t'
config device
option type 'bridge'
option name 'brvlan243'
list ports 'lan4'
list ports 'wan'
option bridge_empty '1'
config bridge-vlan
option device 'brvlan243'
option vlan '243'
list ports 'lan4'
list ports 'wan:t'
config device
option type 'bridge'
option name 'brvlan242'
list ports 'lan3'
list ports 'wan'
option bridge_empty '1'
option ipv6 '0'
config bridge-vlan
option device 'brvlan242'
option vlan '242'
list ports 'lan3'
list ports 'wan:t'
config device
option type 'bridge'
option name 'brvlan241'
list ports 'lan2'
list ports 'wan'
option bridge_empty '1'
option ipv6 '0'
config bridge-vlan
option device 'brvlan241'
option vlan '241'
list ports 'lan2'
list ports 'wan:t'
config device
option name 'brvlan114.114'
option type '8021q'
option ifname 'brvlan114'
option vid '114'
option ipv6 '0'
config device
option name 'brvlan241.241'
option type '8021q'
option ifname 'brvlan241'
option vid '241'
option ipv6 '0'
config device
option name 'brvlan242.242'
option type '8021q'
option ifname 'brvlan242'
option vid '242'
option ipv6 '0'
config device
option name 'brvlan243.243'
option type '8021q'
option ifname 'brvlan243'
option vid '243'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '114'
list ports 'lan1'
list ports 'wan:t'
config bridge-vlan
option device 'br-lan'
option vlan '241'
list ports 'lan2'
list ports 'wan:t'
config bridge-vlan
option device 'br-lan'
option vlan '243'
list ports 'lan4'
list ports 'wan:t'
config bridge-vlan
option device 'br-lan'
option vlan '242'
list ports 'lan3'
list ports 'wan:t'
config device
option type '8021q'
option ifname 'wan'
option vid '240'
option name 'wan.240'
option ipv6 '0'
config bridge-vlan
option device 'br-lan'
option vlan '240'
list ports 'wan:t*'
list ports 'wan.240:t*'
/etc/config/firewall :
config defaults
option input 'ACCEPT'
option output 'ACCEPT'
option synflood_protect '1'
option forward 'ACCEPT'
config zone
option name 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
config zone
option name 'wan'
option output 'ACCEPT'
option mtu_fix '1'
list network 'wan'
option input 'ACCEPT'
option forward 'ACCEPT'
config forwarding
option src 'lan'
option dest 'wan'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-Ping'
option src 'wan'
option proto 'icmp'
option icmp_type 'echo-request'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config rule
option name 'Acces_HTTP_depuis_WAN'
option family 'ipv4'
list proto 'tcp'
option src 'wan'
list dest_ip '192.168.240.2'
option dest_port '80'
option target 'ACCEPT'
config zone
option name 'LAN1'
option input 'ACCEPT'
option output 'ACCEPT'
list network 'LAN1'
option forward 'ACCEPT'
config zone
option name 'LAN4'
option input 'ACCEPT'
option output 'ACCEPT'
list network 'LAN4'
option forward 'ACCEPT'
config zone
option name 'LAN3'
option input 'ACCEPT'
option output 'ACCEPT'
list network 'LAN3'
option forward 'ACCEPT'
config zone
option name 'LAN2'
option input 'ACCEPT'
option output 'ACCEPT'
list network 'LAN2'
option forward 'ACCEPT'
config forwarding
option src 'LAN1'
option dest 'wan'
config forwarding
option src 'LAN4'
option dest 'wan'
config forwarding
option src 'LAN3'
option dest 'wan'
config forwarding
option src 'LAN2'
option dest 'wan'
And some screenshots of R5 :
So, R5 is pinged but does not reply. Or it does not reply correctly. I think the reply should be on VLAN240 of WAN.
Something's wrong on R5, which do not reply to ICMP request. I tried everything. It's a Netgear R7000, it should work.
17:24:08.510623 10:7b:ef:ce:61:13 > e4:f4:c6:17:4b:ac, ethertype 802.1Q (0x8100), length 60: vlan 240, p 0, ethertype ARP, Request who-has 192.168.240.2 tell 192.168.114.63, length 42
17:24:08.510671 e4:f4:c6:17:4b:ac > 10:7b:ef:ce:61:13, ethertype 802.1Q (0x8100), length 46: vlan 240, p 0, ethertype ARP, Reply 192.168.240.2 is-at e4:f4:c6:17:4b:ac, length 28
17:24:09.463680 10:7b:ef:ce:61:13 > e4:f4:c6:17:4b:ac, ethertype 802.1Q (0x8100), length 102: vlan 240, p 0, ethertype IPv4, 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6864, seq 2354, length 64
17:24:10.464170 10:7b:ef:ce:61:13 > e4:f4:c6:17:4b:ac, ethertype 802.1Q (0x8100), length 102: vlan 240, p 0, ethertype IPv4, 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6864, seq 2355, length 64
17:24:11.464644 10:7b:ef:ce:61:13 > e4:f4:c6:17:4b:ac, ethertype 802.1Q (0x8100), length 102: vlan 240, p 0, ethertype IPv4, 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6864, seq 2356, length 64
17:24:12.465135 10:7b:ef:ce:61:13 > e4:f4:c6:17:4b:ac, ethertype 802.1Q (0x8100), length 102: vlan 240, p 0, ethertype IPv4, 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6864, seq 2357, length 64
17:24:13.465621 10:7b:ef:ce:61:13 > e4:f4:c6:17:4b:ac, ethertype 802.1Q (0x8100), length 102: vlan 240, p 0, ethertype IPv4, 192.168.114.63 > 192.168.240.2: ICMP echo request, id 6864, seq 2358, length 64
I'm not sure exactly what the final intent is here. It looks like a bunch of locally routed user / guest / iot networks, but then you want them all to link to each other, which defeats the purpose of having isolated guest and iot networks.
It appears that you want the 114 network available everywhere, so the first step would be to bridge it through on all the Ethernet cables. Indeed, it almost looks like the 114 network is the only one that needs to pass between routers, so you could just have it on the Ethernet cables untagged. Each router would hold an IP on .114.X, and then routes to the other networks are installed as .Y.0/24 via .114.X, with X being the IP of the router operating network Y. These routes need only be installed on the main router (3) since it is the ultimately default route for everything else, and it can "bounce" attempting to access a Y network (.240, etc) back into the LAN.
A more advanced topic, which you should try only after fully understanding the basics, would be to route networks like .240 in a device with a better CPU than the GS1900 (such as router 5), then trunk them back to the GS1900 with VLANs on the cable, so then they can take advantage of hardware switching in the 1900 to be dispatched to endpoint devices plugged into the GS1900 at full giagabit speed. First though you'd need to make 114 also tagged on the cable, at least on the one connecting the GS1900 and router 5. Do not mix tagged and untagged on the same cable.
In a DSA device, ethX is usually an internal link from the CPU to the switch, and you should not reference it directly anywhere in your configurations. If ethX is not part of the default configuration, don't add it in.
The problem was in the firewall settings. I get everything working now, without VLAN, except for the 114 subnet part. I'll open another topic if needed.
Thank you for your help !