Router on a stick configuration - Not sure why ACL doesn't work

sputnik · December 5, 2024, 10:21pm

Hi,

I have configured a router on a stick with two VLANs and am trying to get the ACLs working.

My configuration is simple. I use an OpenWRT 25.05 router and a managed switch that supports VLAN.
On both I used Port 1 as trunk.

VLAN1 = LAN IP (192.168.1.2)
VLAN20 = DMZ IP (192.168.20.1)
PC IP on LAN = 192.168.1.20
Server IP on DMZ = 192.168.20.10

Router configuration
IP: 192.168.1.2
Mask: 255.255.255.0

VLAN ID	Description	CPU(eth0)	CPU(eth1)	LAN 1 (trunk)	LAN 2 (PC)	WAN
1	LAN	off	tagged	untagged	untagged	off
7	ISP	tagged	off	off	off	tagged
20	DMZ	off	tagged	tagged	off	off

config interface 'loopback'       
        option device 'lo'        
        option proto 'static'     
        option ipaddr '127.0.0.1'              
        option netmask '255.0.0.0'             
                                               
config globals 'globals'                       
        option ula_prefix ''
                            
config device               
        option name 'br-lan'
        option type 'bridge'  
        list ports 'eth1.1'   
                                   
config interface 'lan'                
        option device 'br-lan'        
        option proto 'static'         
        option ipaddr '192.168.1.2'   
        option netmask '255.255.255.0'
        option ip6assign '60'       
        option gateway '192.168.1.1'
                                                                  
config interface 'wan'                                            
        option device 'eth0.7'                                    
        option proto 'pppoe'                                      
        option username ''
        option password ''
        option ipv6 'auto' 
        option disabled '1'   
        option auto '0'       
                                                                  
config interface 'wan6'                                           
        option device 'eth0.7'                                    
        option proto 'pppoe'                                      
        option username ''
        option password ''
        option ipv6 'auto' 
        option disabled '1'  
        option auto '0'      
                              
config switch                 
        option name 'switch0' 
        option reset '1'       
        option enable_vlan '1' 
                                 
config switch_vlan               
        option device 'switch0'  
        option vlan '1'          
        option ports '1 2 3 4 6t'
        option vid '1'          
        option description 'LAN'
                               
config switch_vlan              
        option device 'switch0' 
        option vlan '3'         
        option vid '20'         
        option description 'DMZ'
        option ports '4t 6t'  
                                    
config interface 'dmz'                
        option proto 'static'         
        option device 'br-dmz'        
        option ipaddr '192.168.20.1'  
        option netmask '255.255.255.0'

config route                                                                                                                                                                               
        option interface 'lan'                                                                                                                                                             
        option target '192.168.20.10/24'                                                                                                                                                   
        option disabled '1'

Switch configuration
IP: 192.168.1.3
Mask: 255.255.255.0
Gateway: 192.168.1.2

VLAN ID	Description	LAN 1 (trunk)	...	LAN 9 Server (DMZ)
1	LAN	untagged
20	DMZ	tagged		untagged

Firewall configuration

config defaults                                 
        option input 'REJECT'                   
        option output 'ACCEPT'                  
        option forward 'DROP'                   
        option synflood_protect '1'             
        option drop_invalid '1'                 
                                                
config zone                                     
        option name 'lan'                       
        option input 'ACCEPT'                   
        option output 'ACCEPT'                  
        option forward 'ACCEPT'                 
        list network 'lan'                      
                                                
config forwarding                               
        option src 'lan'                        
        option dest 'wan'                       
                                                
config rule                                     
        option name 'Allow-DHCP-Renew'          
        option src 'wan'                        
        option proto 'udp'                      
        option dest_port '68'                   
        option target 'ACCEPT'                  
        option family 'ipv4'                    
                                                
config rule                                     
        option name 'Allow-Ping'                
        option src 'wan'                        
        option proto 'icmp'                     
        option icmp_type 'echo-request'         
        option family 'ipv4'                    
        option target 'ACCEPT'                  
                                                
config rule                                     
        option name 'Allow-IGMP'                
        option src 'wan'                        
        option proto 'igmp'                     
        option family 'ipv4'                    
        option target 'ACCEPT'                  
                                                
config rule                                     
        option name 'Allow-DHCPv6'              
        option src 'wan'                        
        option proto 'udp'                      
        option dest_port '546'                  
        option family 'ipv6'                    
        option target 'ACCEPT'                 
                                                
config rule                                     
        option name 'Allow-MLD'                 
        option src 'wan'                        
        option proto 'icmp'                     
        option src_ip 'fe80::/10'               
        list icmp_type '130/0'                  
        list icmp_type '131/0'                  
        list icmp_type '132/0'                  
        list icmp_type '143/0'                  
        option family 'ipv6'                    
        option target 'ACCEPT'         

config rule                                                                                                                                                                                
        option name 'Allow-ICMPv6-Input'                                                                                                                                                   
        option src 'wan'                                                                                                                                                                   
        option proto 'icmp'                                                                                                                                                                
        list icmp_type 'echo-request'                                                                                                                                                      
        list icmp_type 'echo-reply'                                                                                                                                                        
        list icmp_type 'destination-unreachable'                                                                                                                                           
        list icmp_type 'packet-too-big'                                                                                                                                                    
        list icmp_type 'time-exceeded'                                                                                                                                                     
        list icmp_type 'bad-header'                                                                                                                                                        
        list icmp_type 'unknown-header-type'                                                                                                                                               
        list icmp_type 'router-solicitation'                                                                                                                                               
        list icmp_type 'neighbour-solicitation'                                                                                                                                            
        list icmp_type 'router-advertisement'                                                                                                                                              
        list icmp_type 'neighbour-advertisement'                                                                                                                                           
        option limit '1000/sec'                                                                                                                                                            
        option family 'ipv6'                                                                                                                                                               
        option target 'ACCEPT'                                                                                                                                                             
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'Allow-ICMPv6-Forward'                                                                                                                                                 
        option src 'wan'                                                                                                                                                                   
        option dest '*'                                                                                                                                                                    
        option proto 'icmp'                                                                                                                                                                
        list icmp_type 'echo-request'                                                                                                                                                      
        list icmp_type 'echo-reply'                                                                                                                                                        
        list icmp_type 'destination-unreachable'                                                                                                                                           
        list icmp_type 'packet-too-big'                                                                                                                                                    
        list icmp_type 'time-exceeded'                                                                                                                                                     
        list icmp_type 'bad-header'                                                                                                                                                        
        list icmp_type 'unknown-header-type'                                                                                                                                               
        option limit '1000/sec'                                                                                                                                                            
        option family 'ipv6'                                                                                                                                                               
        option target 'ACCEPT'                                                                                                                                                             
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'Allow-IPSec-ESP'                                                                                                                                                      
        option src 'wan'                                                                                                                                                                   
        option dest 'lan'                                                                                                                                                                  
        option proto 'esp'                                                                                                                                                                 
        option target 'ACCEPT'                                                                                                                                                             
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'Allow-ISAKMP'                                                                                                                                                         
        option src 'wan'                                                                                                                                                                   
        option dest 'lan'                                                                                                                                                                  
        option dest_port '500'                                                                                                                                                             
        option proto 'udp'                                                                                                                                                                 
        option target 'ACCEPT'                                                                                                                                                             
                                                                                                                                                                                           
config zone                                                                                                                                                                                
        option name 'DMZ'                                                                                                                                                                  
        option input 'REJECT'                                                                                                                                                              
        option output 'ACCEPT'                                                                                                                                                             
        option forward 'DROP'                                                                                                                                                              
        option log '1'                                                                                                                                                                     
        list network 'dmz'                                                                                                                                                                 
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'DMZ-DNS'                                                                                                                                                              
        option src 'DMZ'                                                                                                                                                                   
        option dest_port '53'                                                                                                                                                              
        option target 'ACCEPT'

config forwarding                                                                                                                                                                          
        option src 'DMZ'                                                                                                                                                                   
        option dest 'wan'                                                                                                                                                                  
                                                                                                                                                                                           
config zone                                                                                                                                                                                
        option name 'wan'                                                                                                                                                                  
        option input 'DROP'                                                                                                                                                                
        option output 'ACCEPT'                                                                                                                                                             
        option forward 'DROP'                                                                                                                                                              
        option masq '1'                                                                                                                                                                    
        option mtu_fix '1'                                                                                                                                                                 
        list network 'wan'                                                                                                                                                                 
        list network 'wan6'                                                                                                                                                                
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'LAN-NAS-Ping'                                                                                                                                                         
        list proto 'icmp'                                                                                                                                                                  
        option src 'lan'                                                                                                                                                                   
        option dest 'DMZ'                                                                                                                                                                  
        option target 'ACCEPT'                                                                                                                                                             
        list src_ip '192.168.1.20'                                                                                                                                                         
        list dest_ip '192.168.20.10'                                                                                                                                                       
        option family 'ipv4'                                                                                                                                                               
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'LAN-NAS-HTTPS'                                                                                                                                                        
        list proto 'tcp'                                                                                                                                                                   
        option src 'lan'                                                                                                                                                                   
        option dest 'DMZ'                                                                                                                                                                  
        option target 'ACCEPT'                                                                                                                                                             
        option family 'ipv4'                                                                                                                                                               
        option dest_port '443'                                                                                                                                                             
                                                                                                                                                                                           
config rule                                                                                                                                                                                
        option name 'LAN-NAS-ssh'                                                                                                                                                          
        list proto 'tcp'                                                                                                                                                                   
        option src 'lan'                                                                                                                                                                   
        list src_ip '192.168.1.20'                                                                                                                                                         
        option dest 'DMZ'                                                                                                                                                                  
        list dest_ip '192.168.20.10'                                                                                                                                                       
        option dest_port '22'                                                                                                                                                              
        option target 'ACCEPT'

Problem description
I managed to setup the ACL for a ping from LAN to DMZ.
My problem is that when I try to connect either via ssh or https from LAN to DMZ, I get debug1: connect to address 192.168.20.10 port 22: Network is unreachable.
What I can do is to setup a static route from LAN to DMZ or from my PC to the Server, but then the ACLs are not needed anymore and I have full access to the Server.
I would like to be able to limit the access on IP and Port level.

Does anyone have a clue what I am doing wrong? I checked multiple sources but couldn't find the reason, why the ACLs are not working.

Thank you!

psherman · December 5, 2024, 11:40pm

Please post the complete firewall config. And also, if the network config is not the full ouput, please post that complete file, too.

psherman · December 5, 2024, 11:57pm

I'm noticing that this router doesn't appear to be the primary/only router -- there seems to be an upstream router at 192.168.1.1. Is that correct?

sputnik:

config interface 'lan'
        option device 'br-lan'
        option proto 'static'
        option ipaddr '192.168.1.2'
        option netmask '255.255.255.0'
        option ip6assign '60'
        option gateway '192.168.1.1'

This would be expected if the upstream network (192.168.1.0/24) and its router (192.168.1.1) is not aware of the downstream network (192.68.20.0/24).

Does your upstream router support static routes? That is a requirement to achieve what you are asking based on your current topology.

mk24 · December 6, 2024, 1:37am

option src_port will prevent those rules from ever matching. When a client opens a ssh or https connection it chooses a random port as the source. The dest_port is fixed at the server depending on the type of service, that is what the firewall should match. Remove any src_port settings.

sputnik · December 6, 2024, 3:51pm

Hi @psherman , currently this setup is offline. I try first to make everything work, before connecting it to the internet.

There is router with integrated modem, running in modem mode with the IP 192.168.1.1. But this is the connection to the internet and I try to focus on the local network first. So the Router/Modem is not connected yet.

Regarding the configuration, I will update it in the main post. I doubt it will help as I left the standard configuration and firewall entries out.

sputnik · December 6, 2024, 3:51pm

Hi @mk24 I have updated the firewall rules and removed the src_port, still the issue remained

psherman · December 6, 2024, 3:56pm

Even though you haven't yet connected this router on a stick to the upstream modem+router described, this still applies:

It applies in the immediate term for whatever your upstream network looks like, and it will be critical in the intended network.

sputnik · December 6, 2024, 4:11pm

As far as I understand, yes, the router 192.168.1.2 supports static routes

I have difficulties following you on the issue :-/

psherman · December 6, 2024, 4:16pm

What you're showing is the router you're configuring now, right? 192.168.1.2 seems to be the IP address of the OpenWrt router we've been talking about.

You need to have static routes installed on the upstream router.

Can you draw a quick diagram that shows us how you have this stuff physically connected right now? Is there a router currently upstream of this "router on a stick" device?

sputnik · December 6, 2024, 4:32pm

Yes, I am showing router 192.168.1.2
Here is the diagram

psherman · December 6, 2024, 5:40pm

Is the router at 192.168.1.1 configured with a static route? Like I said before, this will be required. And, more importantly, does the modem+router in the final network plan (when you actually deploy this) support static routes? If not, the plan you have will not work as expected.

sputnik · December 6, 2024, 7:49pm

Thank you for the heads up.
I have another router here that has OpenWRT on it. If I configure the network this way, would it solve the problem? I guess it does, just don't know yet, how.

Also I can install OpenWRT firmware on the switch. Will it be possible to use it as a Layer 3 switch with OpenWRT? I never had a switch with OpenWRT unti now.

psherman · December 6, 2024, 7:51pm

Yes, it should, although I haven't looked at the config in detail to ensure there aren't any errors.... but fundamentally, this is required.

However, the critical question is if this functionality will be available on the modem+router in the proposed location.

sputnik · December 6, 2024, 7:54pm

The modem+router does not have the possibility to setup static routes.

How about installing OpenWRT on the switch? Will it become a Layer 3 switch?

psherman · December 6, 2024, 7:57pm

This will be a deal breaker.

Not a simple answer.... if your switch is supported, yes, you can install OpenWrt. But routing (L3) on a switch is usually not going to work well unless the switch is actually sold (from the vendor) as an L3 switch. This is because of CPU limitations... expect <=20Mbps or so for routing on a normal managed switch.

And this would only work if the two devices that are trying to connect to each other both connect through the switch itself.

sputnik · December 6, 2024, 8:23pm

I do not understand the problem.
If the VLAN management and routing is only taken care of by the switch and the router 192.168.1.2, what is the problem with the router+modem?

And if there is a problem, why is another router not solving it? I thought the hop until the router 192.168.1.4 would have no VLAN tags etc. in the frames and data would be able to go back and forth

psherman · December 6, 2024, 8:26pm

The issue is that the devices on 192.168.1.0/24 will be using the main router (likely 192.168.1.1) as its gateway. A request will come in from one of the hosts for a device on 192.168.20.0/24. The main router will not know where to direct that traffic.

This is why static routes are required. This allows the main router (192.168.1.1) to understand that it can send traffic to 192.168.20.0/24 via 192.168.1.2.

sputnik · December 6, 2024, 8:39pm

If I setup a static route on the router 192.168.1.2, I can reach the VLAN 20.
So I setup LAN (VLAN 1) -> 192.168.20.10/24 and I can access the VLAN 20 from VLAN 1.
The problem is that the ACLs are not triggering. Only ICMP was working via ACL, I could activate and deactivate the rule and it suppressed the ping or let it go trough.

And what I still don't understand is the following.
The router 192.168.1.2 and the switch 192.168.1.3 are responsible to manage the VLAN.
Router+modem 192.168.1.1 is only needed for IPs outside the local network.
I could also use another device as the gateway. For example a second OpenWRT router 192.168.1.4

How is it usually solved? Or what are the best practices to manage VLAN at home? I thought I have already all necessary hardware. Could I fix it by buying one more or another component?

psherman · December 6, 2024, 8:53pm

Kind of... but you're missing the point of the static routes.

Router+modem 192.168.1.1 is the main router. The DHCP server will issue a DHCP lease to a device connected to the network. Let's say it issues an address 192.168.1.213 with a subnet 255.255.255.0 (/24). It will also send option 3 (gateway) and option 6 (DNS) so that the host at .213 knows how to reach the internet and get DNS services.

Now the host at 192.168.1.213 wants to connect to another host.

If that other host is on the 192.68.1.0/24 network (i.e. 192.168.1.1 - 192.168.1.254), no routing is required -- this happens at L2 and it just sends the request.
If that host is on the internet (say openwrt.org which is 64.226.122.113), the request goest to the main router and then out to the internet because the main router recognizes that it does not have a direct connection to the network that begins with 64.
Likewise, if the target host is on 192.168.20.0/24, say 192.168.20.47, the host 192.168.1.213 will recognize that that is a different network and so it will send the request to the main router to route to the other network. The problem here is that the main router isn't aware of the 192.168.20.0/24 network, so it doesn't know where to send the traffic. It may or may not send it out via the ISP, but obviously that's not going to help.

The static route is there to inform the main router that there is this network 192.168.20.0, and if you see connections with that destination address, send those to 192.168.1.2 (behind which is this new 192.168.20.0/24 network).

You can simplistically think of this as the "care of" method of addressing mail to someone. Let's say someone wants to send a package to a family member via your work address. The mailroom at your office would not know where to send this package because they don't recognize that name... nobody with that name works at the company. So, if the sender says <family member name> c/o <employee name>, the mail can be appropriately routed by the company mail room to the employee who will, in turn, give it to the family member.

If that static route cannot be added, the main router will not be able to route the traffic appropriately, so it will not work.

sputnik · December 6, 2024, 9:51pm

Thank you for trying and explaining it to me.

What bothers me is the fact that in all scenarios I have read and watched, the router on a stick can be done with one router and a switch. I wonder why this does not work for me, or how to realize it with OpenWRT.

Here is one example, but there are many more

Or do you want to tell me that I could realize inter-VLAN communication but it would not help me in the end, because my main router can not setup static routes and the communication to the internet would fail?

What would be a possible solution be? I don't know many routers with modem that have this feature.
Also I didn't see any router+modem that is supporting OpenWRT, so I could operate a router+modem that has static routing.

How did the others realize it at home using VLAN?

So one of the problems is that I would have to tell the 192.168.1.1 that packages that are addressed to 192.168.20.0/24 need to be send to 192.168.1.2.
This static route would be needed to make it work, correct? At least from the internet to the VLAN. Outbound there would also a route needed etc.? And that per VLAN and so on?