OpenWRT LAN Routing and Firewall Setting

woogx · September 9, 2024, 4:07am

I have been using an Arm-based openWRT R4S device for the past 3 years. Recently, I decided to upgrade my internal network capability to 10G/2.5G, which requires me to update the hardware from R4S to Mini Route R1. The upgrade was rather straight forward and went without a hitch. That said, this did require me to migrate some of my internal service under R4S to Mini Route. I had a thought about it, Instead of migration, I can still retain R4S as a redundancy for some non-critical service just in case my Mini Route decide to quit. So I designed the following topology and implemented it under GNS3. After a bit of tuning, it worked (I do not have the proper firmware for NX30, NX54 and TP-Link Layer 2 Switch, so I used Cisco IOSv and IOSv2 instead.).

However, after I implemented the same configuration in the real environment, it just didn’t work. I would love for the some input so that I can proceed further. Please find my configuration below;

Static Routing

NX54

ip route 0.0.0.0 0.0.0.0 192.168.11.1

ip route 192.168.1.0 255.255.255.0 192.168.11.1

ip route 192.168.125.0 255.255.255.0 192.168.11.1

ip route 192.168.101.0 255.255.255.0 192.168.11.1

ip route 192.168.10.0 255.255.255.0 192.168.11.1

R4S

ip route 0.0.0.0 0.0.0.0 192.168.1.1

ip route 192.168.124.0 255.255.255.0 192.168.11.104

ip route 192.168.125.0 255.255.255.0 192.168.1.2

ip route 192.168.10.0 255.255.255.0 192.168.1.2

ip route 192.168.101.0 255.255.255.0 192.168.1.2

NX30

Ip route 0.0.0.0 0.0.0.0 192.168.1.1

Ip route 192.168.10.0 255.255.255.0 192.168.125.9

ip route 192.168.101.0 255.255.255.0 192.168.125.9

ip route 192.168.11.0 255.255.255.0 192.168..1.3

ip route 192.168.124.0 255.255.255.0 192.168.1.3

MiniRoute R1

ip route 0.0.0.0 0.0.0.0 192.168.125.1

ip route 192.168.1.0 255.255.255.0 192.168.125.1

ip route 192.168.11.0 255.255.255.0 192.168.125.1

ip route 192.168.124.0 255.255.255.0 192.168.125.1

Firewall Setting

In GNS3, the only change I configured were the following;

R4S, in default, forward changed from ‘reject’ to ‘accept’

MiniRoute R1, WAN -> Reject, disable masquerade

I tried above in live environment, it didn’t work. Both PCs can reach Internet without any problem. However, they cannot reach each other. I tried ping to different segment, and I found the following;

R4S side can only reach up to 192.168.1.2 (Gateway of NX30).

MiniRoute side can only reach up to 192.168.1.3 (Gateway of R4S).

After the some investigation, I really can’t think of anything wrong with the routing. So I decided to continue experimenting on firewall. Because Miniroute is a bit hard to reload if anything goes wrong, I decided to make R4S pingable first. I tried following; but none worked.

Combination 1:

Default Firewall Behavior: accept, accept, accept

WAN -> Reject reject, accept, accept

Combination 2:

Default Firewall Behavior: accept, accept, accept

WAN -> Reject accept, accept, forward

Combination 3

Default Firewall Behavior: accept, accept, accept

WAN -> LAN reject, accept, accept

Combination 4

Default Firewall Behavior: accept, accept, accept

WAN -> Reject reject, accept, reject

A Firewall Rule

Home_Internal any protocol from 192.168.125.0/24 at WAN to all IPs in this device.

Also, interestingly, I can ping from PC under NX54 to 192.168.1.2. But after I sshed into R4s, I can ping 192.168.124.1, but not the terminal PC.

I’d appreciate it if someone can help me further this. Many thanks.

pavelgl · September 9, 2024, 6:52am

If the NX30 is not running OpenWrt, I can't help, but here are some basic principles for the OpenWrt devices.

You can keep the default firewall settings, but you need to exclude local networks from masquerading and create a wan=>lan traffic rule.

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        list network 'wan'
        list masq_dest '!192.168.0.0/16'
...

config rule
        option name 'Allow-WAN-LAN'
        list proto 'all'
        option src 'wan'
        list src_ip '192.168.0.0/16'
        option dest 'lan'
        option target 'ACCEPT'

If this is a Windows machine, the default firewall settings do not allow requests originating outside the local subnet.

When the next hop is the same, it is sufficient to set the default route. Other static routes become redundant.

woogx · September 9, 2024, 2:53pm

Hi Pavel,

Thanks for this. I have given some careful thought over your suggestions. Let me work my way through those. Let's start with the easiest.

NX30, unfortunately, NX30 is not running openWRT. It runs its default firmware from H3C.
Windows firewall issue for subnet 192.168.124.0/24
Actually, Windows firewall issue was the first thing I thought about. So I created an inbound firewall rule Home_Wu.
(upload://4WzhZvsU1FKwquogv7iuylWkQkZ.png)
Please don' t mind the language. Basically, it allows inbound traffic from subnet 192.168.11.0/24 and 192.168.101.0/24. Then I also add the those ip subnets in the remote ip range. Neither worked.

Let's say my Windows firewall inbound rule was correctly configured. I actually did a bit of investigation myself in between. The H3C NX54 router is one of those home routers offering basic Internet connectivity, switching and AP function consisting of 1 WAN and 3 LAN ports. My suspision is that all 192.168.124.0/24 were natted since WAN port was applied. Hence, my laptop on 124 range can reach out, but 192.168.11.0/24 cannot get in. Well, just my theory.

Default and Static Route
In principle, I'd agree with you that only a default route is needed since the next hop is the same (192.168.11.1). However, my test on Internet capability on Friday in GNS3 was not successful. So I just repeated it on GNS3. It went through fine. I believe the issue was due to International connectivity issue. So I have removed those static routes (thank you for pushing me to test it again.).
Now, the exciting stuff. I like your suggestion, but I haven't implemented it yet. if I am not wrong, you are worried that all private IPs will be translated into public IPs. Therefore, they will not be able to reach LANs on either side. Question though, let's say if I exclude 192.168.0.0/16 range from masquerading, will it still allow the lan clients on both R4S and MiniRoute to reach Internet?

I also did further investigation on my GNS3 setup. Apparently, the nat function provided on NX30 didn't have any hit with the following configuration. This may have something to do with my misunderstanding on the Cloud usage offered by GNS3.

The NX30 in GNS3 does have nat configuration.
access-list 1 permit 192.168.125.0 255.255.255.0
access-list 1 permit 192.168.101.0 255.255.255.0
access-list 1 permit 192.168.10.0 255.255.255.0
int g0/0
ip nat outside
exit
int g0/1
ip nat inside
exit
ip nat inside source list 1 interface g0/0 overload

pavelgl · September 9, 2024, 6:16pm

If masquerading is enabled, the client request is SNAT'ed to the device's WAN IP address, whether private or public.

When you log into the R4S from the PC, check the initiator IP address.
If it's 192.168.11.104 (and not 192.168.124.X), then masquerading on NX54 is enabled, which would explain why you can't ping the PC from R4S.

Masquerading should only be enabled on both gateways with the exception of LANs.
You could also disable masquerading on the gateways, but you must set static routes on the ISP device.

Hope this helps.

woogx · September 10, 2024, 12:42pm

Hi Pavel,

If I take out NX30, and have MiniRoute connected to ISP, and configure the following on both R4S and MiniRoute, does it mean that I will not be able to access the Internet, but instead, both network on R4S and Miniroute side can communicate with each other?

config zone
option name 'wan'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
list network 'wan'
list masq_dest '!192.168.0.0/16'
...

config rule
option name 'Allow-WAN-LAN'
list proto 'all'
option src 'wan'
list src_ip '192.168.0.0/16'
option dest 'lan'
option target 'ACCEPT'

pavelgl · September 10, 2024, 3:33pm

Both will work. For starters, isolate the non-OpenWrt devices and connect the PCs to R4S and Miniroute R1 directly.

Assuming R1 keeps the lan IP and the wan IP changes to 192.168.1.2, add the following static routes in /etc/config/network (in addition to the firewall rules).

# Miniroute R1
config route
        option interface 'wan'
        option target '192.168.11.0/24'
        option gateway '192.168.1.3'

#R4S
config route
        option interface 'wan'
        option target '192.168.10.0/24'
        option gateway '192.168.1.2'

Both PCs should have Internet access and be able to see each other (if you open the windows firewalls).

woogx · September 15, 2024, 11:33pm

Hi Pavel,

I am sorry this took a bit of time. I finally took down the network and did a bit of re-wiring to get rid of NX30. Unfortunately, this didn't quite work.

Please see the config below

MiniRoute R1
config interface 'lan'
option device 'br-lan'
option proto 'static'
option ipaddr '192.168.10.1'
option netmask '255.255.255.0'
option ip6assign '60'

config interface 'vlan101'
option proto 'static'
option ipaddr '192.168.101.1'
option netmask '255.255.255.0'
option device 'eth0.101'
option gateway '192.168.10.1'

config route
option interface 'wan'
option target '192.168.11.0/24'
option gateway '192.168.1.3'

config route
option interface 'wan'
option target '192.168.124.0/24'
option gateway '192.168.1.3'

config rule
option name 'Home_Internal'
option src 'wan'
list src_ip '192.168.0.0/16'
option dest 'lan'
option target 'ACCEPT'
list proto 'all'

config zone
option name 'wan'
list network 'wan'
list network 'wan6'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
option auto_helper '0'
list masq_dest '192.168.0.0/16'

R4S
config route
option interface 'wan'
option target '192.168.101.0'
option gateway '192.168.1.2'
option netmask '255.255.255.0'

config route
option interface 'wan'
option target '192.168.10.0'
option netmask '255.255.255.0'
option gateway '192.168.1.2'

config rule
option target 'ACCEPT'
option src 'wan'
option name 'home-internal'
option proto 'all'
option src_ip '192.168.0.0/16'
option dest 'lan'

config zone
option name 'wan'
option output 'ACCEPT'
option mtu_fix '1'
option input 'REJECT'
option forward 'REJECT'
option log '1'
option masq '1'
option network 'wan wan6'
list masq_dest '192.168.0.0/16'

I actually logged onto both R4S and MiniRoute R1 and ping each other's plan interface 192.168.11.1, 192.168.10.1 and 192.168.101.1. They can't get to each other. But they still can get to 192.168.1.0/24 range. Further, PCs on both end can't reach the Internet. I sort of did this config prior to the re-wiring on just R4S side, and I couldn't get to the Internet, either. Originally, the system language on the openWRT was not in English, which somehow didn't quite make sense to me on the masquerading part. But after I set to the English, it made sense. If I am not wrong, it was sort of like a conditional masquerade depending on the destination address. If the destination address falls into 192.168.0.0/16 range, then traffic will not be natted and will be forwarded onto the next hop of either 192.168.1.2 or 192.168.1.3 for internal lan communication. Otherwise, it will be natted with a default gateway of 192.168.1.1 for Internet communication. Can you suggest what else I may have missed out? thanks.

woogx · September 15, 2024, 11:37pm

By the way, once I took out the 192.168.0.0/16 configuration for Restrict Masquerading to given destination subnets. Internet came back up right away.

pavelgl · September 16, 2024, 5:14am

The idea was to EXCLUDE given networks from masquerading.

How did the exclamation mark magically disappear?

woogx · September 16, 2024, 7:13am

Honestly, this is very new. I didn't even notice the '!' mark. The way I did was via the interface like the following;

I suppose I can use 'nano' or 'vim' to hard code it in the firewall. Just out of curiosity, is there any way you can point me where to enable that '！‘ in the web interface? Thanks.

woogx · September 16, 2024, 7:17am

Oh, no, you do not have to. I figured it out. Basically, the field took string instead any network command. This is pretty cool.

I will take down the network when my family stop playing video games to test. Great help.

woogx · September 17, 2024, 12:10am

Hi Pavel,

The good news is that both networks can reach the Internet after the implementation of '!' mark. the bad news is that they still cannot reach each other. Before the successful implementation of Masquerade Restriction. PCs on both ends can reach 192.168.1.2 or 192.168.1.3 gateway. But now, PCs cannot reach those gateways anymore. That said, if I log onto the Miniroute or R4S, they still can reach the gateway, but they cannot reach each other's LAN segment. Please see all associated config below. Would you mind casting your eyes over these? Thanks.

MiniRoute R1
config zone
option name 'wan'
list network 'wan'
list network 'wan6'
option input 'REJECT'
option output 'ACCEPT'
option forward 'REJECT'
option masq '1'
option mtu_fix '1'
option auto_helper '0'
list masq_dest '!192.168.0.0/16'

config rule
option name 'Home_Internal'
option src 'wan'
list src_ip '192.168.0.0/16'
option dest 'lan'
option target 'ACCEPT'
list proto 'all'

config route
option interface 'wan'
option target '192.168.11.0/24'
option gateway '192.168.1.3'

config route
option interface 'wan'
option target '192.168.124.0/24'
option gateway '192.168.1.3'

config interface 'FVLAN101'
option proto 'static'
option device 'eth2.101'
option ipaddr '192.168.101.1'
option netmask '255.255.255.0'
option gateway '192.168.10.1'
list dns '192.168.10.1'

config interface 'wan'
option device 'eth1'
option proto 'static'
option ipaddr '192.168.1.2'
option netmask '255.255.255.0'
option gateway '192.168.1.1'

R4S
config rule
option target 'ACCEPT'
option src 'wan'
option name 'home-internal'
option proto 'all'
option src_ip '192.168.0.0/16'
option dest 'lan'

config zone
option name 'wan'
option output 'ACCEPT'
option mtu_fix '1'
option log '1'
option masq '1'
option input 'REJECT'
option forward 'REJECT'
option network 'wan wan6'
list masq_dest '!192.168.0.0/16'

config route
option interface 'wan'
option target '192.168.101.0'
option gateway '192.168.1.2'
option netmask '255.255.255.0'

config route
option interface 'wan'
option target '192.168.10.0'
option netmask '255.255.255.0'
option gateway '192.168.1.2'

config route
option interface 'lan'
option target '192.168.124.0'
option netmask '255.255.255.0'
option gateway '192.168.11.104'

config interface 'lan'
option type 'bridge'
option ifname 'eth1'
option proto 'static'
option ipaddr '192.168.11.1'
option netmask '255.255.255.0'
option ip6assign '60'

config interface 'wan'
option ifname 'eth0'
option _orig_ifname 'eth0'
option _orig_bridge 'false'
option proto 'static'
option ipaddr '192.168.1.3'
option netmask '255.255.255.0'
option gateway '192.168.1.1'
option broadcast '192.168.1.255'