Cascaded port-forward: ISP Cable Modem to OpenWRT Router to Local Linux Host Hanging

NYCeyes · December 3, 2024, 5:29am

Hello Friends:

I've port-forwarded many times, but this is my first attempt at a cascaded port-forwards across two routers, which is not working for me. I must be missing a network setting needed for this case. I'll try to present my issue as simply as I can via bullets.

ISP Cable Modem-IP (LAN-SIDE): 192.168.1.1/24
OpenWRT Router-IP (WAN-SIDE): 192.168.1.2/24
OpenWRT Router-IP (LAN-SIDE): 192.168.0.1/24
End Host (Fedora/Linux) IP (LAN): 192.168.0.36/24

Now, except for Port-Forwarding, I don't use any feature of the ISP Cable Modem (not WIFI, Not Firewall, Not DHCP, not anything). Although it's a robust ISP Modem (an ARRIS G36) it's only purpose is getting me to my ISP datacenter.

The end host - 192.168.0.36 Fedora/Linux - has SSH port 22 open and, as the following images show, this is what has been cascade-forwarded. (Note: We'll only focus on port-22 since solving for that will solve for everything else).

ISP CABLE MODEM ROUTER (192.168.1.1 WAN):

OPENWRT INTERNAL ROUTER (192.168.1.2 WAN || 192.168.0.1 LAN):

From my cloud webhost port (out on the internet) I try this:

user@cloud$ ssh -p 22 -l user my.wap.ip

but that hangs.

Now, I know that the ISP CABLE MODEM ROUTER forwarding capability correctly works because, when I temporarily forward it to a different Fedora/Linux Host directly connected to one of it's Ethernet ports (e.g., to 192.168.1.15) , the above SSH succeeds - no issues.

The problem is cascaded hops from the ISP MODEM to OPENWRT ROUTER to above mentioned internal Fedora/Linux HOST (at 192.168.0.36). It doesn't appear to take the 2nd hop.

Perhaps there's a network parameter I'm missing on OPENWRT (e.g. hop count or TTL). Again, this is the first time I've every tried a multi-hop / cascaded scenario.

Any help or guidance would be appreciated. Thank you all in advance!

brada4 · December 3, 2024, 5:53am

Please post output of

ubus call system board

from your OpenWRT device.

SSH does not use UDP jic

eduperez · December 3, 2024, 6:25am

Remove the LAN to LAN redirection, you do not need it here.

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/firewall

NYCeyes · December 3, 2024, 8:16am

Hi @eduperez @brada4

Thank you both for getting back to me. You're correct that the LAN to LAN mappings were unnecessary. I was troubleshooting and forgot to remove those entries, but did so prior to collecting the following outputs requested.

Incidentally, you'll notice that the version of OpenWRT is a somewhat dated, and tied to the appliance it's associated to, which is updated by the manufacturer. I wish it were newer, but at the same time it's stable and works well for what it does. Just FYI that I'm aware.

Finally, there wasn't much sensitive to redact, but I did check. Thank you.

=====================================================
ubus call system board
=====================================================
{
	"kernel": "5.4.211",
	"hostname": "GL-MT2500",
	"system": "ARMv8 Processor rev 4",
	"model": "GL.iNet GL-MT2500",
	"board_name": "glinet,mt2500-emmc",
	"release": {
		"distribution": "OpenWrt",
		"version": "21.02-SNAPSHOT",
		"revision": "r15812+901-46b6ee7ffc",
		"target": "mediatek/mt7981",
		"description": "OpenWrt 21.02-SNAPSHOT r15812+901-46b6ee7ffc"
	}
}
=====================================================



=====================================================
cat /etc/config/network
=====================================================
config interface 'loopback'
	option device 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'
	option ula_prefix 'fd3e:5c8c:0415::/48'

config device
	option name 'br-lan'
	option type 'bridge'
	list ports 'eth1'
	option ipv6 '0'

config device
	option name 'eth1'
	option macaddr '94:83:c4:2a:69:8a'
	option ipv6 '0'

config interface 'lan'
	option device 'br-lan'
	option proto 'static'
	option netmask '255.255.255.0'
	option ip6assign '60'
	option isolate '0'
	option broadcast '192.168.0.255'
	option ipaddr '192.168.0.1'
	option gateway '192.168.1.2'

config device
	option name 'eth0'
	option macaddr '94:83:c4:2a:69:89'
	option ipv6 '0'

config interface 'wan'
	option device 'eth0'
	option force_link '0'
	option ipv6 '0'
	option proto 'static'
	option netmask '255.255.255.0'
	option peerdns '0'
	option gateway '192.168.1.1'
	option metric '10'
	option ipaddr '192.168.1.2'
	option vlanid '0'
	option dns '94.140.14.14 94.140.15.15'

config interface 'wan6'
	option proto 'dhcpv6'
	option disabled '1'
	option device '@wan'

config interface 'tethering6'
	option proto 'dhcpv6'
	option disabled '1'
	option device '@tethering'

config interface 'wwan6'
	option proto 'dhcpv6'
	option disabled '1'
	option device '@wwan'

config interface 'modem_1_1_2_6'
	option proto 'dhcpv6'
	option disabled '1'
	option device '@modem_1_1_2'

config rule 'policy_direct_rt'
	option lookup 'main'
	option suppress_prefixlength '0'
	option priority '1100'

config rule 'policy_default_rt_vpn'
	option mark '0x8000/0xc000'
	option lookup '8000'
	option priority '1101'
	option invert '1'

config rule6 'policy_direct_rt6'
	option lookup 'main'
	option suppress_prefixlength '0'
	option priority '1100'

config rule6 'policy_default_rt_vpn6'
	option mark '0x8000/0xc000'
	option lookup '8000'
	option priority '1101'
	option invert '1'

config rule 'policy_default_rt_vpn_ts'
	option lookup 'main'
	option priority '1099'
	option mark '0x80000/0xc0000'
	option invert '0'
=====================================================



=====================================================
cat /etc/config/firewall
=====================================================

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	list network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'ACCEPT'

config zone
	option name 'wan'
	list network 'wan'
	list network 'wan6'
	list network 'wwan'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'
	option input 'DROP'

config forwarding
	option src 'lan'
	option dest 'wan'
	option enabled '1'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-IGMP'
	option src 'wan'
	option proto 'igmp'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-MLD'
	option src 'wan'
	option proto 'icmp'
	option src_ip 'fe80::/10'
	list icmp_type '130/0'
	list icmp_type '131/0'
	list icmp_type '132/0'
	list icmp_type '143/0'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-IPSec-ESP'
	option src 'wan'
	option dest 'lan'
	option proto 'esp'
	option target 'ACCEPT'

config rule
	option name 'Allow-ISAKMP'
	option src 'wan'
	option dest 'lan'
	option dest_port '500'
	option proto 'udp'
	option target 'ACCEPT'

config rule
	option name 'Support-UDP-Traceroute'
	option src 'wan'
	option dest_port '33434:33689'
	option proto 'udp'
	option family 'ipv4'
	option target 'REJECT'
	option enabled 'false'

config include
	option path '/etc/firewall.user'

config include 'nat6'
	option path '/etc/firewall.nat6'
	option reload '1'

config include 'vpn_server_policy'
	option type 'script'
	option path '/etc/firewall.vpn_server_policy.sh'
	option reload '1'
	option enabled '1'

config rule 'process_mark'
	option name 'process_mark'
	option dest '*'
	option proto 'all'
	option extra '-m owner --gid-owner 65533'
	option target 'MARK'
	option set_xmark '0x8000/0xc000'

config rule 'wan_in_conn_mark'
	option name 'wan_in_conn_mark'
	option src 'wan'
	option dest '*'
	option target 'MARK'
	option enabled '0'
	option set_xmark '0x8000/0xc000'
	option extra '-m mark --mark 0x0/0x3f00 -j CONNMARK --set-xmark 0x8000/0xc000'

config rule 'lan_in_conn_mark_restore'
	option name 'lan_in_conn_mark_restore'
	option src 'lan'
	option dest '*'
	option target 'MARK'
	option enabled '0'
	option set_xmark '0x8000/0xc000'
	option extra '-m connmark --mark 0x8000/0xc000 -j CONNMARK --restore-mark --nfmask 0xc000 --ctmask 0xc000'

config rule 'out_conn_mark_restore'
	option name 'out_conn_mark_restore'
	option dest '*'
	option target 'MARK'
	option enabled '0'
	option set_xmark '0x8000/0xc000'
	option extra '-m connmark --mark 0x8000/0xc000 -j CONNMARK --restore-mark --nfmask 0xc000 --ctmask 0xc000'

config include 'swap_wan_in_conn_mark'
	option type 'script'
	option reload '1'
	option enabled '0'
	option path '/etc/firewall.swap_wan_in_conn_mark.sh'

config rule 'sambasharewan'
	option src 'wan'
	option dest_port '137 138 139 445'
	option dest_proto 'tcpudp'
	option target 'DROP'

config rule 'sambasharelan'
	option src 'lan'
	option dest_port '137 138 139 445'
	option dest_proto 'tcpudp'
	option target 'ACCEPT'

config rule 'glnas_ser'
	option src 'wan'
	option dest_port '6000-6002'
	option dest_proto 'tcp'
	option target 'DROP'

config rule 'webdav_wan'
	option src 'wan'
	option dest_port '6008'
	option dest_proto 'tcp'
	option target 'DROP'

config rule 'process_mark_dns'
	option name 'process_mark_dns'
	option dest '*'
	option proto 'all'
	option extra '-m owner --gid-owner 453'
	option target 'MARK'
	option set_xmark '0x8000/0xc000'

config rule 'process_explict_vpn'
	option name 'process_explict_vpn'
	option dest '*'
	option proto 'all'
	option extra '-m owner --gid-owner 20000'
	option target 'MARK'
	option set_xmark '0x20000/0x20000'

config rule 'process_mark_stubby'
	option name 'process_mark_stubby'
	option dest '*'
	option proto 'all'
	option extra '-m owner --gid-owner 410'
	option target 'MARK'
	option set_xmark '0x8000/0xc000'

config redirect 'adguard_home'
	option name 'Adguard Home'
	option src 'lan'
	option src_dport '53'
	option dest 'lan'
	option dest_port '3053'
	option proto 'tcp udp'
	option enabled '1'

config redirect 'dns_vpn'
	option name 'dns for vpn'
	option src 'lan'
	option src_dport '53'
	option dest 'lan'
	option dest_port '1653'
	option mark '!0x8000/0xc000'
	list proto 'tcp'
	list proto 'udp'
	option enabled '0'

config redirect 'dns_vpn_guest'
	option name 'dns for vpn guest'
	option src 'guest'
	option src_dport '53'
	option dest 'guest'
	option dest_port '1653'
	option mark '!0x8000/0xc000'
	list proto 'tcp'
	list proto 'udp'
	option enabled '0'

config include 'vpn_client_deal_leak'
	option type 'script'
	option reload '1'
	option path '/etc/firewall.vpn_client_deal_leak.sh'
	option enabled '1'

config include 'glblock'
	option type 'script'
	option path '/usr/bin/gl_block.sh'
	option reload '1'

config redirect
	option proto 'tcp udp'
	option dest_ip '192.168.0.36'
	option src 'wan'
	option dest 'lan'
	option src_dport '5678'
	option dest_port '5678'
	option name 'GL-N8N WEBHOOKS (WAN)'
	option idx '2'
	option enabled '0'

config redirect
	option proto 'tcp udp'
	option src_dport '22'
	option dest_ip '192.168.0.36'
	option dest_port '22'
	option src 'wan'
	option dest 'lan'
	option name 'GL-SSH (WAN / LAN)'
	option idx '4'
	option enabled '1'
=====================================================

brada4 · December 3, 2024, 8:39am

You have to ask gl.inet for help with gl.inet firmware. Your device is supported here with future stable version 24.10 and current snapshots via firmware-selector.openwrt.org

NYCeyes · December 3, 2024, 8:57am

Ah, thank you for bringing that to my attention. I assumed it wouldn't be so upgradable only because I also own a NanoPi R6S, which is excellent, but has very different hardware, and has to wait for things to catch up to it.

But before I perform any O/S upgrade, I'm hoping that with the currently installed O/S version, a network tweak is possible to get the port forwarding to work. Reason: I have a separate AI project that time critically depends on it to move forward.

brada4 · December 3, 2024, 8:58am

Critically you are not using OpenWRT. What works and what does not is between you and commercial support channels of OpenWRT offshoots you are stubbornly using.

NYCeyes · December 3, 2024, 9:07am

Noted. I'll now wait for community members with network-centric suggestions. Thanks.

brada4 · December 3, 2024, 9:25am

Thats vendor extension, you have to deal with them. In general one adds point-to-point wireguard to keep badies out.

lleachii · December 3, 2024, 12:02pm

aside from the fact you have the wrong gateway IP - so traffc never leaves the LAN

The community is not best to ask GLInet-related questions. Your unfamiliar screenshot and inquiring about unknown configs reiterate that. I'm not sure if the other users clearly impressed that clearly, but we generally use this template-based reply:

It appears you are using firmware that is not from the official OpenWrt project.

When using forks/offshoots/vendor-specific builds that are "based on OpenWrt", there may be many differences compared to the official versions (hosted by OpenWrt.org). Some of these customizations may fundamentally change the way that OpenWrt works. You might need help from people with specific/specialized knowledge about the firmware you are using, so it is possible that advice you get here may not be useful.

You may find that the best options are:

Install an official version of OpenWrt, if your device is supported (see https://firmware-selector.openwrt.org).
Ask for help from the maintainer(s) or user community of the specific firmware that you are using.
Provide the source code for the firmware so that users on this forum can understand how your firmware works (OpenWrt forum users are volunteers, so somebody might look at the code if they have time and are interested in your issue).

If you believe that this specific issue is common to generic/official OpenWrt and/or the maintainers of your build have indicated as such, please feel free to clarify.

mk24 · December 3, 2024, 1:43pm

A few points of general advice:

Since you're using a single routing table, there can only be one option gateway in the whole configuration. It needs to be on the wan interface and point at the next router upstream toward the Internet.
If you can install routes in the first router, you can configure the network to not NAT in the second router. When the first router knows how to reach the second LAN network (by forwarding to the second router's wan IP on it's lan) the first router can reverse NAT ("port-forward") directly to the IP of the server machine. This is still subject to the firewall of the second router allowing the packet to continue to the inner LAN, but that will be a simple traffic rule (or even a general forward the whole zone if you don't need security).
Running tcpdump on OpenWrt is useful for troubleshooting. First look for packets to arrive on wan, then check that they leave lan, and that the server answers them and the answer is forwarded back the other way.

NYCeyes · December 3, 2024, 2:09pm

@lleachii

Thank you.

Setting aside vendor differences for a moment, and honing in on your point about having the wrong gateway IP:

Given the following output from an SSH session within the OpenWRT router (with the MAC-Addresses redacted, but nevertheless where br-lan and eth1 share the same MAC-Address), what do you propose should be the correct gateway IP?

Do you mean it's problematic for my port-forwarding use-case, or in general it's incorrect? Currently, 50+ of my LAN-side devices (including the very laptop I'm typing this message on @ 192.168.0.31) have been successfully communicating LAN to WAN for years?

Thank you!

=============================
Router LAN-Port-IP (br-lan & eth1): 192.168.0.1
Router WAN-Port-IP (eth0):          192.168.1.2
=============================

root@GL-MT2500# ifconfig -a
br-lan    Link encap:Ethernet  HWaddr 11:11:11:11:11:11
          inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:456338 errors:0 dropped:36381 overruns:0 frame:0
          TX packets:360420 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:86973723 (82.9 MiB)  TX bytes:140168700 (133.6 MiB)

eth0      Link encap:Ethernet  HWaddr 22:22:22:22:22:22
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2343252 errors:0 dropped:0 overruns:0 frame:0
          TX packets:725172 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2926434356 (2.7 GiB)  TX bytes:173472373 (165.4 MiB)
          Interrupt:75 

eth1      Link encap:Ethernet  HWaddr 11:11:11:11:11:11
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:751495 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2313001 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:180023129 (171.6 MiB)  TX bytes:2937193564 (2.7 GiB)
          Interrupt:75 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:88 errors:0 dropped:0 overruns:0 frame:0
          TX packets:88 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:11691 (11.4 KiB)  TX bytes:11691 (11.4 KiB)

root@GL-MT2500# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG        0 0          0 eth0
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 br-lan
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth0
=============================

NYCeyes · December 3, 2024, 2:11pm

Hi @mk24 I'm reading and digesting your reply now. Thank you for it.

I used to be amazing with tcpdump(1), ethtool(1), etc when I worked at core of a stock exchange (ECN) years back; but alas, I forgot it all as I pivoted to other technologies.

NYCeyes · December 3, 2024, 2:29pm

Given the topology described in the OP, as well as the provided command outputs, do the values in these forms look correct (which I've used for a few years without issue. It's just the present port-forwarding requirement, which I didn't expect such headaches from. LoL):

LAN Configuration:

WAN Configuration:

lleachii · December 3, 2024, 2:50pm

OK, it seems you overlooked a basic understanding of networking. I was simply highlighting that it's impossible for 192.168.1.2 to be the gateway for 192.168.0.0/24.

Although, you screenshot and response shows WAN. I'm not sure why you have both a LAN and WAN gateway configuration.

So to answer you - I would guess a valid IP of your upstream router in the 192.168.0.x range.

I merely saw a reason your connection would hang having made a NAT on a router that has an incorrect gateway configuration. A lot of your further discussion seems to again be seeking help in the wrong form. Sorry. I'm sure you're just as frustrated as we are, especially since we don't have any direct relationship to the firmware you're seeking assistance with.

NYCeyes · December 3, 2024, 3:26pm

I do understand networking. And the point you made gnawed at me a few years ago also (i.e., that they are on two different subnets). So, yes, I agree with you in networking principle.

However, on the 192.168.0.x LAN-side network, there are only clients (laptops, phones, etc; but no other routers), and those clients all point to eth1@192.168.0.1 as their default router, which works.

Now, once said client traffic reaches and enters that OpenWRT router's eth1@192.168.0.1 LAN-side-port, it's configuration (see above image fillout-forms) specifies that traffic now gets to the WAN via its IPv4 gateway @ 192.168.1.2 (which is the IP-Address of the WAN-side-OpenWRT-router-port). Indeed, that was initially odd for me to specify when I first set this up long back (being on different "subnets"); however, I knew that the LAN-to-WAN NAT translation would handle that, and it successfully has.

I've done all of this cascading before; I've simply never tried to port-forward through a cascade (until now).

PS: I'm not frustrated. We're all just having a constructive dialogue.

lleachii · December 3, 2024, 3:34pm

In OpenWrt it's generally invalid to make a gateway config on the LAN side, so I'm not sure how constructive this dialog is. Maybe it's valid at GL.INET.

All I know is that since you want a NAT to use an invalid gateway, that explains why your Port Forward doesn't work.

You admit yourself, there's no router on the LAN side, only clients.

NYCeyes · December 3, 2024, 3:58pm

We disagree. I'll work with @mk24

lleachii · December 3, 2024, 4:05pm

No worries, cool - good luck!

Additionally, mk24 explained in the last point of his post that the LAN side NATed IP will incorrectly use the invalid LAN gateway config, hence your normal traffic works, but your port forward doesn't - which uses the invalid gateway. That can be verified using the tools. It's not clear what further assistance you working with him on, especially since this is the OpenWrt forum, and you seem to have GL.INET-specific configs, or insist that it's valid (not sure which).

brada4 · December 3, 2024, 6:06pm

@NYCeyes pardon me but you have not yet applied a single corrective action. You know you broke it and you have to fix it. No other way.

Please contact gl.inet with regatd to proper configurations.
Your double gateways and extra attributes to firewall rules make them unsupportable here.