Port forward not working

Fidelix · September 23, 2024, 10:25am

I want to be able to access these 2 devices on the internal network from tailscale. Doesn't work:

Interfaces:

Port forwards:

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'modem_fiber'
        option src 'tailscale'
        option src_dport '9001'
        option dest_ip '192.168.3.1'
        option dest_port '8088'

config redirect
        option target 'DNAT'
        option name 'modem_4g'
        option src_dport '9002'
        option dest_ip '192.168.0.254'
        option dest_port '80'
        list proto 'tcp'
        option src 'tailscale'
        option dest 'wan'

tcpdump:

tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
07:19:01.690344 tailscale0 In  ifindex 16 ethertype IPv4 (0x0800), length 72: (tos 0x0, ttl 128, id 41096, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.55289 > 10.1.5.1.9002: Flags [S], cksum 0x6d0b (correct), seq 426809005, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
07:19:01.962173 tailscale0 In  ifindex 16 ethertype IPv4 (0x0800), length 72: (tos 0x0, ttl 128, id 41099, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.55290 > 10.1.5.1.9002: Flags [S], cksum 0xcb74 (correct), seq 2807409245, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
07:19:06.626692 tailscale0 In  ifindex 16 ethertype IPv4 (0x0800), length 72: (tos 0x0, ttl 128, id 41109, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.55291 > 10.1.5.1.9002: Flags [S], cksum 0xd3e0 (correct), seq 1252064933, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0

tailscale has: Allow forward to destination zones : wan and lan

brada4 · September 23, 2024, 10:42am

Add option log '1' to rule(s)

trendy · September 23, 2024, 10:48am

If I understand correctly you want to access the web interface of the ISP wan router and the 4G modem from Tailscale network. And for that you are trying to open the lan IP of OpenWrt on a random port.

In the first redirect the dest zone is wrong, you are using lan instead of wan or whatever zone the ISP router is.
The second redirect looks correct. Expand the tcpdump to all interfaces ( -i any ) to verify that the forwarded packet has correct headers. (use filter tcp port 80 or tcp port 9002 )

Fidelix · September 23, 2024, 10:52am

Mon Sep 23 07:51:01 2024 daemon.err tailscaled[4456]: 2024/09/23 10:51:01 Accept: TCP{10.1.5.20:554 > 100.107.169.12:61968} 1280 ok out
Mon Sep 23 07:51:11 2024 daemon.err tailscaled[4456]: 2024/09/23 10:51:11 Accept: TCP{10.1.5.20:554 > 100.107.169.12:61968} 1280 ok out
Mon Sep 23 07:51:16 2024 kern.warn kernel: [398344.440539] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=10.1.5.1 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=45320 DF PROTO=TCP SPT=55692 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 07:51:17 2024 kern.warn kernel: [398344.695483] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=10.1.5.1 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=45322 DF PROTO=TCP SPT=55693 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 07:51:21 2024 daemon.err tailscaled[4456]: 2024/09/23 10:51:21 Accept: ICMPv4{100.114.203.45:0 > 100.94.38.57:0} 88 ok out
Mon Sep 23 07:51:31 2024 daemon.err tailscaled[4456]: 2024/09/23 10:51:31 Accept: TCP{10.1.5.20:554 > 100.107.169.12:64580} 1280 ok out
Mon Sep 23 07:51:41 2024 daemon.err tailscaled[4456]: 2024/09/23 10:51:41 Accept: ICMPv4{100.114.203.45:0 > 100.94.38.57:0} 88 ok out
Mon Sep 23 07:51:51 2024 daemon.err tailscaled[4456]: 2024/09/23 10:51:51 Accept: ICMPv4{100.114.203.45:0 > 100.94.38.57:0} 88 ok out
Mon Sep 23 07:51:53 2024 kern.warn kernel: [398381.491878] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=10.1.5.1 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=45459 DF PROTO=TCP SPT=55697 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 07:51:54 2024 kern.warn kernel: [398381.755549] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=10.1.5.1 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=45460 DF PROTO=TCP SPT=55698 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 07:51:59 2024 kern.warn kernel: [398387.398365] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=10.1.5.1 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=45471 DF PROTO=TCP SPT=55709 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 07:52:01 2024 daemon.err tailscaled[4456]: 2024/09/23 10:52:01 Accept: ICMPv4{100.114.203.45:0 > 100.94.38.57:0} 88 ok out

Fidelix · September 23, 2024, 10:57am

I tried it with the other zone as well. I was experimenting. Changing the zone did not fix it.

Previous command already had that, and port 9002 as well. I can't add port 80 because I would not be able to read anything, the network has a ton of constant traffic in that port.

trendy · September 23, 2024, 11:20am

Can you try to send the packets to the tailscale IP on OpenWrt and not to lan?

Fidelix · September 23, 2024, 11:51am

The result is the same.

Mon Sep 23 08:50:43 2024 kern.warn kernel: [401910.468501] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=100.114.203.45 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=46620 DF PROTO=TCP SPT=65386 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 08:50:46 2024 kern.warn kernel: [401913.771548] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=100.114.203.45 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=57821 DF PROTO=TCP SPT=65402 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00
Mon Sep 23 08:50:48 2024 kern.warn kernel: [401915.946133] modem_4g: IN=tailscale0 OUT= MAC= SRC=100.103.232.85 DST=100.114.203.45 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=40650 DF PROTO=TCP SPT=65406 DPT=9002 WINDOW=64480 RES=0x00 SYN URGP=0 MARK=0x3f00

trendy · September 23, 2024, 11:58am

Let's check if everything is in order.

Please run the following commands (copy-paste the whole block) and paste the output here, using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have

ubus call system board; \
uci export network; uci export firewall; \
nft list ruleset

Then run the tcpdump as following:
tcpdump -i any -vn '(host 192.168.0.254 and port 80) or ( host 192.168.3.1 and port 8088) or port 9002 or port 9001'

Fidelix · September 23, 2024, 1:42pm

https://dpaste.org/YY5M2

Too much data to paste here.

brada4 · September 23, 2024, 1:52pm

{
"kernel": "6.1.63",

It appears you are using firmware that is not from the official OpenWrt project.

When using forks/offshoots/vendor-specific builds that are "based on OpenWrt", there may be many differences compared to the official versions (hosted by OpenWrt.org). Some of these customizations may fundamentally change the way that OpenWrt works. You might need help from people with specific/specialized knowledge about the firmware you are using, so it is possible that advice you get here may not be useful.

You may find that the best options are:

Install an official version of OpenWrt, if your device is supported (see https://firmware-selector.openwrt.org).
Ask for help from the maintainer(s) or user community of the specific firmware that you are using.
Provide the source code for the firmware so that users on this forum can understand how your firmware works (OpenWrt forum users are volunteers, so somebody might look at the code if they have time and are interested in your issue).

If you believe that this specific issue is common to generic/official OpenWrt and/or the maintainers of your build have indicated as such, please feel free to clarify.

Fidelix · September 23, 2024, 2:00pm

I appreciate the comment, brada4.

But it is highly unlikely that's the issue here. Friendlyelec's build is basically identical to upstream except they add some patches to their hardware and some preinstalled packages.

egc · September 23, 2024, 2:19pm

Perhaps but the problem is we do not know what has been modified so it is very difficult to help as we do not know the software you are using

One tip, you appear to have multiple wan interfaces using metrics, things do not work if traffic goes in via one interface but goes out via another.

brada4 · September 23, 2024, 2:21pm

Was it openwrt src+dst combination should have been backed by route and the src_daddr but since local IP is presumed in absence of src_daddr certainly does not traverse route to reach the rule past forward/accept as in your log.
very friendly wrt may behave very differently in regard.

Fidelix · September 23, 2024, 2:33pm

Very easy to check. I've checked it myself.
The changes that FriendlyElec made are not at all relevant to this discussion as far as I could find. Only 60 commits of difference, mostly related to hardware and default presets.

Only a few of exceptions on NAT Fullcone support (not enabled) improvements and flow offloading.

Does anyone have a functional port forward of this type on a tailscale network on upstream OpenWRT?

brada4 · September 23, 2024, 2:35pm

Remove "dest wan" line.

Fidelix · September 23, 2024, 2:43pm

This is what it looks like atm:

config redirect
        option target 'DNAT'
        option name 'modem_4g'
        option src_dport '9002'
        option dest_ip '192.168.0.254'
        option dest_port '80'
        option src 'tailscale'
        list proto 'tcp'

Result:

tcpdump -i any -vn '(host 192.168.0.254 and port 80) or ( host 192.168.3.1 and port 8088) or port 9002 or port 9001'
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

11:41:52.748208 tailscale0 In  IP (tos 0x0, ttl 128, id 25890, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58438 > 10.1.5.1.9002: Flags [S], cksum 0xf65f (correct), seq 568981650, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
11:41:52.748474 br-lan Out IP (tos 0x0, ttl 127, id 25890, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58438 > 192.168.0.254.80: Flags [S], cksum 0x6695 (correct), seq 568981650, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
11:41:52.748497 eth1  Out IP (tos 0x0, ttl 127, id 25890, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58438 > 192.168.0.254.80: Flags [S], cksum 0x6695 (correct), seq 568981650, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
11:41:53.001284 tailscale0 In  IP (tos 0x0, ttl 128, id 25894, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58439 > 10.1.5.1.9002: Flags [S], cksum 0xf9f6 (correct), seq 3108068770, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
11:41:53.001543 br-lan Out IP (tos 0x0, ttl 127, id 25894, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58439 > 192.168.0.254.80: Flags [S], cksum 0x6a2c (correct), seq 3108068770, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
11:41:53.001565 eth1  Out IP (tos 0x0, ttl 127, id 25894, offset 0, flags [DF], proto TCP (6), length 52)

trendy · September 23, 2024, 2:57pm

One more thing:
The packet is not masqueraded and you send to the ISP router a packet from its lan interface with an IP that belongs to its wan. It will be either discarded or sent to the ISP router wan interface, where it will be discarded.
You have the masquerading enabled in the firewall config, but it's not visible in the nft.

Fidelix · September 23, 2024, 3:05pm

Nice catch, trendy.

So I changed the network of the interface lan_temp to something completely different. It was conflicting because it was in the same subnet.
Now:

root@router-5:~# tcpdump -i any -vn '(host 192.168.0.254 and port 80) or ( host 192.168.3.1 and port 8088) or port 9002 or port 9001'
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
12:03:26.398348 tailscale0 In  IP (tos 0x0, ttl 128, id 27464, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58975 > 10.1.5.1.9002: Flags [S], cksum 0x5d54 (correct), seq 499422122, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
12:03:26.398624 veth2 Out IP (tos 0x0, ttl 127, id 27464, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.1.58975 > 192.168.0.254.80: Flags [S], cksum 0x599d (correct), seq 499422122, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
12:03:26.398637 eth0  Out IP (tos 0x0, ttl 127, id 27464, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.1.58975 > 192.168.0.254.80: Flags [S], cksum 0x599d (correct), seq 499422122, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
12:03:26.399469 eth0  P   IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:26.399478 veth2 In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:26.399681 veth2 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:26.399690 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:27.405365 tailscale0 In  IP (tos 0x0, ttl 128, id 27471, offset 0, flags [DF], proto TCP (6), length 52)
    100.103.232.85.58975 > 10.1.5.1.9002: Flags [S], cksum 0x5d54 (correct), seq 499422122, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
12:03:27.405507 veth2 Out IP (tos 0x0, ttl 127, id 27471, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.1.58975 > 192.168.0.254.80: Flags [S], cksum 0x599d (correct), seq 499422122, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
12:03:27.405516 eth0  Out IP (tos 0x0, ttl 127, id 27471, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.0.1.58975 > 192.168.0.254.80: Flags [S], cksum 0x599d (correct), seq 499422122, win 64480, options [mss 1240,nop,wscale 8,nop,nop,sackOK], length 0
12:03:27.406333 eth0  P   IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:27.406340 veth2 In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:27.406565 veth2 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:27.406575 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:30.402056 eth0  P   IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:30.402065 veth2 In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:30.402240 veth2 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:30.402248 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:36.403523 eth0  P   IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:36.403533 veth2 In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    192.168.0.254.80 > 192.168.0.1.58975: Flags [S.], cksum 0x3382 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:36.403741 veth2 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:36.403753 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0

So it's getting closer. Not sure why it's not working though.

trendy · September 23, 2024, 3:09pm

Because you are forwarding the response to the wrong interface.

12:03:26.399681 veth2 Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0
12:03:26.399690 eth0  Out IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 48)
    10.1.5.1.9002 > 100.103.232.85.58975: Flags [S.], cksum 0x3739 (correct), seq 3853203873, ack 499422123, win 5840, options [mss 1460,nop,wscale 1], length 0

Should go to tailscale, not eth0 and veth2.

Fidelix · September 23, 2024, 3:13pm

Indeed that seems wrong. How do I fix that?

I don't think I made any special config.
Wasn't the port forwarding supposed to just reflect it back?