Port forwarding question while gateway is not the router

Installing and Using OpenWrt Network and Wireless Configuration
1

Hello,
I have a Mac Mini in my LAN which runs a SOCKS5 proxy software. What does this software do is most likely a OpenVPN client, that can decide which packages use proxy while others are not.

And for some reason I set the Mac Mini as my LAN's gateway in dnsmasq like "dhcp-option=3,10.0.0.250". Everything just works fine while I found the port forwarding to other devices on my router doesn't work any more. Any connecting from outside will be timeout.

The ping from WAN to the router is fine and so did the other devices to the router.
I have no idea what's happening so I decide to ask for some help.

2

I would need to see the complete picture, but I guess that the main router is forwarding incoming connections directly to the devices, but you told those devices to send all outgoinfñg traffic via the Mac Mini, which has no knowledge of such connections... see the issue here?

Besides, the Mac Mini redirects all output traffic through the VPN, and that is not where the connection comes from.

3

image
image1472×978 59.1 KB

Here's the picture.

Mac Mini will only forward the packages in special IP range to the VPN server, and none of local addresses are included, so it should forward the packages from the deivce directly to the router.
And I cannot see any request logs on the device in LAN, so I think the packages might be lost on the way to the device.

4

I would use "tcpdump" to see where are the packets being lost.

1 Like
5
  • Your router is no longer the gateway for clients, the Mac is
  • You are sending Internet traffic received on your WAN port to your client
  • Your client sends the reply traffic to the Mac (its gateway)
  • The Mac and router are in the same subnet
  • If you have properly enabled routing on the Mac, did you also enable inter-zone forwarding?
  • It's also possible the port is changing because you're passing the NATed traffic to another router
6

I have another device which can access the WAN port and I ran some tests with it.
I created a PHP web server on my Mac(a device, not the Mac Mini), and I use the tools named "tcpdump" to catch the packages on the port of 1233 on it. The logs looks quite normal, and there is no logs on the PHP web server.

xiaolin@XiaoLins-Mac:~|⇒  sudo tcpdump port 1233 -i en0 -vv
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
08:33:28.972170 IP (tos 0x0, ttl 63, id 56595, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.0.1.40158 > xiaolins-mac.home.univ-appserver: Flags [S], cksum 0xde7f (correct), seq 1709956178, win 29200, options [mss 1460,sackOK,TS val 2460314733 ecr 0,nop,wscale 7], length 0
08:33:28.972233 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->2649)!)
    xiaolins-mac.home.univ-appserver > 10.1.0.1.40158: Flags [S.], cksum 0x14a2 (incorrect -> 0x05e4), seq 1241448826, ack 1709956179, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 156230863 ecr 2460314733,sackOK,eol], length 0
08:33:28.972531 IP (tos 0x0, ttl 254, id 55301, offset 0, flags [none], proto TCP (6), length 40)
    10.1.0.1.40158 > xiaolins-mac.home.univ-appserver: Flags [R.], cksum 0x6b78 (correct), seq 1, ack 1, win 128, length 0
08:33:29.995839 IP (tos 0x0, ttl 63, id 56596, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.0.1.40158 > xiaolins-mac.home.univ-appserver: Flags [S], cksum 0xda7f (correct), seq 1709956178, win 29200, options [mss 1460,sackOK,TS val 2460315757 ecr 0,nop,wscale 7], length 0
08:33:29.995907 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64, bad cksum 0 (->2649)!)
    xiaolins-mac.home.univ-appserver > 10.1.0.1.40158: Flags [S.], cksum 0x14a2 (incorrect -> 0x9f35), seq 3265390521, ack 1709956179, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 156231835 ecr 2460315757,sackOK,eol], length 0
08:33:29.996214 IP (tos 0x0, ttl 254, id 55499, offset 0, flags [none], proto TCP (6), length 40)
    10.1.0.1.40158 > xiaolins-mac.home.univ-appserver: Flags [R.], cksum 0x0c96 (correct), seq 1, ack 1, win 128, length 0
  • xiaolins-mac.home is the device in the LAN and 10.1.0.1 is another device in the WAN

On my Mac Mini, I can only find some connection refuesd error like below, which I can recurrent easily by curl 10.1.0.1:40158

2020-02-19 07:39:35.498656 <WARNING> [SGDirectConnector-288112] Connection setup failed with error: Connection refused, 10.1.0.1:44881
2020-02-19 07:39:35.501704 <WARNING> [SGDirectConnector-288113] Connection setup failed with error: Connection refused, 10.1.0.1:44881
2020-02-19 07:39:36.273433 <WARNING> [SGDirectConnector-288114] Connection setup failed with error: Connection refused, 10.1.0.1:44881
2020-02-19 07:39:36.277306 <WARNING> [SGDirectConnector-288115] Connection setup failed with error: Connection refused, 10.1.0.1:40158
2020-02-19 07:39:36.280694 <WARNING> [SGDirectConnector-288116] Connection setup failed with error: Connection refused, 10.1.0.1:40158
2020-02-19 07:43:14.979012 <WARNING> [SGDirectConnector-288158] Connection setup failed with error: Connection refused, 10.1.0.1:40158
7

If you have properly enabled routing on the Mac, did you also enable inter-zone forwarding?

The Mac Mini is configured that all local packages should be sent directly to its gateway(for Mac Mini it's 10.0.0.1), and I can only see some connection refuesd error on the Mac Mini, which I can easily recurrent by curl the address mentions in the log of tcpdump.

9
  • Does it work if the port forwarded host has its normal gateway as the router?
  • If your Mac is just a proxy server, why did you change the LAN clients' gateways to the Mac?
10

I'm convinced this setup is never going to work, or is going to need such a convoluted configuration that is not going to be worth the effort. However, I would love to be proved wrong, so I remain expectant.

11
  • The port forwarding is working correctly until I change the gateway to Mac.
  • Beacuse, you know, in China there is a national firewall that prevnet people in China to access sites like Google or Twitter. The only way to break it is to forward the package to a foreign server and that's what exactly the Mac Mini does. It makes the blocked sites accessable.
12

Yes, correct. It's a proxy, not a router. You don't need the gateway setting. You're only sending TCP and/or HTTP traffic to the Mac's port.

2 Likes
13

If you want to sent the mac mini up as a router it could work, but you have to understand what a router's job is, and then tell it to do that job.

In this case packets come WAN Router -> LAN machine and then the LAN machine wants to reply, but it only knows to send LAN machine -> Mac Mini. Now the Mac Mini needs to realize that these packets aren't for its VPN and send them Mac Mini -> WAN Router.

How it is supposed to know which things go over the VPN vs which go back to LAN is up to you to decide. For example it could know which ports are forwarded, and mark the packets in the firewall.

Possibly an easier solution is to have the WAN router machine as the default gateway, and it looks for outgoing stuff on port 80 or 443 and sends it to the Mac Mini for encapsulation into the VPN, otherwise it sends everything else out to WAN

14

As I configured all IP that belongs to LAN should send to WAN Router directly but it just doesn't work. It might be something wrong with the proxy software and I will check that later.

BTW please allow me to comfirm the forwarding chain of the package comes from WAN that the whole stuff should behave:
WAN client -> Route -> Device -> Mac Mini -> Route -> WAN client

15

When a device on your LAN is acting as a server, it sends packets to whoever sent to you so the return address is some internet device not some LAN device. Hence, it needs to know to send return packets to WAN even though they are headed to the internet.

16

In my opinion, when I sending a package to a server, the package headers should contain a destination. When the package reach the route, which is connected to both WAN and LAN, the package destination would change to the LAN device by the route. Then the device handled the package and then return the package to the address that it came from via the gateway(normally it should be the route but in this case it's the Mac Mini) and then the gateway sent it back to the device in the Internet.

So I think the role Mac Mini plays is most likely a "switch" beacuse it only proxy package when the destination is a remote IP address and for local IP it just send the packages to the route via the gateway(Mac Mini's gateway is the route).

it sends packets to whoever sent to you so the return

Whoever sent the package to the device in LAN is the route
Whoever sent the package to the Mac Mini is the device
So you means that the "destination" of the package which the device sent to the Mac is the device? And the package just trap into a loop like the figure below? I'm a little confused.

image
image1942×1484 220 KB

It's all my guesses and I would love to be proved wrong.

Thanks a lot!

17

Yes, I see you are confused about how routing works. A router does not change the destination address of a packet unless it is doing DNAT (otherwise how would we figure out where it was supposed to go?).

Incoming packet goes like this: Internet -> Router (DNAT) -> LAN device

So during DNAT here it changes the destination to the internal IP of the LAN device so that the internal LAN device knows to receive that packet.

Now outgoing reply packet, how it should work if you want to keep things the way they are:

LAN device -> Mac Mini Gateway (looks at packet and chooses WAN) -> WAN gateway (Undo DNAT) -> Internet

In the Undo DNAT step, it changes the source to be the public IP of the WAN gateway so that the packet appears to come from the WAN router.

The parts you need to figure out are the (looks at packet and chooses WAN) step. This step is called "policy routing" where you route the destination based on some policy. For example if your LAN device operates a TCP based server on port 1234, then your policy on Mac Mini could be "any packet from LAN device with ip address xxxxx that's a TCP packet from port 1234 goes to the WAN"

Such rules can be set up in static routing section.

for example, using the full ip package, you can set up a rule like:

ip rule add from 192.168.x.x to 0.0.0.0/0 ipproto tcp sport 1234 priority 10 table 100
ip route add default via 192.168.x.1 table 100

So, now there's a special table numbered 100 for looking up routes, and a special rule that selects table 100 when the packet matches the policy.

1 Like
18

Now in my understanding everything is probably fine except the Mac based on what you said.The rule I set up in proxy settings looks like this:

image
image710×240 59.2 KB

So could I draw the conclusion that the proxy software is to blame for the fault of port forwarding?

19

Correct, that rules says "anything that needs to go to a LAN device, send it direct", but the packet from your LAN server is going to the internet (back to whoever sent it to your WAN router), so that rule doesn't apply.

In any case, LAN devices won't send packets to the Mac MINI if they need to talk to each other, so that rule is unlikely to be involved ever.

What software are you running on this Mac Mini? It's not OpenWrt I guess?

Here is my suggestion. If all you need is to access HTTP/HTTPS sites through the mac mini... instead of using it as a router use it as a directly configured PROXY. Basically this means your LAN devices you tell programs that need access to HTTP to use the mac as a proxy, but you keep your WAN router as the default gateway.

A proxy is a little like an old-school telephone operator. You "pick up the phone" and "ask the operator to connect you" to the remote website. So your LAN computer programs can have a proxy configured, and then the web browser will do this dance where it connects to the proxy and asks it to connect to the remote world for it. That is entirely different from Routing

20

I don't quite understand. Now the gateway of devices in LAN is all the Mac, so when they wants to communicate with each other, shouldn't the packages pass through the gateway?

My route is running OpenWrt but not the Mac, hope them are not crossing a certain rule of the forum (lol

As you said here, destination of packages should still the route(private IP) until they reached and handled by the route, not the public one. But what you said here:

is not quite same as the last reply(in my understanding).

What I may not have emphasized is the route is the gateway of the mac, and mac is the gateway of other devices.

Thanks again for your patience :smile:

21

No, they will only send packets to the gateway if the destinations they want to reach are not on the local network. If they can send a packet and have it received directly by the other machine they will do that.

So if your subnet is say 192.168.1.0/24 then any device on your LAN trying to talk to say 192.168.1.33 will simply send a packet direct to 192.168.1.33 with no machine in the middle.

2 Likes