Hi, I am debugging an issue which has gone beyond my experience level, any help appreciated, thanks.
tl;dr;
Look at this line from an iptables trace and note the interface vlan interface br-lan.99
source
May 10 18:39:54 turris kernel: [88215.170984] TRACE: raw:PREROUTING:rule:6 IN=br-lan.99 OUT= MAC=<mac> SRC=192.168.1.2 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=51679 DF PROTO=TCP SPT=50868 DPT=4567 SEQ=2655642190 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080ABD91B5470000000001030307)
Here's another one from a different client in the same subnet which is coming in via the br-lan
bridge itself not the vlan interface.
May 10 18:38:49 turris kernel: [88150.006717] TRACE: raw:PREROUTING:rule:6 IN=br-lan OUT= PHYSIN=lan2 MAC=<mac> SRC=192.168.1.138 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=5988 DF PROTO=TCP SPT=49846 DPT=4567 SEQ=1976936805 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AFB5B50DA0000000001030305)
I'm trying to figure out why this is happening.
Setup
I have 2 DSA openwrt devices.
Device 1 is a router
Device 2 is a dumb AP
They are linked with a VLAN trunk.
Server A is connected to an untagged port on Device 2.
Client A is connected to Device 1 untagged
Client B is connected to Device 2 untagged.
The untagged vlan is 99, this is my default lan subnet, so for the purposes of this post everything is using the same subnet.
I am running an http service on Server A (192.168.1.35) on port 4567. I'm using a basic echo server for testing purposes. I set up a port forward on Device 1 (192.168.1.1) 4567->192.168.1.35.
ββββββββββββββ ββββββββββββββββ
β β β β
β client B β β client A β
β β β β
ββββββ¬ββββββββ ββββββββ¬ββββββββ
β β
β β
β β
β β
β β
β β
βββββ΄ββββββββββ βββββββ΄βββββββββ
β β VLAN Trunk β β
β AP (2) ββββββββββββββββββββββββββββββββββββββββ€ Router (1) ββββββββββββport 4567
β β β β
βββββ¬ββββββββββ ββββββββββββββββ
β
β
β
β
βport 4567
βββββ΄βββββββ
β β
β β
β server A β
β β
β β
ββββββββββββ
The problem
I curl the echo service from client B using <wan ip>:4567
. This works.
I curl the echo service from client A using <wan ip>:4567
. This generates a connection refused error.
I want to be able to access the service on server A from a client connected to Device 1 using my public IP. This is simply so that I can use one url no matter if I am inside my house or out.
Debugging
I assumed the issue was firewall related so I added a TRACE rule
iptables -t raw -I PREROUTING -p tcp --destination 0.0.0.0/0 --dport 4567 -j TRACE
The difference between the curl from client A and B is interesting. This is what happens on the client that works:
May 10 18:39:54 turris kernel: [88215.170984] TRACE: raw:PREROUTING:rule:6 IN=br-lan.99 OUT= MAC=<mac> SRC=192.168.1.2 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=51679 DF PROTO=TCP SPT=50868 DPT=4567 SEQ=2655642190 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080ABD91B5470000000001030307)
#... 4 unrelated rules removed
May 10 18:39:54 turris kernel: [88215.308981] TRACE: nat:PREROUTING:rule:3 IN=br-lan.99 OUT= MAC=<mac> SRC=192.168.1.2 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=51679 DF PROTO=TCP SPT=50868 DPT=4567 SEQ=2655642190 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080ABD91B5470000000001030307)
#...
May 10 18:39:54 turris kernel: [88215.393192] TRACE: nat:zone_lan_prerouting:rule:10 IN=br-lan.99 OUT= MAC=<mac> SRC=192.168.1.2 DST=86.20.35.0 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=51679 DF PROTO=TCP SPT=50868 DPT=4567 SEQ=2655642190 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080ABD91B5470000000001030307)
#num target prot opt source destination
#10 DNAT tcp -- 192.168.1.0/24 <public ip> tcp dpt:4567 /* !fw3: WHOAMI TEST (reflection) */ to:192.168.1.35:4567
May 10 18:39:54 turris kernel: [88215.421654] TRACE: mangle:FORWARD:rule:3 IN=br-lan.99 OUT=br-lan.99 MAC=<mac> SRC=192.168.1.2 DST=192.168.1.35 LEN=60 TOS=0x00 PREC=0x00 TTL=62 ID=51679 DF PROTO=TCP SPT=50868 DPT=4567 SEQ=2655642190 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080ABD91B5470000000001030307)
And this is what happens for the client which disconnects
May 10 18:38:49 turris kernel: [88150.006717] TRACE: raw:PREROUTING:rule:6 IN=br-lan OUT= PHYSIN=lan2 MAC=<mac> SRC=192.168.1.138 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=5988 DF PROTO=TCP SPT=49846 DPT=4567 SEQ=1976936805 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AFB5B50DA0000000001030305)
#...same 4 unrelated rules
May 10 18:38:49 turris kernel: [88150.149078] TRACE: nat:PREROUTING:rule:8 IN=br-lan OUT= PHYSIN=lan2 MAC=<mac> SRC=192.168.1.138 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=5988 DF PROTO=TCP SPT=49846 DPT=4567 SEQ=1976936805 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AFB5B50DA0000000001030305)
#rule 8 is just the policy rule (ACCEPT) so no NAT happens in this case
May 10 18:38:49 turris kernel: [88150.177530] TRACE: filter:INPUT:rule:2 IN=br-lan.99 OUT= MAC=<mac> SRC=192.168.1.138 DST=<public ip> LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=5988 DF PROTO=TCP SPT=49846 DPT=4567 SEQ=1976936805 ACK=0 WINDOW=64240 RES=0x00 SYN URGP=0 OPT (020405B40402080AFB5B50DA0000000001030305)
So here are the things I don't understand:
- Why does the input arrive on the lan bridge
br-lan
for a client connected directly to the router? If the port is marked as untagged forbr-lan.99
I would expect the packet to arrive on the vlan interface before it gets to iptables. - If this is working as intended and I have simply misunderstood how VLANs work then why don't I get a NAT in the second case?
I feel I have debugged this setup as far as I can, I can't reconcile my understanding of how I think this works with what I'm actually seing, any help appreciated.