Nat issue, reply packets in txpdump, not natted to internal client

GrapeCent · January 23, 2023, 2:05am

I have an easy tether interface that has been working on another router for a while, but im seeing an issue with this on a new rpi openwrt setup.

I have an upstream router that connects to this rpi via ospf so the only masquerade that occurs is on the edge, this is a new config which i think is related.

Im only able to get out to the internet when i connect the openvpn interface. And with openvpn down, i can see reply packets on the easy tether interface with the interface's ip as the destination, so it appears to be a nat issue.

One test i should be able to do is ping from an upstream host to the easy tether router ip. I can do this from my main router, but not from a host. This suggests that there is an implicit rule to only masquerade LAN ip hosts, which is why my ospf routed network cant get out.

What do i need to do to allow all ips to masquerade? Enabling a specific nat rule with no specified source ip doesnt help, and setting up a normal ethernet connection on eth0 exposes the same problem.

ipv4     2 icmp     1 29 src=10.0.18.79 dst=192.168.117.1 type=8 code=0 id=8 packets=290 bytes=24360 src=192.168.117.1 dst=192.168.117.2 type=0 code=0 id=8 packets=290 bytes=24360 mark=16128 zone=0 use=2

Could it be that the ospf routes arent living in the right routing table??

ip route
0.0.0.0/1 via 10.8.0.1 dev tun0
default via 192.168.117.1 dev tap-easytether proto static src 192.168.117.2 metric 10
10.0.0.0/8 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
10.0.18.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
10.0.19.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
10.0.26.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
10.0.27.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
10.0.53.0/29 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
10.8.0.0/24 dev tun0 proto kernel scope link src 10.8.0.5
10.234.3.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
128.0.0.0/1 via 10.8.0.1 dev tun0
192.168.0.0/22 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
192.168.2.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20
192.168.8.0/29 dev dummy0 proto kernel scope link src 192.168.8.1
192.168.8.8/29 dev eth1 proto kernel scope link src 192.168.8.9
192.168.8.16/29 dev wlan0 proto kernel scope link src 192.168.8.17
192.168.117.0/24 dev tap-easytether proto static scope link metric 10
193.29.61.76 via 192.168.117.1 dev tap-easytether

GrapeCent · January 23, 2023, 2:59am

I am able to see the test pings appear in conntrack

conntrack -L -n|grep 192.168.117
icmp     1 8 src=10.0.18.79 dst=192.168.117.1 type=8 code=0 id=44960 packets=3 bytes=252 src=192.168.117.1 dst=192.168.117.2 type=0 code=0 id=44960 packets=3 bytes=252 mark=16128 use=1
conntrack v1.4.6 (conntrack-tools): 293 flow entries have been shown.
icmp     1 29 src=192.168.8.10 dst=192.168.117.1 type=8 code=0 id=7749 packets=4 bytes=336 src=192.168.117.1 dst=192.168.117.2 type=0 code=0 id=7749 packets=4 bytes=336 mark=16128 use=1

I'm just not seeing any traffic return to the upstream interface, noe the ping replies from 1.1.1.1 are going from the ovpn default gateway

tcpdump -i tap-easytether icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap-easytether, link-type EN10MB (Ethernet), capture size 262144 bytes
17:58:17.896979 IP 192.168.117.2 > 192.168.117.1: ICMP echo request, id 3, seq 176, length 64
17:58:17.898383 IP 192.168.117.1 > 192.168.117.2: ICMP echo reply, id 3, seq 176, length 64
17:58:18.920208 IP 192.168.117.2 > 192.168.117.1: ICMP echo request, id 3, seq 177, length 64
17:58:18.922297 IP 192.168.117.1 > 192.168.117.2: ICMP echo reply, id 3, seq 177, length 64
17:58:19.943637 IP 192.168.117.2 > 192.168.117.1: ICMP echo request, id 3, seq 178, length 64
17:58:19.945134 IP 192.168.117.1 > 192.168.117.2: ICMP echo reply, id 3, seq 178, length 64
^C
6 packets captured
8 packets received by filter
0 packets dropped by kernel
root@npancvpngw01:~# tcpdump -i eth1 icmp|grep reply
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
17:58:35.479321 IP one.one.one.one > 10.0.18.79: ICMP echo reply, id 45278, seq 3, length 64
17:58:36.429251 IP one.one.one.one > 10.0.18.79: ICMP echo reply, id 45278, seq 5, length 64

GrapeCent · January 23, 2023, 3:08am

I think I found the issue checking logs on reboot of the rpi router. it's an issue with mwan3


Sun Jan 22 18:07:03 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.0.18.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:03 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.0.19.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:03 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.0.26.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:03 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.0.27.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:04 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.0.53.0/29 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:04 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.234.3.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:04 2023 kern.warn kernel: [   63.668977] drop easytether in: IN=tap-easytether OUT= MAC=02:00:54:74:68:72:02:00:6d:73:62:62:08:00 SRC=1.1.1.1 DST=192.168.117.2 LEN=1453 TOS=0x00 PREC=0x00 TTL=128 ID=39582 PROTO=TCP SPT=443 DPT=51928 WINDOW=11680 RES=0x00 ACK PSH URGP=0 MARK=0x3f00
Sun Jan 22 18:07:04 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 10.0.0.0/8 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:04 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 192.168.0.0/22 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.
Sun Jan 22 18:07:04 2023 user.warn mwan3rtmon[1111]: failed: 'ip -4 route replace table 2 192.168.2.0/24 nhid 94 via 192.168.8.10 dev eth1 proto ospf metric 20 ' - error: Error: Nexthop specification and nexthop id are mutually exclusive.

damn, it's a bug and this patch doesn't seem to work for me. stopping mwan3 allows packets to route as expected, but I lose gateway tracking....

github.com/openwrt/packages

mwan3: mwan3rtmon unable to copy nhid route to target route table

opened 04:21PM - 07 Nov 22 UTC

kghost

Maintainer: @feckert Environment: OpenWrt 22.03.2, r19803-9a599fee93 Descri…ption: My tool is using nexthop to add routes, the route table looks like this: ``` root@openwrt:~# ip -6 ro default nhid 1018 via fe80::882c:c1ff:fe7a:ae94 dev tap0 proto ospf metric 20 pref medium ``` Basically, the route is managed by nexthop id, explained [here](https://lwn.net/Articles/763950/), the `via ... dev ...` part is managed by `nhid`. when mwan3rtmon encounter this route entry, it calls `ip route add ... table N` to copy it to the rt table, but it doesn't work. > Mon Nov 7 16:10:15 2022 user.warn mwan3rtmon[6169]: failed: 'ip -6 route replace table 4 default nhid 1018 via fe80::882c:c1ff:fe7a:ae94 dev tap0 proto ospf metric 20 pref medium' - error: Error: Nexthop specification and nexthop id are mutually exclusive. The the `nhid` part already refers the `via ... dev ...` part, such that they are duplicated. The benefit of using nexthop id is that we don't need to consistently monitor the route, my routing daemon will update nexthop information when nexthop changes, such that all routing entry refer to that nexthop will be updated.