Both CGNAT and "proper" IPv4 Internet on WAN; inbound TCP issues

Installing and Using OpenWrt
1

My ISP offers access to the IPv4 Legacy Internet via CGNAT. One can also pay them a few EUR a month extra to have a static, properly end-to-end routed IPv4 address on the public Internet assigned. Still, even when paying that extra buck, once can get the "free" CGNAT address assigned via DHCP, and access the Internet over the CGNAT gateway, which provides a little pseudonymity at least - which is why I would like to use that in addition to my static address for "outbound" IPv4 traffic from LAN clients.

With my setup (see below), there's an annoying problem with inbound TCP4 connections to services available via my static address which, I would like to understand better and solve.

Right now, I have my WAN link configured via pppoe, which yields the static, public IPv4 address on the link, and have another interface created that performs DHCP on the Ethernet interface for the CGNAT address (providing the default gateway), like this:

config interface 'wan'
        option device 'eth5'
        option proto 'pppoe'
        option username 'myusername@example.com'
        option password 'somepassword'
        option peerdns '0'
        option ipv6 '1'
        list dns '9.9.9.9'
        list dns '2620:fe::fe'
        option defaultroute '0'
 
config interface 'wancgnat'
        option proto 'dhcp'
        option device 'eth5'
        option peerdns '0'
        option hostname '*'
        option delegate '0'
        list dns '9.9.9.9'

... which results in an IPv4 address configuration like this:

7: eth5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc cake state UP qlen 1000
    inet 100.64.126.234/18 brd 100.64.127.255 scope global eth5
       valid_lft forever preferred_lft forever

18: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN qlen 3
    link/ppp
    inet 193.a.b.c peer 100.126.0.243/32 scope global pppoe-wan
       valid_lft forever preferred_lft forever

The IPv4 routing table looks like this:

# ip r
default via 100.64.64.1 dev eth5  src 100.64.126.234 
100.64.64.0/18 dev eth5 scope link  src 100.64.126.234 
100.126.0.243 dev pppoe-wan scope link  src 193.a.b.c
172.18.3.0/24 dev br-lan scope link  src 172.18.3.254

The resulting problem I experience is this: On my router, which performs NAT for the LAN on 193.a.b.c, I have several services exposed for IPv4 clients via port forwards. This generally works, but some hosts on some networks on the Internet cannot properly establish/maintain TCP conversations. On hosts affected by this problem, when I connect to a TCP port bound on 193.a.b.c, I see duplicate ACKs and spurious as well as actual retransmissions which I do not understand the reasons for.

What I would like to find is the cause of this issue and then the minimal change I have to implement to fix it.

Thanks to anyone who read this far, and for any resulting advice! :slight_smile:

2

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/dhcp
cat /etc/config/firewall
cat /etc/config/upnp
3

There's a lot in my current configuration (regarding IPv6 in particular) that would be a drag to redact and not useful to get to the bottom of the problem at hand. I tried to compile the relevant data above already, if you feel anything in particular is missing, please let me know. Regarding release/device info, here goes:

# ubus call system board
{
        "kernel": "5.15.162",
        "hostname": "lag17",
        "system": "Intel(R) Celeron(R) CPU 3865U @ 1.80GHz",
        "model": "Default string Default string",
        "board_name": "default-string-default-string",
        "rootfs_type": "squashfs",
        "release": {
                "distribution": "OpenWrt",
                "version": "23.05.4",
                "revision": "r24012-d8dd03c46f",
                "target": "x86/64",
                "description": "OpenWrt 23.05.4 r24012-d8dd03c46f"
        }
}
4

How do you acquire that open external port? natpmpc? upnpc?

5

With a redirect originally configured via luci, like the following:

config redirect
        option dest 'lan'
        option target 'DNAT'
        option name 'tcp-service-nas'
        option family 'ipv4'
        list proto 'tcp'
        option src 'wan'
        option src_dport '2879'
        option dest_ip '172.18.3.199'
        option dest_port '2879'
6

I have multi-wan setup on my network, one static ip and one CGNat on different interfaces, and have no isues so far.

Have you isolated the issue by not using the CGNat IP at all and using the that single Public static ip only for all traffic, does the issue persist?

If yes, do you have any rate limiting enabled? On my end I figured inbound connections were rate-limited by banIP and Firewall "SYN Flood protection", the latter usually shows in the logs, while the former just silently rate limits by default, which was a pain to diagnose...

7

If I do not configure and use the CGNAT IPv4 address at all, the problem does not manifest (i.e., all networks/hosts, not only most of them, can connect to TCP services available at the "proper" IPv4 address without hiccups).

The rate limiting hint is a very interesting one, but I am not aware of any IPv4-based limiting my ruleset should perform apart from the defaults... The SYN-flood limit seems high enough that I have a hard time believing it could be hit, but I will surely check (and report back)!

8

syn flood limit applies to input to host itself like luci or ssh.

1 Like