Incoming TCP MSS clamping and large packets in outgoing direction

openwrticon · September 30, 2024, 2:12pm

I have a measly ADSL 10Mbit/1Mbit. I modify incoming TCP sessions to reduce the tcp packets to 540 in order to reduce upload latency. iptables rules seem to be applied.

iptables -t mangle -A FORWARD -p tcp --tcp-flags FIN,SYN,RST SYN -i pppoe-wan -j TCPMSS --set-mss 540
iptables -t mangle -A FORWARD -p tcp --tcp-flags FIN,SYN,RST SYN -i pppoe-wan -j ACCEPT

(I see them on receiving devices with tcpdump)
pppoe-wan is the outgoing device and ifb4pppoe-wan is the incoming IFB.

However when I monitor outgoing packets with iptables -j LOG I see tcp packets LEN=1120 which implies that clamping is ignored.
Could clients ignore TCP MSS clamping and deliver big packets?
Any idea to troubleshoot it deeper?

brada4 · September 30, 2024, 4:04pm

Check this out:

Can you tell why are you stuck with fw3?
(fw3 version here https://github.com/openwrt/firewall3/pull/1)

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:

Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

openwrticon · September 30, 2024, 6:04pm

I am using Turris Omnia which uses older openwrt.

root@turris:~# ubus call system board
{
        "kernel": "5.15.148",
        "hostname": "turris",
        "system": "ARMv7 Processor rev 1 (v7l)",
        "model": "Turris Omnia",
        "board_name": "cznic,turris-omnia",
        "rootfs_type": "btrfs",
        "release": {
                "distribution": "TurrisOS",
                "version": "7.0.3",
                "revision": "r20343+127-4e1d1b7df0",
                "target": "mvebu/cortexa9",
                "description": "TurrisOS 7.0.3 4e1d1b7df0ce6fa96d7462dc883917682f428046"
        }
}

root@turris:~# cat /etc/config/network

config interface 'loopback'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option device 'lo'

config globals 'globals'
        option ula_prefix 'dsfsdfsdfsdfdsfsd'

config interface 'lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option _orig_ifname 'eth0 eth2 wlan0 wlan1'
        option _orig_bridge 'true'
        option delegate '0'
        option device 'br-lan'

config interface 'wan'
        option proto 'pppoe'
        option ipv6 '0'
        option username 'vxcvxcvxvx'
        option password 'xcvxcvxcvxcvxcvxc'
        #option pppd_options 'debug mtu 1452'
        option device 'eth2'
        option mtu '1452'

config interface 'guest_turris'
        option proto 'static'
        option bridge_empty '1'
        option enabled '1'
        option _orig_ifname 'guest_turris_0 guest_turris_0'
        option _orig_bridge 'true'
        option netmask '255.255.255.0'
        option ipaddr '192.168.3.1'
        option delegate '0'

config interface 'modem'
        option proto 'dhcp'
        option delegate '0'
        option hostname 'turris'
        option device 'eth2'

config device
        option name 'eth1'
        option mtu '9000'

config device
        option name 'lan0'
        option mtu '8000'

config device
        option name 'lan1'
        option mtu '8000'

config device
        option name 'lan2'
        option mtu '8000'

config device
        option name 'lan3'
        option mtu '8000'

config interface 'LANWIFI'
        option proto 'static'
        option ipaddr '192.168.4.1'
        option netmask '255.255.255.0'
        option enabled '1'
        option delegate '0'
        option device 'br-LANWIFI'

config device 'br_lan'
        option name 'br-lan'
        option stp '1'
        list ports 'lan0'
        list ports 'vethvm1'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        option type 'bridge'
        option mtu '8000'

config device 'br_LANWIFI'
        option name 'br-LANWIFI'
        option bridge_empty '1'
        option stp '1'
        option type 'bridge'

firewall rules are applied by SQM service.
I see in tcpdump of the clients that

tcpdump -i wlan0 -n -v "tcp[tcpflags] & (tcp-syn) != 0"

TCP sessions have the correct mss

20:54:20.312829 IP (tos 0x0, ttl 50, id 32678, offset 0, flags [DF], proto TCP (6), length 60)
    fghfghfghgfhgfhf > fhgfhfghfghfghfgh: Flags [S], cksum 0xd364 (correct), seq 1288761723, win 65535, options [mss 540,sackOK,TS val 3147656100 ecr 0,nop,wscale 10], length 0

brada4 · September 30, 2024, 6:28pm

You need mangle postrouting -o ppoe and mangle forward -i ppoe
and set small mtu (1280 for ip6 or 576 for ip4 and not less) on the interface as 20% +- of traffic us udp http/3.
Drop a line if you need exact commands.

Also check whether turris has fw patched or no

iptables-save | grep TCPMSS

(confirm here but ultimately turris has to fix fw3 if it is wrong)

moeller0 · September 30, 2024, 6:38pm

Turris is about to finally switch to fw4... (scheduled for the next minor point release) so maybe the best is to hobble along.. However here is what I used on my turris omnia with turrisOS successfully in the past as firewall custom rule (/etc/firewall.user):

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.


# special rules to allow MSS clamping for in and outbound traffic
# use ip6tables -t mangle -S ; iptables -t mangle -S to check
forced_MSS=1220 # 300, 536, 1280-60
# affects both down- and upstream, egress seems to require at least 216 on macos without setting a sysctl value
#iptables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom: Zone wan MTU fixing" -j TCPMSS --set-mss ${forced_MSS}
#ip6tables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom6: Zone wan MTU fixing" -j TCPMSS  --set-mss ${forced_MSS}

# the following does work under TOS5 as of 20220618
# affects UPLOAD transfers 
#iptables -t mangle -A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom: Zone wan ingress MTU fixing" -j TCPMSS --set-mss ${forced_MSS}
#ip6tables -t mangle -A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom6: Zone wan ingress  MTU fixing" -j TCPMSS --set-mss ${forced_MSS}

# affects DOWNLOAD transfers
ingress_forced_MSS=${forced_MSS}
#iptables -t mangle -A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom: Zone wan egress2 MTU fixing" -j TCPMSS --set-mss ${ingress_forced_MSS}
#ip6tables -t mangle -A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom6: Zone wan egress2 MTU fixing" -j TCPMSS --set-mss ${ingress_forced_MSS}

I think I last tested the "affects UPLOAD transfers" and the "affects DOWNLOAD transfers" variants... as you note from my macos comment OS are free to ignore these values to a degree...

moeller0 · September 30, 2024, 6:39pm

root@turris:/etc$ iptables-save | grep TCPMSS
-A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth2.42 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth2.42 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu

brada4 · September 30, 2024, 6:39pm

mangle forward -o ppoe foes not change mtu as out interface is not determined beefore routing.

brada4 · September 30, 2024, 6:40pm

You have to patch fw3 or add mangle postrouting rules
you can backt$ack to original broken pbr and ppoe threds from fw3 patch.

moeller0 · September 30, 2024, 6:44pm

Well, that used to work, and I confirmed both directions with packet captures... but that was ages ago, well possible that regressed. I wonder whether it is worth the hassle to repeat the measurement, given that the switch to fw4 is eminent when things will change in how to issue that command.

brada4 · September 30, 2024, 6:50pm

Original thread

github.com/openwrt/openwrt

MSS clamping does not play nice with policy-based routing (mwan3, pbr)

opened 06:02AM - 03 Mar 23 UTC

closed 01:42PM - 03 Nov 23 UTC

patrakov

bug

### Describe the bug OpenWrt applies MSS clamping in the `firewall` and `firewa…ll4` packages, by hooking into the `FORWARD `chain of the `mangle` table, or its `nftables` equivalent. This is done on the assumption that locally generated SYN packets will never need their MSS fixed up - but I found a situation where this assumption is wrong, and packets with MSS=1460 are wrongly sent through the VPN. ### OpenWrt version 21.02.1 and 22.03.3 ### OpenWrt target/subtarget ath79/generic and mediatek/mt7622 ### Device TP-Link Archer C7 V2 and Linksys E8450 ### Image kind Official downloaded image ### Steps to reproduce Create a usual DHCP WAN connection, and a PPTP VPN. Set the metrics, as required by mwan3 which we will install at the next step: 80 on WAN, 90 on PPTP VPN. Assign both to the WAN firewall zone. At this point we have a default routing table with two default routes with different metrics, with the traffic going over WAN, because its default route has a lower metric. In the firewall settings, enable MSS clamping for the WAN zone. Install mwan3 and curl. Install luci-app-mwan3 if you want. Configure mwan3. The default policies that prefer wan over wanb, and wanb over wan, should be sufficient, when "wanb" is globally replaced with "vpn" in the config where it refers to the interface name. The default policy should be wan_wanb, thus preferring the WAN. Then create a rule in mwan3: if the destination IP is 139.59.209.225 (or whatever the current IPv4 address of openwrt.org is), then use the wanb_wan policy, thus preferring the VPN. Push the rule to the top, so that it is evaluated first. Delete the load-balancing rule. Save and apply. The result at this point should be that everything goes through the WAN, except 139.59.209.225, which goes through the VPN. The real use case is to have an ipset instead of the single IP, but for testing, a single IP will suffice. Then, on the router itself, while observing the resulting packets, run: ``` curl -4 https://openwrt.org/ ``` ### Actual behaviour The initial SYN packet indicates MSS=1460, despite the rule that fixes the MTU. If the VPN blackholes "Fragmentation Needed" packets, then the page won't load. ### Expected behaviour The initial SYN packet should have MSS clamped to 1360. ### Additional info Applying the MSS clamp in the POSTROUTING chain, in addition to FORWARD (or maybe instead of FORWARD?), would fix the issue. OUTPUT is not sufficient. Let's see why the wrong MSS exists in the first place, though. When determining the MSS for locally-created SYN packets, apparently, the kernel only looks into the default routing table. Indeed, this decision is made before running through iptables/netfilter, and therefore, the kernel cannot know that it will select a non-default routing table based on the not-yet-existing firewall mark. Therefore, as the default routing table clearly says "go through WAN, MTU=1500", it selects MSS=1460. The stock MSS-clamping rule is applied only on forward, but the packet is locally-originated, not forwarded, so it doesn't apply. End result: packet with MSS=1460 (which is wrong) goes out. The POSTROUTING chain of the mangle table would still catch it. ### Diffconfig _No response_ ### Terms - [X] I am reporting an issue for OpenWrt, not an unsupported fork.

Im not at keuboard, but mangle forward passed unmodified from mtu 1500 wifi client in other issue...

openwrticon · September 30, 2024, 6:51pm

I tried every idea possible in all chains....
setting it on public ip, setting it on local bridges.
Yet, I see big packets.
I only want to change MSS for outgoing TCP connections, thus I cannot change the mtu of the pppoe-wan.Could IFB be the culprit somehow?

root@turris:~# iptables-save | grep TCPMSS
-A PREROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A PREROUTING ! -s 192.168.0.0/16 -d 192.168.1.1/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A PREROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A PREROUTING -i pppoe-wan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A INPUT -i pppoe-wan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A INPUT ! -s 192.168.0.0/16 -d 192.168.0.0/16 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A INPUT ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A FORWARD ! -s 192.168.0.0/16 -d 192.168.0.0/16 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A FORWARD ! -s 192.168.0.0/16 -d public/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A FORWARD -o eth2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A OUTPUT ! -s 192.168.0.0/16 -o br-lan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A OUTPUT ! -s 192.168.0.0/16 -o wlan1 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A OUTPUT ! -s 192.168.0.0/16 -o br-LANWIFI -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -m owner --uid-owner 453 -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -o br-lan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -o wlan1 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -o br-LANWIFI -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540

brada4 · September 30, 2024, 6:54pm

-t mangle -A POSTROUTING -o pppoe -p tcp -j TCPMSS --clamp/set

openwrticon · September 30, 2024, 7:18pm

This mangles the outgoing handshake of a session.
I want to mangle the incoming handshake of a TCP session.

I have the rule below:

-A POSTROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540

and I tried the rule:

A POSTROUTING -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540

and I see no packets hitting them.

brada4 · September 30, 2024, 7:24pm

Ermm mine hits all connections on a pc.

openwrticon · October 1, 2024, 7:14am

Anyway,

I see in clients devices with tcpdump that clamping is applied.
However, some applications seem to ignore that packet size.

Is there a linux setting that should be lowered?
Can applications enforce somehow the minimum packet size?

brada4 · October 1, 2024, 7:15am

Yes, firewalls do that en masse.

openwrticon · October 1, 2024, 7:48am

I don't have firewall in client devices.

moeller0 · October 1, 2024, 12:55pm

Yes, that is indeed possible.

Sure if an application only writes a single byte to a tcp socket it can either try to tickle the socket to transmit the data or it can wait long enough for the data getting transferred eventually. Unlike with UDP with TCP things are a bit approximate, if the application writes one byte now, and another byte just before the timeout causes a transmission, tcp will send both bytes together in one segment. So sure applications are in control, but TCP does not guarantee to package data into packets based on the applications "batches". But it is the TCP stack that uses the MSS information to deduce what maximum segment size to use, so the whole MSS clamping exercise is depending less on the application and more on the tcp stacks on both ends.

TCP stacks might still not accept arbitrarily low MSS values (as that can create DOS vectors) and your bet is as good as mine which MSS a stack uses that you asked for MSS 100 and that considers that insane, could be the lowest MSS it will accept or the MSS deduced from the path MTU...

wilsonyan · October 1, 2024, 1:43pm

openwrticon:

iptables -t mangle -A FORWARD -p tcp --tcp-flags FIN,SYN,RST SYN -i pppoe-wan -j TCPMSS --set-mss 540
iptables -t mangle -A FORWARD -p tcp --tcp-flags FIN,SYN,RST SYN -i pppoe-wan -j ACCEPT

might wanna change that

-i pppoe-wan
to
-o pppoe-wan

It's really the packets you send out that you want to change up first, so if you have a pc with an mtu of 1500 on your network and it gets to your router you want it adjusted then to fit 1492 or whatever as it goes out etc, not for stuff that has nothing to do with that problem coming in

brada4 · October 1, 2024, 1:47pm

One can adjust syn and syn/ack so one direction suffices.