Incoming TCP MSS clamping and large packets in outgoing direction

I have a measly ADSL 10Mbit/1Mbit. I modify incoming TCP sessions to reduce the tcp packets to 540 in order to reduce upload latency. iptables rules seem to be applied.

iptables -t mangle -A FORWARD -p tcp --tcp-flags FIN,SYN,RST SYN -i pppoe-wan -j TCPMSS --set-mss 540
iptables -t mangle -A FORWARD -p tcp --tcp-flags FIN,SYN,RST SYN -i pppoe-wan -j ACCEPT

(I see them on receiving devices with tcpdump)
pppoe-wan is the outgoing device and ifb4pppoe-wan is the incoming IFB.

However when I monitor outgoing packets with iptables -j LOG I see tcp packets LEN=1120 which implies that clamping is ignored.
Could clients ignore TCP MSS clamping and deliver big packets?
Any idea to troubleshoot it deeper?

Check this out:

Can you tell why are you stuck with fw3?
(fw3 version here https://github.com/openwrt/firewall3/pull/1)

Please connect to your OpenWrt device using ssh and copy the output of the following commands and post it here using the "Preformatted text </> " button:
grafik
Remember to redact passwords, MAC addresses and any public IP addresses you may have:

ubus call system board
cat /etc/config/network
cat /etc/config/wireless
cat /etc/config/dhcp
cat /etc/config/firewall

I am using Turris Omnia which uses older openwrt.

root@turris:~# ubus call system board
{
        "kernel": "5.15.148",
        "hostname": "turris",
        "system": "ARMv7 Processor rev 1 (v7l)",
        "model": "Turris Omnia",
        "board_name": "cznic,turris-omnia",
        "rootfs_type": "btrfs",
        "release": {
                "distribution": "TurrisOS",
                "version": "7.0.3",
                "revision": "r20343+127-4e1d1b7df0",
                "target": "mvebu/cortexa9",
                "description": "TurrisOS 7.0.3 4e1d1b7df0ce6fa96d7462dc883917682f428046"
        }
}
root@turris:~# cat /etc/config/network

config interface 'loopback'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'
        option device 'lo'

config globals 'globals'
        option ula_prefix 'dsfsdfsdfsdfdsfsd'

config interface 'lan'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option _orig_ifname 'eth0 eth2 wlan0 wlan1'
        option _orig_bridge 'true'
        option delegate '0'
        option device 'br-lan'

config interface 'wan'
        option proto 'pppoe'
        option ipv6 '0'
        option username 'vxcvxcvxvx'
        option password 'xcvxcvxcvxcvxcvxc'
        #option pppd_options 'debug mtu 1452'
        option device 'eth2'
        option mtu '1452'

config interface 'guest_turris'
        option proto 'static'
        option bridge_empty '1'
        option enabled '1'
        option _orig_ifname 'guest_turris_0 guest_turris_0'
        option _orig_bridge 'true'
        option netmask '255.255.255.0'
        option ipaddr '192.168.3.1'
        option delegate '0'

config interface 'modem'
        option proto 'dhcp'
        option delegate '0'
        option hostname 'turris'
        option device 'eth2'

config device
        option name 'eth1'
        option mtu '9000'

config device
        option name 'lan0'
        option mtu '8000'

config device
        option name 'lan1'
        option mtu '8000'

config device
        option name 'lan2'
        option mtu '8000'

config device
        option name 'lan3'
        option mtu '8000'

config interface 'LANWIFI'
        option proto 'static'
        option ipaddr '192.168.4.1'
        option netmask '255.255.255.0'
        option enabled '1'
        option delegate '0'
        option device 'br-LANWIFI'

config device 'br_lan'
        option name 'br-lan'
        option stp '1'
        list ports 'lan0'
        list ports 'vethvm1'
        list ports 'lan1'
        list ports 'lan2'
        list ports 'lan3'
        option type 'bridge'
        option mtu '8000'

config device 'br_LANWIFI'
        option name 'br-LANWIFI'
        option bridge_empty '1'
        option stp '1'
        option type 'bridge'

firewall rules are applied by SQM service.
I see in tcpdump of the clients that

tcpdump -i wlan0 -n -v "tcp[tcpflags] & (tcp-syn) != 0"

TCP sessions have the correct mss

20:54:20.312829 IP (tos 0x0, ttl 50, id 32678, offset 0, flags [DF], proto TCP (6), length 60)
    fghfghfghgfhgfhf > fhgfhfghfghfghfgh: Flags [S], cksum 0xd364 (correct), seq 1288761723, win 65535, options [mss 540,sackOK,TS val 3147656100 ecr 0,nop,wscale 10], length 0

You need mangle postrouting -o ppoe and mangle forward -i ppoe
and set small mtu (1280 for ip6 or 576 for ip4 and not less) on the interface as 20% +- of traffic us udp http/3.
Drop a line if you need exact commands.

Also check whether turris has fw patched or no

iptables-save | grep TCPMSS

(confirm here but ultimately turris has to fix fw3 if it is wrong)

Turris is about to finally switch to fw4... (scheduled for the next minor point release) so maybe the best is to hobble along.. However here is what I used on my turris omnia with turrisOS successfully in the past as firewall custom rule (/etc/firewall.user):

# This file is interpreted as shell script.
# Put your custom iptables rules here, they will
# be executed with each firewall (re-)start.

# Internal uci firewall chains are flushed and recreated on reload, so
# put custom rules into the root chains e.g. INPUT or FORWARD or into the
# special user chains, e.g. input_wan_rule or postrouting_lan_rule.


# special rules to allow MSS clamping for in and outbound traffic
# use ip6tables -t mangle -S ; iptables -t mangle -S to check
forced_MSS=1220 # 300, 536, 1280-60
# affects both down- and upstream, egress seems to require at least 216 on macos without setting a sysctl value
#iptables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom: Zone wan MTU fixing" -j TCPMSS --set-mss ${forced_MSS}
#ip6tables -t mangle -A FORWARD -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom6: Zone wan MTU fixing" -j TCPMSS  --set-mss ${forced_MSS}

# the following does work under TOS5 as of 20220618
# affects UPLOAD transfers 
#iptables -t mangle -A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom: Zone wan ingress MTU fixing" -j TCPMSS --set-mss ${forced_MSS}
#ip6tables -t mangle -A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom6: Zone wan ingress  MTU fixing" -j TCPMSS --set-mss ${forced_MSS}

# affects DOWNLOAD transfers
ingress_forced_MSS=${forced_MSS}
#iptables -t mangle -A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom: Zone wan egress2 MTU fixing" -j TCPMSS --set-mss ${ingress_forced_MSS}
#ip6tables -t mangle -A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "custom6: Zone wan egress2 MTU fixing" -j TCPMSS --set-mss ${ingress_forced_MSS}

I think I last tested the "affects UPLOAD transfers" and the "affects DOWNLOAD transfers" variants... as you note from my macos comment OS are free to ignore these values to a degree...

root@turris:/etc$ iptables-save | grep TCPMSS
-A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth2.42 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth2.42 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
1 Like

mangle forward -o ppoe foes not change mtu as out interface is not determined beefore routing.

You have to patch fw3 or add mangle postrouting rules
you can backt$ack to original broken pbr and ppoe threds from fw3 patch.

Well, that used to work, and I confirmed both directions with packet captures... but that was ages ago, well possible that regressed. I wonder whether it is worth the hassle to repeat the measurement, given that the switch to fw4 is eminent when things will change in how to issue that command.

Original thread

Im not at keuboard, but mangle forward passed unmodified from mtu 1500 wifi client in other issue...

I tried every idea possible in all chains....
setting it on public ip, setting it on local bridges.
Yet, I see big packets.
I only want to change MSS for outgoing TCP connections, thus I cannot change the mtu of the pppoe-wan.Could IFB be the culprit somehow?

root@turris:~# iptables-save | grep TCPMSS
-A PREROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A PREROUTING ! -s 192.168.0.0/16 -d 192.168.1.1/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A PREROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A PREROUTING -i pppoe-wan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A INPUT -i pppoe-wan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A INPUT ! -s 192.168.0.0/16 -d 192.168.0.0/16 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A INPUT ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A FORWARD ! -s 192.168.0.0/16 -d 192.168.0.0/16 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A FORWARD ! -s 192.168.0.0/16 -d public/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A FORWARD -o eth2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A OUTPUT ! -s 192.168.0.0/16 -o br-lan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A OUTPUT ! -s 192.168.0.0/16 -o wlan1 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A OUTPUT ! -s 192.168.0.0/16 -o br-LANWIFI -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -m owner --uid-owner 453 -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -o br-lan -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -o wlan1 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540
-A POSTROUTING ! -s 192.168.0.0/16 -o br-LANWIFI -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540

-t mangle -A POSTROUTING -o pppoe -p tcp -j TCPMSS --clamp/set

This mangles the outgoing handshake of a session.
I want to mangle the incoming handshake of a TCP session.

I have the rule below:

-A POSTROUTING ! -s 192.168.0.0/16 -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540

and I tried the rule:

A POSTROUTING -d publicip/32 -p tcp -m tcp --tcp-flags FIN,SYN,RST SYN -j TCPMSS --set-mss 540

and I see no packets hitting them.

Ermm mine hits all connections on a pc.

Anyway,

I see in clients devices with tcpdump that clamping is applied.
However, some applications seem to ignore that packet size.

Is there a linux setting that should be lowered?
Can applications enforce somehow the minimum packet size?

Yes, firewalls do that en masse.

I don't have firewall in client devices.

Yes, that is indeed possible.

Sure if an application only writes a single byte to a tcp socket it can either try to tickle the socket to transmit the data or it can wait long enough for the data getting transferred eventually. Unlike with UDP with TCP things are a bit approximate, if the application writes one byte now, and another byte just before the timeout causes a transmission, tcp will send both bytes together in one segment. So sure applications are in control, but TCP does not guarantee to package data into packets based on the applications "batches". But it is the TCP stack that uses the MSS information to deduce what maximum segment size to use, so the whole MSS clamping exercise is depending less on the application and more on the tcp stacks on both ends.

TCP stacks might still not accept arbitrarily low MSS values (as that can create DOS vectors) and your bet is as good as mine which MSS a stack uses that you asked for MSS 100 and that considers that insane, could be the lowest MSS it will accept or the MSS deduced from the path MTU...

might wanna change that

-i pppoe-wan
to
-o pppoe-wan

It's really the packets you send out that you want to change up first, so if you have a pc with an mtu of 1500 on your network and it gets to your router you want it adjusted then to fit 1492 or whatever as it goes out etc, not for stuff that has nothing to do with that problem coming in

1 Like

One can adjust syn and syn/ack so one direction suffices.

1 Like