Larger ping packets get not fragmented

Hi guys,

I have three internet connection with two different providers.
Larger ping requests get not fragmented with one of the providers (UPC, two internernet connections are from this one), so the limit there is 1472 byte for some reason.

Larger ping do not get through, even if I lower down the MTU on my router.

Internet Connection 1 (IC1):
Same thing with this connection, which is a symetric UPC / Inode DSL line:

eth1.301  Link encap:Ethernet  HWaddr 58:EF:68:B7:6D:A4
          inet addr:83.65.96.212  Bcast:83.65.96.223  Mask:255.255.255.240
          inet6 addr: fe80::5aef:68ff:feb7:6da4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:117466237 errors:0 dropped:0 overruns:0 frame:0
          TX packets:89747404 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:44893152409 (41.8 GiB)  TX bytes:49781448291 (46.3 GiB)

Internet Connection 2 (IC2):
On my third connection, which is a A1 LTE modem line, larger pings are working.

eth2      Link encap:Ethernet  HWaddr 00:A0:C6:00:00:00
          inet addr:90.152.149.110  Bcast:255.255.255.255  Mask:255.255.255.255
          inet6 addr: fe80::2a0:c6ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:203376486 errors:0 dropped:0 overruns:0 frame:0
          TX packets:132904330 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1604613911 (1.4 GiB)  TX bytes:3256878506 (3.0 GiB)
ping 88.198.132.225 -l 2500

Ping wird ausgeführt für 88.198.132.225 mit 2500 Bytes Daten:
Antwort von 88.198.132.225: Bytes=2500 Zeit=80ms TTL=53
Antwort von 88.198.132.225: Bytes=2500 Zeit=57ms TTL=53
Antwort von 88.198.132.225: Bytes=2500 Zeit=48ms TTL=53
Antwort von 88.198.132.225: Bytes=2500 Zeit=59ms TTL=53

Ping-Statistik für 88.198.132.225:
    Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0
    (0% Verlust),
Ca. Zeitangaben in Millisek.:
    Minimum = 48ms, Maximum = 80ms, Mittelwert = 61ms
ping 88.198.132.225 -l 1472 -f

Ping wird ausgeführt für 88.198.132.225 mit 1472 Bytes Daten:
Antwort von 88.198.132.225: Bytes=1472 Zeit=56ms TTL=53
Antwort von 88.198.132.225: Bytes=1472 Zeit=92ms TTL=53
Antwort von 88.198.132.225: Bytes=1472 Zeit=79ms TTL=53
Antwort von 88.198.132.225: Bytes=1472 Zeit=82ms TTL=53

Ping-Statistik für 88.198.132.225:
    Pakete: Sent = 4, Received = 4, Lost = 0
    (0% Loss),
Ca. Zeitangaben in Millisek.:
    Minimum = 56ms, Maximum = 92ms, Mittelwert = 77ms
ping 88.198.132.225 -l 1473 -f

Ping wird ausgeführt für 88.198.132.225 mit 1473 Bytes Daten:
Paket müsste fragmentiert werden, DF-Flag ist jedoch gesetzt.
Paket müsste fragmentiert werden, DF-Flag ist jedoch gesetzt.
Paket müsste fragmentiert werden, DF-Flag ist jedoch gesetzt.
Paket müsste fragmentiert werden, DF-Flag ist jedoch gesetzt.

Ping-Statistik für 88.198.132.225:
    Pakete: Sent = 4, Received = 0, Lost = 4
    (100% Loss),

(Sorry for the German there in the command, I have only a German Windows PC on this location and I have not seen a DF option with the OpenWRT ping command)

Internet Connection 3 (IC3):
This is a UPC / Chello Internet connection via a cable TV line.

eth1.2    Link encap:Ethernet  HWaddr 58:EF:68:B7:62:F4
          inet addr:84.115.50.36  Bcast:84.115.50.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:576  Metric:1
          RX packets:5241 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3697 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1087994 (1.0 MiB)  TX bytes:699318 (682.9 KiB)
root@uFW1:~# ping  88.198.132.225 -s 1473
PING 88.198.132.225: 1473 data bytes
^C
---  88.198.132.225 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@uFW1:~# ping  88.198.132.225 -s 1472
PING www.nahtec.at (88.198.132.225): 1472 data bytes
1480 bytes from 88.198.132.225: seq=0 ttl=54 time=33.075 ms
1480 bytes from 88.198.132.225: seq=1 ttl=54 time=32.231 ms

So 1072 bytes are the limit, even when the DF bit is not set.

Since I already lowered down the packages with the low MTU size on my side, I guess the problem is the return package.
Has this provider a problem with black wholes?
If yes, how can I get around this?

On your WAN:

Screenshot%20from%202018-11-01%2010-31-34

This is what I did, but I don't get the large ping through it.

Your MTU is 576...this is a problem!

How can a too low MTU be a problem, except in the total throughput?

A low MTU is your problem because:

This is clear to me so far, that larger packets than MTU - 28 bytes overhead (1472 byte if MTU is 1500) cannot get through, if DF is set.
But if DF is not set, than larger packets should get through by fragmenting, but this is not always the case.

I noticed some weird behaviour with the router (R1) on site 1 (S1) which runs multiwan3 with IC1 (eth1.301) & IC2 (eth1.302).

I actually get a ping through with 1473 bytes via IC1 to a webserver (WS1, 88.198.132.225), but only when IC2 is up and running.
This ping works, as long as the IC2 (eth1.302) is available.

ping -I eth1.301 -s 1473 88.198.132.225

IC2 (eth1.302) is a another OpenWRT Router (R2) which is connected to a LTE modem via USB.
When I shut down the WWAN (USB) interface there on R2, then the ping on R1 via IC1 (eth1.301) stops, which has actually nothing to do with this IC2 (eth1.302) connection.

Site 2 (S2) is connected via IC3 and router 3 (R3).
I can ping from IC3 to IC1 with 1473 byte (or larger) packtes without problems, independent from IC2. The other way around does not work, the limit is there 1472.
I can also ping IC3 to IC2, but the limit there is 1420 byte and the packets get not fragmented if they are larger.
The packet size from S2 to WS1 is also limited to 1472 bytes, even when a ping from S2/IC3 to S1/IC1 is not.

S1/R1:
LEDE Reboot 17.01.4
mwan3 + SQM
IC1 via DSL,routed, public IP is at eth1.301; IC2 via R2
IP: 83.65.96.212

S1/R2:
LEDE Reboot 17.01.4
IC2 via USB LTE modem, public IP is at eth2
IP: 90.152.149.110

S2/R3:
OpenWrt 18.06.1
IC3 via cable modem, public IP is at eth1.2
IP: 84.115.50.36

Not sure how that all plays together...

To make it more confusing, it was actually working before, to send packets larger than 1472 from IC1 to IC3, but this does not work anymore for some reason.
I found this in my terminal where I did some tests before:

ping -I eth1.301 -s 1473 84.115.50.36
PING 84.115.50.36 (84.115.50.36) from 83.65.96.212 eth1.301: 1473(1501) bytes of data.
1481 bytes from 84.115.50.36: icmp_req=1 ttl=61 time=33.8 ms
1481 bytes from 84.115.50.36: icmp_req=2 ttl=61 time=27.5 ms
1481 bytes from 84.115.50.36: icmp_req=3 ttl=61 time=26.4 ms
1481 bytes from 84.115.50.36: icmp_req=4 ttl=61 time=28.1 ms
1481 bytes from 84.115.50.36: icmp_req=5 ttl=61 time=33.0 ms
1481 bytes from 84.115.50.36: icmp_req=6 ttl=61 time=29.1 ms
1481 bytes from 84.115.50.36: icmp_req=7 ttl=61 time=30.5 ms
1481 bytes from 84.115.50.36: icmp_req=8 ttl=61 time=40.4 ms
1481 bytes from 84.115.50.36: icmp_req=9 ttl=61 time=30.1 ms
1481 bytes from 84.115.50.36: icmp_req=10 ttl=61 time=27.3 ms
^C
--- 84.115.50.36 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 26.494/30.672/40.495/3.998 ms

This means to me that the behavoir changes, probably somewhere at the provider, I already rebooted all the routers.

I wonder, maybe the LTE thing uses IPv4 as a service over IPv6, and in that case they might a) have a higher overhead ad b) might simply disallow IPv4 fragmentation at all. Make sure to set the MSS clamping values accordingly for each uplink if this hypothesis is correct....

So far as I understand is MSS clamping only relevant for TCP connections, but not for UDP nor ICMP echo requests.

I did another test with R2 and did set the MTU there down to 1300 on the interface of the public IP address IC2 (LTE).

eth2      Link encap:Ethernet  HWaddr 00:A0:C6:00:00:00
          inet addr:90.152.149.110  Bcast:255.255.255.255  Mask:255.255.255.255
          inet6 addr: fe80::2a0:c6ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:213 errors:0 dropped:0 overruns:0 frame:0
          TX packets:185 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:36514 (35.6 KiB)  TX bytes:64986 (63.4 KiB)

Why is it actually still possible to ping from IC3 to IC2 with DF set and 1420 bytes?

ping -M do -s 1420 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1420(1448) bytes of data.
1428 bytes from 90.152.149.110: icmp_req=1 ttl=54 time=47.1 ms
1428 bytes from 90.152.149.110: icmp_req=2 ttl=54 time=375 ms
1428 bytes from 90.152.149.110: icmp_req=3 ttl=54 time=52.7 ms
^C
--- 90.152.149.110 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 47.164/158.395/375.227/153.340 ms
ping -M do -s 1421 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1421(1449) bytes of data.
^C
--- 90.152.149.110 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3147ms
ping -s 1421 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1421(1449) bytes of data.
^C
--- 90.152.149.110 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4195ms

Should this not limit the maximum possible size without fragmentation to MTU - 28 bytes, means to 1272 bytes?

When I try to ping IC3 from IC2 then it is limited according to the MTU.

ping -M do -s 1472 84.115.50.36
PING 84.115.50.36 (84.115.50.36) 1472(1500) bytes of data.
ping: local error: Message too long, mtu=1300
ping: local error: Message too long, mtu=1300
^C
--- 84.115.50.36 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1004ms

But even with fragmenation is the limit 1472.

ping -s 1472 84.115.50.36
PING 84.115.50.36 (84.115.50.36) 1472(1500) bytes of data.
1480 bytes from 84.115.50.36: icmp_req=1 ttl=54 time=51.0 ms
1480 bytes from 84.115.50.36: icmp_req=2 ttl=54 time=67.6 ms
1480 bytes from 84.115.50.36: icmp_req=3 ttl=54 time=87.8 ms
^C
--- 84.115.50.36 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 51.074/68.886/87.888/15.052 ms
ping -s 1473 84.115.50.36
PING 84.115.50.36 (84.115.50.36) 1473(1501) bytes of data.
^C
--- 84.115.50.36 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5004ms

When I change the MTU on IC3 also down to 1300, then I am able to get packets through up to 1472 with fragmentation enabled and 1272 byte (what makes sense to me) with DF set.

eth1.2    Link encap:Ethernet  HWaddr 58:EF:68:B7:62:F4
          inet addr:84.115.50.36  Bcast:84.115.50.255  Mask:255.255.255.0
          inet6 addr: fe80::5aef:68ff:feb7:62f4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1300  Metric:1
          RX packets:4640 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2303 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4244631 (4.0 MiB)  TX bytes:250853 (244.9 KiB)

Without fragmentation is the limit 1272 what makes sense to me since the MTU of 1300:

ping -M do -s 1272 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1272(1300) bytes of data.
1280 bytes from 90.152.149.110: icmp_req=1 ttl=54 time=47.5 ms
1280 bytes from 90.152.149.110: icmp_req=2 ttl=54 time=90.5 ms
^C
--- 90.152.149.110 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 47.519/69.053/90.587/21.534 ms
ping -M do -s 1273 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1273(1301) bytes of data.
ping: local error: Message too long, mtu=1300
ping: local error: Message too long, mtu=1300
ping: local error: Message too long, mtu=1300
^C
--- 90.152.149.110 ping statistics ---
3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2071ms

With fragmentation limited to 1472 bytes:

ping -s 1472 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1472(1500) bytes of data.
1480 bytes from 90.152.149.110: icmp_req=1 ttl=54 time=92.5 ms
1480 bytes from 90.152.149.110: icmp_req=2 ttl=54 time=49.5 ms
1480 bytes from 90.152.149.110: icmp_req=3 ttl=54 time=47.1 ms
1480 bytes from 90.152.149.110: icmp_req=4 ttl=54 time=58.2 ms
^C
--- 90.152.149.110 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 47.144/61.879/92.574/18.200 ms
ping -s 1473 90.152.149.110
PING 90.152.149.110 (90.152.149.110) 1473(1501) bytes of data.
^C
--- 90.152.149.110 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3143ms

Where comes the limit of 1472 byte from, even when fragmentation is allowed?
Should not OpenWRT put the packets into 1300 byte fragments since the MTU ist set to 1300?