How to set correct ipsec VPN MTU

Hi guys.

my schema looks like:

LAN1 (MTU 1500) <> (br-lan "MTU 1500" openwrt router eth0.2 (WAN) "MTU 1000" <> Cisco router cable modem <> cisco router "MTU1500" <> LAN2 "MTU 1500"

ipsec VPN from openwrt router established no problem. I cam ping LAN1 from LAN2 w/ default ICMP packets size 64 bit and wise versa but. As soon as I ssh-ing from LAN2 to LAN1 openwrt router via VPN tunnel and run any command w/ long output like: "cat /etc/config/firewall" my session freezes for couple minutes closed finally with message:

packet_write_wait: Connection to 192.168.173.1 port 22: Broken pipe

At the same time. Whenever I login to LAN1 openwrt router eth.2 (wan interface with public IP assigned to it) I do not get any drop, sesson close and timeout..

It force me to think MTU for ipsec tunnel from LAN1 openwrt to LAN2 cisco has incorrect MTU.

My LAN 1 openwrt config looks like:

/etc/config/network
config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.10'
        option proto 'static'
        option ipaddr '192.168.173.1'
        option netmask '255.255.255.240'
        option ipv6 '0'
#       option mtu '1200'

config interface 'voice'
        option type 'bridge'
        option ifname 'eth0.13'
        option proto 'static'
        option ipaddr '192.168.173.33'
        option netmask '255.255.255.248'
        option ipv6 '0'
#       option mtu '1200'

config device 'wan_eth0_2_dev'
        option name 'eth0.2'
        option macaddr '12:34:56:78:90:ab'
        option mtu '1000'

Could you guys explain me general rules to setup MTU for multi interface openwrt router with ipsec vpn. I read a lot of documentation but didn't get final conclusion...

Thank you.

Is there a good reason you have mtu mismatch between the wan interface and the cisco router?

my cisco router has 1500 for int gi8 (connected via cable modem to ISP):
int gi8 has default MTU 1500:

gate(config-if)#do sh ip int gi 8 | i MTU
  MTU is 1500 bytes
gate(config-if)#

openwrt router from other side of ipsec VPN tunnel has default MTU as well:

root@a81m5:/home/sam# ip link sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:45 brd ff:ff:ff:ff:ff:ff
3: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:44 brd ff:ff:ff:ff:ff:ff
4: wlan1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:45 brd ff:ff:ff:ff:ff:ff
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:45 brd ff:ff:ff:ff:ff:ff
6: eth0.10@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:45 brd ff:ff:ff:ff:ff:ff
7: br-voice: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:45 brd ff:ff:ff:ff:ff:ff
8: eth0.13@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-voice state UP mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:45 brd ff:ff:ff:ff:ff:ff
9: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e8:de:27:ff:9f:46 brd ff:ff:ff:ff:ff:ff
root@a81m5:/home/sam#

Bellow relives me suspect I have MTU issue:

Whenever I login from LAN2 linux to LAN1 openwrt via ipsectunnel I'm able to read small files (/etc/hosts) but big one (etc/config/firewall) freezes:

[sam@st1 open-wrt]$ ssh 192.168.173.1 -p52407
sam@192.168.173.1's password: 

BusyBox v1.28.4 () built-in shell (ash)

 ----------------------------------
 OpenWrt 18.06.7, r7976-ca47026b7d
 ----------------------------------
sam@a81m5:~$ sudo ash
Password: 


BusyBox v1.28.4 () built-in shell (ash)

root@a81m5:/home/sam# 
root@a81m5:/home/sam# 
root@a81m5:/home/sam# 
root@a81m5:/home/sam# cat /etc/hosts 
127.0.0.1 localhost

192.168.172.2   main.st1.org  main
192.168.172.1   gate.st1.org  gate
192.168.172.3   db4.st1.org  db4
192.168.172.9   ns3.st1.org   ns3
192.168.172.9   ap.sit1.org    ap
192.168.172.4   ns2.st1.org   ns2
192.168.172.38  spa.st1.org   spa
192.168.174.34  spa-c7.st1.org   spa-c7
192.168.174.4   ltop-c7.st1.org  ltop-c7


#::1     localhost ip6-localhost ip6-loopback
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters
root@a81m5:/home/sam# 
root@a81m5:/home/sam# 
root@a81m5:/home/sam# 
root@a81m5:/home/sam# cat /etc/config/firewall

**-= couple minutes freeze here ..... =-**

packet_write_wait: Connection to 192.168.173.1 port 52407: Broken pipe

I think I need to setup proper MTU for openwrt interfaces but completely got in stack w/ my experiments.

I've fount icmp packet size 1410 fit the tunnel:

[sam@st15 open-wrt]$ ping -M do -s 1411 192.168.173.1 -c2
PING 192.168.173.1 (192.168.173.1) 1411(1439) bytes of data.

--- 192.168.173.1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1001ms

[sam@st15 open-wrt]$ ping -M do -s 1410 192.168.173.1 -c2
PING 192.168.173.1 (192.168.173.1) 1410(1438) bytes of data.
1418 bytes from 192.168.173.1: icmp_seq=1 ttl=63 time=18.0 ms
1418 bytes from 192.168.173.1: icmp_seq=2 ttl=63 time=19.1 ms

--- 192.168.173.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 17.951/18.526/19.101/0.575 ms
[sam@st15 open-wrt]$

but where I have to setup MTU 1410 at openwrt router (a81m5) behind the tunnel? interfaces eth.0,.2 (wan) eth0, eth0.10, eth0.13, br-lan, br-wan?

Before you setup anything anywhere remove the MTU values you have added in the config files and restart the router to get the correct values.
Second, what are the connection types from both ends? If it is DHCP or STATIC with a cable connected to the modem of the ISP, then 1500 is generally fine (unless instructed by the ISP). If it is PPPOE then it will automatically adjust to the lower value.
Are you experiencing such issues only on the ipsec traffic or in general?

  • All MTU values returned to default 1500.
  • Router restart didn't change situation.
  • 1G fiber link to ISP for LAN1, cable internet connection for LAN2.
    this is just for VPN traffic. there is no issue if I'm connection to/via openwrt int eth0.2 (WAN)

Things to look at that I can think of:

  1. Make sure you have option mtu_fix '1' in your firewall for your VPN zone (wan and/or IPSec tunnel)
  2. Test which MTU works once you have the ipsec tunneling up and running using ping (see below section and pretty much what you did to determing the 1410 value)
  3. Set that MTU correctly for connecting devices:
    3.1. maybe using DHCP option if your are using IPSec without a tunnel and mtu_fix doesn't do the job for you
    3.2. by setting the MTU on the IPSec tunnel interface (for example vti) - if you use a tunnel

Some commands that could help testing
Typically, you can use ping to determine the highest MTU that works on your network (ping with the MTU value - 28). To run them on the openWRT side, you would have to install iputils-ping first to be able to have all the options.

for example, for an mtu of 1500, you want to ping with 1500 - 28 = 1472. To check it (example will use eth0.2), run:

ping -s 1472 -c1 -M do -I eth0.2 8.8.8.8

With a size too long (for example 1473), you would get an error like this:

ping: local error: Message too long, mtu=1500

To get the current mtu on an interface (e.g. eth0.2):

cat /sys/class/net/eth0.2/mtu

To temporarily set the mtu to a specific value on an interface to test if it works (worth a try on your wan interface):

ifconfig eth0.2 mtu 1472 up

Thank you Vincent.

of course I did all MTU calculation&checking actions prior I wrote question to forum.
nonetheless. issue has been resolved. it was some trailing space symbols in firewall config. As results NAT exceptions rules did not place correctly.

1 Like

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.