Strange behavior LEDE + Windows + IPSec

Yojimbo · May 9, 2018, 10:26am

I'm facing a strange issue with LEDE router + Windows laptop + IPSec server.
It would be great to hear that somebody solved the same issue or at least to hear some words of help

Initial configuration:

My home router is TP-Link TL-WDR4300 with latest LEDE 17.01.4
There is an private IPSec server in the cloud

Configuration #1 (LEDE router):
I can easily connect to this IPsec server from my iMac or iPhone through this router. But connecting from Windows 10 laptop which is actually Dell XPS13 generates well-known error 809.

Configuration #2 (Mobile phone in personal hotspot mode):
But the strange thing here is that connecting IPSec from this laptop through mobile phone in personal hotspot mode goes well.
So this issue maybe laying somewhere between Windows 10 and LEDE router.

I've visited many webpages describing similar issues and most of advices are dealing with MTU settings. Accoring to sniffer on Windows side, IKE_AUTH payload length is 1896 bytes, which is larger than 1500.

Tried to incerease/decrease MTU in LEDE GUI and on Windows size but no luck.
It looks like LEDE can't perform what Windows asks him to do, or Windows is asking for something tricky.

Please can you advice anything to do in this case?

lleachii · May 9, 2018, 3:38pm

If you cloud-based VPN server is behind a NAT, have you see this:

This won't work, ISPs are usually 1500 MTU.

Or it could be your ISP, as your iMac and iPhone still work. I say it's Windows.

What happens if you plug the laptop directly into the Internet connection to test (make sure your Windows firewall is turned on)?

Yojimbo · May 9, 2018, 8:00pm

Thank you for this link, my cloud-based VPN server is behind a NAT, but it's Ubuntu Server, not Windows
Anyway, I've already tried these registry keys on my Windows laptop.

Yes, my ISP's MTU is 1500.

Very useful advice. Thank you! Just tried to connect Windows laptop directly to ISP cable.
And IPSec is not working too!
Does it mean that root cause is somewhere between Windows laptop and ISP hardware?

lleachii · May 9, 2018, 9:32pm

Again, it's probably just an issue with NAT.

It definitely means it isn't your router...but you haven't accounted for the fact your Apple devices work in both setups. It's likely a packet that your Windows software needs to receive.

Let me try to help you again...this time, with a non-OS specific link (the issue is NAT, not WINDOWS):

https://www.networkworld.com/article/2288666/lan-wan/chapter-4--common-ipsec-vpn-issues.html?page=6

jeff · May 9, 2018, 10:16pm

Yes, IPv4 IPsec is incompatible with NAT, as NAT rewrites headers, violating the integrity checks of the security layer. It is possible to use NAT-T, but the recommended course of action is to remove NAT from IPsec (period).

It is also possible that the reason your iPhone and Mac work is that they are using IPv6, which is generally used without NAT.

mpa · May 9, 2018, 10:17pm

It might help to compare successful and failed IKE negotiations using a traffic dump.
Look at IKE fragmentation, IP fragmentation, DF bit, and perhaps ICMP failure messages.
To watch how packets go missing, dump at your client and the Ubuntu server in parallel.

Can you retrieve the IKE encryption keys from one of the peers?
Those allow you to see cleartext IKE messages with wireshark.
If you are using strongswan, an IKE loglevel of 4 reveals the keys.

Yojimbo · May 10, 2018, 7:39pm

Thank you for pointing on this. I'm too old for these IPv6 things, but I'm sure that Mac and iPhone are connecting to IPv4 address provided with certificate. There is even a tcpdump from Mac example below.

Thank you! You are right, it's Strongswan. I've captured dumps and logs in two cases: Windows (not ok) and Mac (ok)
At least one thing looks strange in tcpdump on VPN server - with Mac we get " (n: prot_id=#0 type=16430(status))", and with Windows we get some garbage and then traffic stops.

“MY_VPN_SERVER” stands for my server real IP address below.

Windows scenario

Tcpdump on VPN server side

root@algo# tcpdump -vv  -n dst port 500 or dst port 4500
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:15:44.634121 IP (tos 0x28, ttl 110, id 3010, offset 0, flags [none], proto UDP (17), length 372)
    178.215.84.88.4273 > MY_VPN_SERVER.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie aee335df7ee476bc->0000000000000000: parent_sa ikev2_init[I]:
    (sa: len=44
        (p: #1 protoid=isakmp transform=4 len=44
            (t: #1 type=encr id=aes (type=keylen value=0080))
            (t: #2 type=integ id=#13 )
            (t: #3 type=prf id=#6 )
            (t: #4 type=dh id=#19 )))
    (v2ke: len=64 group=#19)
    (nonce: len=48 nonce=(beef93d8789f3a0407eb9bb342bec8cdfa42414e2fe3543677ae037e87b97af85b5cdaa7ebe8d1d5b6c292cf4934d288) )
    (n: prot_id=#0 type=16388(nat_detection_source_ip))
    (n: prot_id=#0 type=16389(nat_detection_destination_ip))
    (v2vid: len=20 vid=.+Qi...}|......a....)
    (v2vid: len=16 vid=.....A.......U. )
    (v2vid: len=16 vid=&$M8..a..*6.....)
    (v2vid: len=20 vid=.R.......I...[*Q....)

Wireshark on Windows laptop side

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       10.0.0.142            MY_VPN_SERVER         ISAKMP   386    IKE_SA_INIT MID=00 Initiator Request

Frame 1: 386 bytes on wire (3088 bits), 386 bytes captured (3088 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
User Datagram Protocol, Src Port: 500, Dst Port: 500
Internet Security Association and Key Management Protocol

No.     Time           Source                Destination           Protocol Length Info
      3 0.103427       10.0.0.142            MY_VPN_SERVER         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=0bc3) [Reassembled in #4]

Frame 3: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
Data (1480 bytes)

No.     Time           Source                Destination           Protocol Length Info
      4 0.103442       10.0.0.142            MY_VPN_SERVER         ISAKMP   462    IKE_AUTH MID=01 Initiator Request

Frame 4: 462 bytes on wire (3696 bits), 462 bytes captured (3696 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
User Datagram Protocol, Src Port: 4500, Dst Port: 4500
UDP Encapsulation of IPsec Packets
Internet Security Association and Key Management Protocol

No.     Time           Source                Destination           Protocol Length Info
      5 1.116230       10.0.0.142            MY_VPN_SERVER         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=0bc4) [Reassembled in #6]

Frame 5: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
Data (1480 bytes)

No.     Time           Source                Destination           Protocol Length Info
      6 1.116296       10.0.0.142            MY_VPN_SERVER         ISAKMP   462    IKE_AUTH MID=01 Initiator Request

Frame 6: 462 bytes on wire (3696 bits), 462 bytes captured (3696 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
User Datagram Protocol, Src Port: 4500, Dst Port: 4500
UDP Encapsulation of IPsec Packets
Internet Security Association and Key Management Protocol

No.     Time           Source                Destination           Protocol Length Info
      7 2.117666       10.0.0.142            MY_VPN_SERVER         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=0bc5) [Reassembled in #8]

Frame 7: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
Data (1480 bytes)

No.     Time           Source                Destination           Protocol Length Info
      8 2.117711       10.0.0.142            MY_VPN_SERVER         ISAKMP   462    IKE_AUTH MID=01 Initiator Request

Frame 8: 462 bytes on wire (3696 bits), 462 bytes captured (3696 bits) on interface 0
Ethernet II, Src: D-LinkIn_d4:92:b3 (c4:12:f5:d4:92:b3), Dst: Tp-LinkT_fc:0e:9c (c0:4a:00:fc:0e:9c)
Internet Protocol Version 4, Src: 10.0.0.142, Dst: MY_VPN_SERVER
User Datagram Protocol, Src Port: 4500, Dst Port: 4500
UDP Encapsulation of IPsec Packets
Internet Security Association and Key Management Protocol

Strongswan logs

May 10 21:15:44 algo charon: 09[NET] received packet: from 178.215.84.88[4273] to MY_VPN_SERVER[500]
May 10 21:15:44 algo charon: 09[NET] waiting for data on sockets
May 10 21:15:44 algo charon: 04[MGR] checkout IKE_SA by message
May 10 21:15:44 algo charon: 04[MGR] created IKE_SA (unnamed)[6]
May 10 21:15:44 algo charon: 04[NET] received packet: from 178.215.84.88[4273] to MY_VPN_SERVER[500] (344 bytes)
May 10 21:15:44 algo charon: 04[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) V V V V ]
May 10 21:15:44 algo charon: 04[CFG] looking for an ike config for MY_VPN_SERVER...178.215.84.88
May 10 21:15:44 algo charon: 04[CFG]   candidate: %any...%any, prio 28
May 10 21:15:44 algo charon: 04[CFG] found matching ike config: %any...%any with prio 28
May 10 21:15:44 algo charon: 04[IKE] received MS NT5 ISAKMPOAKLEY v9 vendor ID
May 10 21:15:44 algo charon: 04[IKE] received MS-Negotiation Discovery Capable vendor ID
May 10 21:15:44 algo charon: 04[IKE] received Vid-Initial-Contact vendor ID
May 10 21:15:44 algo charon: 04[ENC] received unknown vendor ID: 01:52:8b:bb:c0:06:96:12:18:49:ab:9a:1c:5b:2a:51:00:00:00:02
May 10 21:15:44 algo charon: 04[IKE] 178.215.84.88 is initiating an IKE_SA
May 10 21:15:44 algo charon: 04[IKE] IKE_SA (unnamed)[6] state change: CREATED => CONNECTING
May 10 21:15:44 algo charon: 04[CFG] selecting proposal:
May 10 21:15:44 algo charon: 04[CFG]   no acceptable ENCRYPTION_ALGORITHM found
May 10 21:15:44 algo charon: 04[CFG] selecting proposal:
May 10 21:15:44 algo charon: 04[CFG]   no acceptable PSEUDO_RANDOM_FUNCTION found
May 10 21:15:44 algo charon: 04[CFG] selecting proposal:
May 10 21:15:44 algo charon: 04[CFG]   proposal matches
May 10 21:15:44 algo charon: 04[CFG] received proposals: IKE:AES_CBC_128/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/ECP_256
May 10 21:15:44 algo charon: 04[CFG] configured proposals: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_512/ECP_256, IKE:AES_CBC_128/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/ECP_256, IKE:AES_CBC_128/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/ECP_256
May 10 21:15:44 algo charon: 04[CFG] selected proposal: IKE:AES_CBC_128/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/ECP_256
May 10 21:15:44 algo charon: 04[IKE] natd_chunk => 22 bytes @ 0x7f6550002430
May 10 21:15:44 algo charon: 04[IKE]    0: AE E3 35 DF 7E E4 76 BC 00 00 00 00 00 00 00 00  ..5.~.v.........
May 10 21:15:44 algo charon: 04[IKE]   16: 50 D3 B2 43 01 F4                                P..C..
May 10 21:15:44 algo charon: 04[IKE] natd_hash => 20 bytes @ 0x7f6550002310
May 10 21:15:44 algo charon: 04[IKE]    0: A7 61 93 89 9F 38 93 C2 A0 51 4C 90 51 BB 7A 0C  .a...8...QL.Q.z.
May 10 21:15:44 algo charon: 04[IKE]   16: 1C 25 4F A4                                      .%O.
May 10 21:15:44 algo charon: 04[IKE] natd_chunk => 22 bytes @ 0x7f6550002430
May 10 21:15:44 algo charon: 04[IKE]    0: AE E3 35 DF 7E E4 76 BC 00 00 00 00 00 00 00 00  ..5.~.v.........
May 10 21:15:44 algo charon: 04[IKE]   16: B2 D7 54 58 10 B1                                ..TX..
May 10 21:15:44 algo charon: 04[IKE] natd_hash => 20 bytes @ 0x7f6550002910
May 10 21:15:44 algo charon: 04[IKE]    0: C7 39 E7 BA 38 65 26 75 40 4C 52 44 FA CE 03 7A  .9..8e&u@LRD...z
May 10 21:15:44 algo charon: 04[IKE]   16: 6B 94 43 B1                                      k.C.
May 10 21:15:44 algo charon: 04[IKE] precalculated src_hash => 20 bytes @ 0x7f6550002910
May 10 21:15:44 algo charon: 04[IKE]    0: C7 39 E7 BA 38 65 26 75 40 4C 52 44 FA CE 03 7A  .9..8e&u@LRD...z
May 10 21:15:44 algo charon: 04[IKE]   16: 6B 94 43 B1                                      k.C.
May 10 21:15:44 algo charon: 04[IKE] precalculated dst_hash => 20 bytes @ 0x7f6550002310
May 10 21:15:44 algo charon: 04[IKE]    0: A7 61 93 89 9F 38 93 C2 A0 51 4C 90 51 BB 7A 0C  .a...8...QL.Q.z.
May 10 21:15:44 algo charon: 04[IKE]   16: 1C 25 4F A4                                      .%O.
May 10 21:15:44 algo charon: 04[IKE] received src_hash => 20 bytes @ 0x7f6550001540
May 10 21:15:44 algo charon: 04[IKE]    0: 7C 47 73 92 27 84 89 E4 11 21 8B 49 40 44 B0 71  |Gs.'....!.I@D.q
May 10 21:15:44 algo charon: 04[IKE]   16: 9C 87 21 07                                      ..!.
May 10 21:15:44 algo charon: 04[IKE] received dst_hash => 20 bytes @ 0x7f6550001660
May 10 21:15:44 algo charon: 04[IKE]    0: A7 61 93 89 9F 38 93 C2 A0 51 4C 90 51 BB 7A 0C  .a...8...QL.Q.z.
May 10 21:15:44 algo charon: 04[IKE]   16: 1C 25 4F A4                                      .%O.
May 10 21:15:44 algo charon: 04[IKE] remote host is behind NAT
May 10 21:15:44 algo charon: 04[IKE] shared Diffie Hellman secret => 32 bytes @ 0x7f6550002080
May 10 21:15:44 algo charon: 04[IKE]    0: 1D C2 4F 10 B4 34 0C EE D7 E8 00 0E F7 76 B5 B6  ..O..4.......v..
May 10 21:15:44 algo charon: 04[IKE]   16: 9D DA 3E 07 EE 28 E9 4A B9 A2 4F 9C 7B 92 87 7B  ..>..(.J..O.{..{
May 10 21:15:44 algo charon: 04[IKE] SKEYSEED => 48 bytes @ 0x7f6550004760
May 10 21:15:44 algo charon: 04[IKE]    0: 49 2D 0B BD CE 8A 5D 6C 15 48 D6 EE C3 E9 ED FE  I-....]l.H......
May 10 21:15:44 algo charon: 04[IKE]   16: C1 36 5E BB E7 32 3C C7 02 78 03 68 C0 38 C3 F6  .6^..2<..x.h.8..
May 10 21:15:44 algo charon: 04[IKE]   32: 90 D6 4A FA 6B 4E 97 A2 36 0E E6 B4 9A 8F 58 64  ..J.kN..6.....Xd
May 10 21:15:44 algo charon: 04[IKE] Sk_d secret => 48 bytes @ 0x7f6550002c50
May 10 21:15:44 algo charon: 04[IKE]    0: 3E 0C 86 F3 55 0A 92 28 8A 3D A9 BF F2 7A 2D 00  >...U..(.=...z-.
May 10 21:15:44 algo charon: 04[IKE]   16: 55 80 02 85 C2 21 ED 7E D9 32 49 39 8D 56 1D 6F  U....!.~.2I9.V.o
May 10 21:15:44 algo charon: 04[IKE]   32: 4F FB 45 C9 9F 1E 8E 6E 1A 96 38 5D 00 3E 1A 23  O.E....n..8].>.#
May 10 21:15:44 algo charon: 04[IKE] Sk_ai secret => 48 bytes @ 0x7f65500022c0
May 10 21:15:44 algo charon: 04[IKE]    0: 67 2A 32 D6 D1 BA 13 92 B7 AA C3 44 DC AA B2 C0  g*2........D....
May 10 21:15:44 algo charon: 04[IKE]   16: 47 3B 04 C3 45 43 20 08 D1 08 2E 22 A1 51 F7 6B  G;..EC ....".Q.k
May 10 21:15:44 algo charon: 04[IKE]   32: 3E 9F F2 A0 25 7F 7B 0C 77 F0 F5 39 96 90 8F D0  >...%.{.w..9....
May 10 21:15:44 algo charon: 04[IKE] Sk_ar secret => 48 bytes @ 0x7f65500022c0
May 10 21:15:44 algo charon: 04[IKE]    0: 68 2C C5 E5 FB 82 73 82 5C E6 E5 07 61 AC 62 D1  h,....s.\...a.b.
May 10 21:15:44 algo charon: 04[IKE]   16: 44 4F BA 47 BF 75 CE C0 8C C1 0B 5D 2B 56 FF 67  DO.G.u.....]+V.g
May 10 21:15:44 algo charon: 04[IKE]   32: EB 61 76 5C 5C 79 C0 7B 0F 5F F1 90 67 64 B1 12  .av\\y.{._..gd..
May 10 21:15:44 algo charon: 04[IKE] Sk_ei secret => 16 bytes @ 0x7f6550003490
May 10 21:15:44 algo charon: 04[IKE]    0: 8F F0 6A 51 A3 08 3F E7 72 7A 1F B0 D2 6F 31 6E  ..jQ..?.rz...o1n
May 10 21:15:44 algo charon: 04[IKE] Sk_er secret => 16 bytes @ 0x7f6550003490
May 10 21:15:44 algo charon: 04[IKE]    0: B9 DB 77 03 08 7B E8 F2 E9 AE B4 69 79 90 2B 53  ..w..{.....iy.+S
May 10 21:15:44 algo charon: 04[IKE] Sk_pi secret => 48 bytes @ 0x7f65500022f0
May 10 21:15:44 algo charon: 04[IKE]    0: D5 10 C0 6F 34 1F A7 C6 6A B6 CC 54 CD 9B 01 32  ...o4...j..T...2
May 10 21:15:44 algo charon: 04[IKE]   16: 85 D4 2A 19 FF 2D 67 CA C7 BC 8F 06 DB 48 FF DD  ..*..-g......H..
May 10 21:15:44 algo charon: 04[IKE]   32: 61 EC 49 71 CD CA BC 76 79 FB 9C 2D DF 74 96 51  a.Iq...vy..-.t.Q
May 10 21:15:44 algo charon: 04[IKE] Sk_pr secret => 48 bytes @ 0x7f6550002980
May 10 21:15:44 algo charon: 04[IKE]    0: 7C 8D A0 25 45 31 16 A2 2E BD A8 84 3F 35 CD 11  |..%E1......?5..
May 10 21:15:44 algo charon: 04[IKE]   16: E6 57 2C 35 82 91 09 F2 6A 62 DB 8A FF 6A 0E D1  .W,5....jb...j..
May 10 21:15:44 algo charon: 04[IKE]   32: D4 96 D9 1D 72 BB D7 99 79 2B 25 5E 4C 18 17 A7  ....r...y+%^L...
May 10 21:15:44 algo charon: 04[IKE] natd_chunk => 22 bytes @ 0x7f6550001270
May 10 21:15:44 algo charon: 04[IKE]    0: AE E3 35 DF 7E E4 76 BC 28 44 34 27 91 73 D8 A1  ..5.~.v.(D4'.s..
May 10 21:15:44 algo charon: 04[IKE]   16: 50 D3 B2 43 01 F4                                P..C..
May 10 21:15:44 algo charon: 04[IKE] natd_hash => 20 bytes @ 0x7f6550002930
May 10 21:15:44 algo charon: 04[IKE]    0: 06 C3 0B 2A A6 0F 30 12 0A 12 B2 68 82 44 91 C1  ...*..0....h.D..
May 10 21:15:44 algo charon: 04[IKE]   16: AC 6F 49 1C                                      .oI.
May 10 21:15:44 algo charon: 04[IKE] natd_chunk => 22 bytes @ 0x7f65500033d0
May 10 21:15:44 algo charon: 04[IKE]    0: AE E3 35 DF 7E E4 76 BC 28 44 34 27 91 73 D8 A1  ..5.~.v.(D4'.s..
May 10 21:15:44 algo charon: 04[IKE]   16: B2 D7 54 58 10 B1                                ..TX..
May 10 21:15:44 algo charon: 04[IKE] natd_hash => 20 bytes @ 0x7f65500028f0
May 10 21:15:44 algo charon: 04[IKE]    0: 90 AF 61 3D 86 9B 8A 1B 88 66 72 85 FD D2 8F AF  ..a=.....fr.....
May 10 21:15:44 algo charon: 04[IKE]   16: 12 A2 B4 62                                      ...b
May 10 21:15:44 algo charon: 04[IKE] sending cert request for "CN=MY_VPN_SERVER"
May 10 21:15:44 algo charon: 04[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
May 10 21:15:44 algo charon: 04[NET] sending packet: from MY_VPN_SERVER[500] to 178.215.84.88[4273] (273 bytes)
May 10 21:15:44 algo charon: 10[NET] sending packet: from MY_VPN_SERVER[500] to 178.215.84.88[4273]
May 10 21:15:44 algo charon: 04[MGR] checkin IKE_SA (unnamed)[6]
May 10 21:15:44 algo charon: 04[MGR] check-in of IKE_SA successful.
May 10 21:16:14 algo charon: 01[MGR] checkout IKE_SA
May 10 21:16:14 algo charon: 01[MGR] IKE_SA (unnamed)[6] successfully checked out
May 10 21:16:14 algo charon: 01[JOB] deleting half open IKE_SA after timeout
May 10 21:16:14 algo charon: 01[MGR] checkin and destroy IKE_SA (unnamed)[6]
May 10 21:16:14 algo charon: 01[IKE] IKE_SA (unnamed)[6] state change: CONNECTING => DESTROYING
May 10 21:16:14 algo charon: 01[MGR] check-in and destroy of IKE_SA successful

Mac scenario

Tcpdump on VPN server side

# tcpdump -vv  -n dst port 500 or dst port 4500
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
21:49:30.049936 IP (tos 0x28, ttl 46, id 32486, offset 0, flags [none], proto UDP (17), length 268)
    178.215.84.88.5104 > MY_VPN_SERVER.500: [udp sum ok] isakmp 2.0 msgid 00000000 cookie 322ae46c75c1389c->0000000000000000: parent_sa ikev2_init[I]:
    (sa: len=36
        (p: #1 protoid=isakmp transform=3 len=36
            (t: #1 type=encr id=#20 (type=keylen value=0080))
            (t: #2 type=prf id=#7 )
            (t: #3 type=dh id=#19 )))
    (v2ke: len=64 group=#19)
    (nonce: len=32 nonce=(fb4af9e55114abd22b28b9488a0f8171ba6ba6da84dee49274d37c968c67a717) )
    (n: prot_id=#0 type=16388(nat_detection_source_ip))
    (n: prot_id=#0 type=16389(nat_detection_destination_ip))
    (n: prot_id=#0 type=16430(status))
21:49:30.181354 IP (tos 0x28, ttl 46, id 32694, offset 0, flags [none], proto UDP (17), length 936)
    178.215.84.88.5108 > MY_VPN_SERVER.4500: [udp sum ok] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 322ae46c75c1389c->4645f99fb7a8bb1c: child_sa  ikev2_auth[I]:
    (v2e: len=872)
21:49:30.320001 IP (tos 0x28, ttl 46, id 9182, offset 0, flags [none], proto UDP (17), length 148)
    178.215.84.88.5108 > MY_VPN_SERVER.4500: [no cksum] UDP-encap: ESP(spi=0xca55d965,seq=0x1), length 120

....

tcpdump on Mac side

# tcpdump -n -vv dst host MY_VPN_SERVER 
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
21:49:30.003657 IP (tos 0x0, ttl 64, id 32486, offset 0, flags [none], proto UDP (17), length 268, bad cksum 0 (->ede2)!)
    10.0.0.2.500 > MY_VPN_SERVER.500: [bad udp cksum 0x0e22 -> 0xba56!] isakmp 2.0 msgid 00000000 cookie 322ae46c75c1389c->0000000000000000: parent_sa ikev2_init[I]:
    (sa: len=36
        (p: #1 protoid=isakmp transform=3 len=36
            (t: #1 type=encr id=#20 (type=keylen value=0080))
            (t: #2 type=prf id=#7 )
            (t: #3 type=dh id=#19 )))
    (v2ke: len=64 group=#19)
    (nonce: len=32 nonce=(fb4af9e55114abd22b28b9488a0f8171ba6ba6da84dee49274d37c968c67a717) )
    (n: prot_id=#0 type=16388(nat_detection_source_ip))
    (n: prot_id=#0 type=16389(nat_detection_destination_ip))
    (n: prot_id=#0 type=16430(status))
21:49:30.088368 IP (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->2d9c)!)
    10.0.0.2.51363 > MY_VPN_SERVER.22: Flags [.], cksum 0x0d3f (incorrect -> 0x5520), seq 2273184550, ack 3447377359, win 19728, options [nop,nop,TS val 856611316 ecr 1426355], length 0
21:49:30.139511 IP (tos 0x0, ttl 64, id 32694, offset 0, flags [none], proto UDP (17), length 936, bad cksum 0 (->ea76)!)
    10.0.0.2.4500 > MY_VPN_SERVER.4500: [bad udp cksum 0x10be -> 0x7ace!] NONESP-encap: isakmp 2.0 msgid 00000001 cookie 322ae46c75c1389c->4645f99fb7a8bb1c: child_sa  ikev2_auth[I]:
    (v2e: len=872)
21:49:30.219341 IP (tos 0x10, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 0 (->2d9c)!)
    10.0.0.2.51363 > MY_VPN_SERVER.22: Flags [.], cksum 0x0d3f (incorrect -> 0x532b), seq 0, ack 325, win 19742, options [nop,nop,TS val 856611446 ecr 1426388], length 0
21:49:30.278275 IP (tos 0x0, ttl 64, id 9182, offset 0, flags [none], proto UDP (17), length 148, bad cksum 0 (->4963)!)
    10.0.0.2.4500 > MY_VPN_SERVER.4500: [no cksum] UDP-encap: ESP(spi=0xca55d965,seq=0x1), length 120
21:49:30.278333 IP (tos 0x0, ttl 64, id 42942, offset 0, flags [none], proto UDP (17), length 144, bad cksum 0 (->c586)!)
    10.0.0.2.4500 > MY_VPN_SERVER.4500: [no cksum] UDP-encap: ESP(spi=0xca55d965,seq=0x2), length 116
21:49:30.278360 IP (tos 0x0, ttl 64, id 56518, offset 0, flags [none], proto UDP (17), length 148, bad cksum 0 (->907a)!)
    10.0.0.2.4500 > MY_VPN_SERVER.4500: [no cksum] UDP-encap: ESP(spi=0xca55d965,seq=0x3), length 120

...

Strongswan logs

May 10 21:49:30 algo charon: 09[NET] received packet: from 178.215.84.88[5104] to MY_VPN_SERVER[500]
May 10 21:49:30 algo charon: 09[NET] waiting for data on sockets
May 10 21:49:30 algo charon: 05[MGR] checkout IKE_SA by message
May 10 21:49:30 algo charon: 05[MGR] created IKE_SA (unnamed)[10]
May 10 21:49:30 algo charon: 05[NET] received packet: from 178.215.84.88[5104] to MY_VPN_SERVER[500] (240 bytes)
May 10 21:49:30 algo charon: 05[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ]
May 10 21:49:30 algo charon: 05[CFG] looking for an ike config for MY_VPN_SERVER...178.215.84.88
May 10 21:49:30 algo charon: 05[CFG]   candidate: %any...%any, prio 28
May 10 21:49:30 algo charon: 05[CFG] found matching ike config: %any...%any with prio 28
May 10 21:49:30 algo charon: 05[IKE] 178.215.84.88 is initiating an IKE_SA
May 10 21:49:30 algo charon: 05[IKE] IKE_SA (unnamed)[10] state change: CREATED => CONNECTING
May 10 21:49:30 algo charon: 05[CFG] selecting proposal:
May 10 21:49:30 algo charon: 05[CFG]   proposal matches
May 10 21:49:30 algo charon: 05[CFG] received proposals: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_512/ECP_256
May 10 21:49:30 algo charon: 05[CFG] configured proposals: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_512/ECP_256, IKE:AES_CBC_128/HMAC_SHA2_512_256/PRF_HMAC_SHA2_512/ECP_256, IKE:AES_CBC_128/HMAC_SHA2_384_192/PRF_HMAC_SHA2_384/ECP_256
May 10 21:49:30 algo charon: 05[CFG] selected proposal: IKE:AES_GCM_16_128/PRF_HMAC_SHA2_512/ECP_256
May 10 21:49:30 algo charon: 05[IKE] natd_chunk => 22 bytes @ 0x7f6554002830
May 10 21:49:30 algo charon: 05[IKE]    0: 32 2A E4 6C 75 C1 38 9C 00 00 00 00 00 00 00 00  2*.lu.8.........
May 10 21:49:30 algo charon: 05[IKE]   16: 50 D3 B2 43 01 F4                                P..C..
May 10 21:49:30 algo charon: 05[IKE] natd_hash => 20 bytes @ 0x7f6554002490
May 10 21:49:30 algo charon: 05[IKE]    0: BA CC C7 28 2F 45 A0 6E 34 55 65 5B 53 FB C2 F6  ...(/E.n4Ue[S...
May 10 21:49:30 algo charon: 05[IKE]   16: 4B 77 A7 BA                                      Kw..
May 10 21:49:30 algo charon: 05[IKE] natd_chunk => 22 bytes @ 0x7f6554002830
May 10 21:49:30 algo charon: 05[IKE]    0: 32 2A E4 6C 75 C1 38 9C 00 00 00 00 00 00 00 00  2*.lu.8.........
May 10 21:49:30 algo charon: 05[IKE]   16: B2 D7 54 58 13 F0                                ..TX..
May 10 21:49:30 algo charon: 05[IKE] natd_hash => 20 bytes @ 0x7f65540027f0
May 10 21:49:30 algo charon: 05[IKE]    0: FF 34 5F 51 11 3E 28 84 60 31 60 BA 9D 4D 07 82  .4_Q.>(.`1`..M..
May 10 21:49:30 algo charon: 05[IKE]   16: 8A B8 9C B6                                      ....
May 10 21:49:30 algo charon: 05[IKE] precalculated src_hash => 20 bytes @ 0x7f65540027f0
May 10 21:49:30 algo charon: 05[IKE]    0: FF 34 5F 51 11 3E 28 84 60 31 60 BA 9D 4D 07 82  .4_Q.>(.`1`..M..
May 10 21:49:30 algo charon: 05[IKE]   16: 8A B8 9C B6                                      ....
May 10 21:49:30 algo charon: 05[IKE] precalculated dst_hash => 20 bytes @ 0x7f6554002490
May 10 21:49:30 algo charon: 05[IKE]    0: BA CC C7 28 2F 45 A0 6E 34 55 65 5B 53 FB C2 F6  ...(/E.n4Ue[S...
May 10 21:49:30 algo charon: 05[IKE]   16: 4B 77 A7 BA                                      Kw..
May 10 21:49:30 algo charon: 05[IKE] received src_hash => 20 bytes @ 0x7f6554006f70
May 10 21:49:30 algo charon: 05[IKE]    0: 3F 97 92 01 21 09 C9 CF 64 03 9A 2D 91 41 94 E9  ?...!...d..-.A..
May 10 21:49:30 algo charon: 05[IKE]   16: 1D 7B B5 11                                      .{..
May 10 21:49:30 algo charon: 05[IKE] received dst_hash => 20 bytes @ 0x7f6554006380
May 10 21:49:30 algo charon: 05[IKE]    0: BA CC C7 28 2F 45 A0 6E 34 55 65 5B 53 FB C2 F6  ...(/E.n4Ue[S...
May 10 21:49:30 algo charon: 05[IKE]   16: 4B 77 A7 BA                                      Kw..
May 10 21:49:30 algo charon: 05[IKE] remote host is behind NAT
May 10 21:49:30 algo charon: 05[IKE] shared Diffie Hellman secret => 32 bytes @ 0x7f6554007c30
May 10 21:49:30 algo charon: 05[IKE]    0: CB 5F F8 FD 62 56 3B C2 AD A2 E3 A6 30 53 8C F6  ._..bV;.....0S..
May 10 21:49:30 algo charon: 05[IKE]   16: 94 EC 49 03 3E CC C3 84 AD 73 27 FB FB C5 C9 7F  ..I.>....s'.....
May 10 21:49:30 algo charon: 05[IKE] SKEYSEED => 64 bytes @ 0x7f6554002070
May 10 21:49:30 algo charon: 05[IKE]    0: F1 53 70 B6 E7 1B A3 7E 5A 15 FE D6 D3 16 C1 40  .Sp....~Z......@
May 10 21:49:30 algo charon: 05[IKE]   16: 26 A5 34 C6 0C 43 33 5E D7 15 F8 50 C3 FD 73 A2  &.4..C3^...P..s.
May 10 21:49:30 algo charon: 05[IKE]   32: C2 30 A5 22 4F 7B 4D C6 18 C7 58 BD 79 9A 82 A7  .0."O{M...X.y...
May 10 21:49:30 algo charon: 05[IKE]   48: 6A 8E F8 45 01 4D 77 A6 FF 43 4C 29 BA F4 C4 93  j..E.Mw..CL)....
May 10 21:49:30 algo charon: 05[IKE] Sk_d secret => 64 bytes @ 0x7f6554002750
May 10 21:49:30 algo charon: 05[IKE]    0: 03 4E 83 A2 9F 29 36 05 F2 8F 38 22 7C 99 02 95  .N...)6...8"|...
May 10 21:49:30 algo charon: 05[IKE]   16: B1 48 F8 8E 0A A4 1C BC 83 E2 BC F0 2F CD B3 28  .H........../..(
May 10 21:49:30 algo charon: 05[IKE]   32: 5A 9B 4E 15 4D 9A 20 66 B7 04 A5 55 AF E6 C2 C7  Z.N.M. f...U....
May 10 21:49:30 algo charon: 05[IKE]   48: 86 51 E7 C5 8D 63 26 4C EE 53 92 5C 90 A0 CE 76  .Q...c&L.S.\...v
May 10 21:49:30 algo charon: 05[IKE] Sk_ei secret => 20 bytes @ 0x7f6554002910
May 10 21:49:30 algo charon: 05[IKE]    0: 02 4C 7F FE 30 DF 75 7A A3 80 DE EE A5 C2 02 7E  .L..0.uz.......~
May 10 21:49:30 algo charon: 05[IKE]   16: 9E 07 A7 DF                                      ....
May 10 21:49:30 algo charon: 05[IKE] Sk_er secret => 20 bytes @ 0x7f6554002910
May 10 21:49:30 algo charon: 05[IKE]    0: FF 0A 75 B9 BE 6B 0E A1 4B 5B F4 B2 5B 93 4C 4B  ..u..k..K[..[.LK
May 10 21:49:30 algo charon: 05[IKE]   16: C6 64 0B 4D                                      .d.M
May 10 21:49:30 algo charon: 05[IKE] Sk_pi secret => 64 bytes @ 0x7f6554002930
May 10 21:49:30 algo charon: 05[IKE]    0: DF 60 BF DE F7 CE CF 95 4A 28 FE 33 A0 51 24 B7  .`......J(.3.Q$.
May 10 21:49:30 algo charon: 05[IKE]   16: ED FE F9 4E FE E7 D3 08 FD F2 12 C2 55 F5 50 32  ...N........U.P2
May 10 21:49:30 algo charon: 05[IKE]   32: 10 00 12 E1 EF E0 9E A4 B1 0C C4 77 4F 39 58 28  ...........wO9X(
May 10 21:49:30 algo charon: 05[IKE]   48: 70 05 0B 0D 33 B9 B3 BE 5D D9 43 2F C1 85 E0 C8  p...3...].C/....
May 10 21:49:30 algo charon: 05[IKE] Sk_pr secret => 64 bytes @ 0x7f65540075e0
May 10 21:49:30 algo charon: 05[IKE]    0: 92 64 D7 BA C7 12 57 EE F3 4A 8B 8F 4F 67 E7 02  .d....W..J..Og..
May 10 21:49:30 algo charon: 05[IKE]   16: 71 3D 1E 8A 6B 8F B4 E9 09 39 CF 0D B5 4A 1B 3C  q=..k....9...J.<
May 10 21:49:30 algo charon: 05[IKE]   32: A9 F3 6A C5 5E A4 2A 39 B1 59 F3 57 7C 35 47 25  ..j.^.*9.Y.W|5G%
May 10 21:49:30 algo charon: 05[IKE]   48: 68 53 AC C7 74 B8 30 98 ED C7 72 B8 B0 76 E5 CA  hS..t.0...r..v..
May 10 21:49:30 algo charon: 05[IKE] natd_chunk => 22 bytes @ 0x7f65540049b0
May 10 21:49:30 algo charon: 05[IKE]    0: 32 2A E4 6C 75 C1 38 9C 46 45 F9 9F B7 A8 BB 1C  2*.lu.8.FE......
May 10 21:49:30 algo charon: 05[IKE]   16: 50 D3 B2 43 01 F4                                P..C..
May 10 21:49:30 algo charon: 05[IKE] natd_hash => 20 bytes @ 0x7f6554002530
May 10 21:49:30 algo charon: 05[IKE]    0: B8 80 34 31 44 52 3E C7 D5 CB E3 9B A1 F1 88 26  ..41DR>........&
May 10 21:49:30 algo charon: 05[IKE]   16: 6E E1 2E D3                                      n...
May 10 21:49:30 algo charon: 05[IKE] natd_chunk => 22 bytes @ 0x7f6554002850
May 10 21:49:30 algo charon: 05[IKE]    0: 32 2A E4 6C 75 C1 38 9C 46 45 F9 9F B7 A8 BB 1C  2*.lu.8.FE......
May 10 21:49:30 algo charon: 05[IKE]   16: B2 D7 54 58 13 F0                                ..TX..
May 10 21:49:30 algo charon: 05[IKE] natd_hash => 20 bytes @ 0x7f65540076c0
May 10 21:49:30 algo charon: 05[IKE]    0: 78 69 3A F7 67 7F ED 83 E7 0F 12 4C 6C B3 87 BD  xi:.g......Ll...
May 10 21:49:30 algo charon: 05[IKE]   16: E6 26 B5 9F                                      .&..
May 10 21:49:30 algo charon: 05[IKE] sending cert request for "CN=MY_VPN_SERVER"
May 10 21:49:30 algo charon: 05[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
May 10 21:49:30 algo charon: 05[NET] sending packet: from MY_VPN_SERVER[500] to 178.215.84.88[5104] (265 bytes)
May 10 21:49:30 algo charon: 10[NET] sending packet: from MY_VPN_SERVER[500] to 178.215.84.88[5104]
May 10 21:49:30 algo charon: 05[MGR] checkin IKE_SA (unnamed)[10]
May 10 21:49:30 algo charon: 05[MGR] check-in of IKE_SA successful.

....

mpa · May 11, 2018, 12:07am

As you noted, the IKE_AUTH from Windows is larger than 1500 bytes.
The client sends it as two IP fragments which don't arrive at the VPN server.
The IKE_AUTH packet from the Mac is only 936 bytes and does arrive.

Can you determine where the IP fragments are lost, maybe with traceroute?
I suspect they are dropped at the NAT gateway or an intermediate firewall.
Do you have control over this device to fix the issue?

A notify payload type 16430 indicates support for IKEv2 fragmentation (RFC 7383).
It would even solve your problem if it were supported by Windows, but alas it is not.

Yojimbo · May 11, 2018, 7:07pm

Yes, just did a traceroute with 1514 byte packets and it looks like ISP's hardware is the last hop

$ traceroute -d MY_VPN_SERVER 1514
traceroute to MY_VPN_SERVER (MY_VPN_SERVER), 64 hops max, 1514 byte packets
 1  lede (10.0.0.1)  0.785 ms  0.538 ms  0.536 ms
 2  10.27.40.254 (10.27.40.254)  2.384 ms  2.225 ms  6.046 ms
 3  core-m9-j1.speedynet.ru (178.215.79.61)  2.133 ms  1.894 ms  2.081 ms
 4  178.215.65.52 (178.215.65.52)  2.050 ms  1.848 ms  1.921 ms
 5  * * *
 6  * * *

Unfortunately, I don't have any access to this hardware.
But maybe there is way to fix the issue on Windows side?
For instance, would it be possible to limit Windows from sending packets larger than 1500 bytes?

lleachii · May 11, 2018, 7:25pm

Your device should NOT send more than 1500, as that's the maximum packet size on the normal Internet. Yes, it is possible to lower the Maximum Transmission Unit (MTU) of an interface in Windows.

netsh interface ipv4 set subinterface "Local Area Connection" mtu=xxxx store=persistent

Yojimbo · May 11, 2018, 7:34pm

I did this already. I even added registry key that override this "netsh".
According to Windows my MTU size is 1500.
But Wiresharks shows 1514 byte packets during IKE negotitation.

lleachii · May 11, 2018, 7:38pm

That's extremly odd, as the packet is 14 bytes larger than normal Internet...

have you tried lowering the MTU of the LAN and/or WAN on the router

Are you sure "Local Area Connection" is the name of your interface???

mpa · May 11, 2018, 10:34pm

Not necessarily. There could be hops that don't answer any traceroute probes at all.
For comparison, please do another traceroute, with smaller packets that can reach the VPN server.

That's OK. The MTU value does not include the ethernet header. However, Wireshark shows the packet length including the ethernet header, which is 14 bytes long in its basic form.

I haven't seen any indication that an MTU change could help here, so I would not recommend it.

mpa · May 31, 2018, 11:25pm

Apparently Windows 10 v1803 now supports IKEv2 fragmentation. You might want to give this another try after upgrading Windows.