Hello,
TLDR: I have some strange issues establishing TLS connections to a few servers. I have debugged quite a while but I didn't find the cause yet. I would really appreciate if anybody could give me a hint debugging this issue.
Thank you so much in advance.
My setup
Tuxedo Notebook
(behind tailscale tunnel)
|
| (LAN)
V
Beryl-AX (OpenWRT) --> acts as tailscale client
|
| (The internet)
V
Orangepi (Exit Node)
|
|
V
Internet
I captured the traffic simultaneoulsy on the notebook (wireshark) and on the openWrt router (tcpdump). The output seems to be exactly the same. The pattern I've found is, that after Client Hello and another ACK, there is a TCP Previous segment not captured that has always the same sequence no. 4097.
To add more complexity, sometimes it works without problems, but mostly I get the same capture.
At the same time when the TLS handshake for some domains fail, the "other" domains connect normally. This means I can use MS Teams Webclient but not the Skype Webclient etc...
Domains that cause problems sometimes:
Debugging and Packet capturing
I used the following command to trigger the handshake:
$ curl -v https://int.id.bund.de/idp/profile/SAML2/POST/SSO
* Trying 80.245.156.58:443...
* Connected to int.id.bund.de (80.245.156.58) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* SSL connection timeout
* Closing connection 0
curl: (28) SSL connection timeout
Capture of the unsuccessful TLS handshake:
I captured this on my openwrt using tcpdump on eth1
19:18:50.429114 IP (tos 0x0, ttl 64, id 5948, offset 0, flags [DF], proto TCP (6), length 60)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [S], cksum 0xc7ee (correct), seq 1127279642, win 64240, options [mss 1460,sackOK,TS val 1599965908 ecr 0,nop,wscale 7], length 0
19:18:50.698823 IP (tos 0x0, ttl 54, id 0, offset 0, flags [DF], proto TCP (6), length 60)
80.245.156.58.443 > tuxedo.lan.46142: Flags [S.], cksum 0x8020 (correct), seq 2100488049, ack 1127279643, win 65160, options [mss 1420,sackOK,TS val 1591705801 ecr 1599965908,nop,wscale 7], length 0
19:18:50.699896 IP (tos 0x0, ttl 64, id 5949, offset 0, flags [DF], proto TCP (6), length 52)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xaa48 (correct), seq 1, ack 1, win 502, options [nop,nop,TS val 1599966179 ecr 1591705801], length 0
19:18:50.730504 IP (tos 0x0, ttl 64, id 5950, offset 0, flags [DF], proto TCP (6), length 569)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [P.], cksum 0xae24 (correct), seq 1:518, ack 1, win 502, options [nop,nop,TS val 1599966208 ecr 1591705801], length 517
19:18:50.982090 IP (tos 0x0, ttl 54, id 14039, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xa703 (correct), seq 1, ack 518, win 506, options [nop,nop,TS val 1591706088 ecr 1599966208], length 0
19:18:50.992094 IP (tos 0x0, ttl 54, id 14043, offset 0, flags [DF], proto TCP (6), length 711)
80.245.156.58.443 > tuxedo.lan.46142: Flags [P.], cksum 0xa261 (correct), seq 4097:4756, ack 518, win 506, options [nop,nop,TS val 1591706093 ecr 1599966208], length 659
19:18:50.992897 IP (tos 0x0, ttl 64, id 5951, offset 0, flags [DF], proto TCP (6), length 64)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0x840a (correct), seq 518, ack 1, win 502, options [nop,nop,TS val 1599966472 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:19:51.198188 IP (tos 0x0, ttl 64, id 5952, offset 0, flags [DF], proto TCP (6), length 64)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0x98d9 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600026681 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:19:51.466297 IP (tos 0x0, ttl 54, id 14051, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xa721 (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591766574 ecr 1599966472], length 0
19:20:52.633787 IP (tos 0x0, ttl 64, id 5953, offset 0, flags [DF], proto TCP (6), length 64)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xa8d8 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600088121 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:20:52.886862 IP (tos 0x0, ttl 54, id 14053, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xb72f (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591827999 ecr 1599966472], length 0
19:21:54.069822 IP (tos 0x0, ttl 64, id 5954, offset 0, flags [DF], proto TCP (6), length 64)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xb8d7 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600149561 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:21:54.330800 IP (tos 0x0, ttl 54, id 14054, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xc726 (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591889447 ecr 1599966472], length 0
19:22:55.506619 IP (tos 0x0, ttl 64, id 5955, offset 0, flags [DF], proto TCP (6), length 64)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [.], cksum 0xc8d6 (correct), seq 517, ack 1, win 502, options [nop,nop,TS val 1600211001 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:22:55.760588 IP (tos 0x0, ttl 54, id 14056, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0xd72a (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1591950882 ecr 1599966472], length 0
19:23:51.033023 IP (tos 0x0, ttl 64, id 5956, offset 0, flags [DF], proto TCP (6), length 64)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [F.], cksum 0xefe9 (correct), seq 518, ack 1, win 502, options [nop,nop,TS val 1600266531 ecr 1591706088,nop,nop,sack 1 {4097:4756}], length 0
19:23:51.085346 IP (tos 0x0, ttl 54, id 14057, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [F.], cksum 0xff1a (correct), seq 4756, ack 518, win 506, options [nop,nop,TS val 1592006192 ecr 1599966472], length 0
19:23:51.086264 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [R], cksum 0x12ec (correct), seq 1127280160, win 0, length 0
19:23:51.294501 IP (tos 0x0, ttl 54, id 14058, offset 0, flags [DF], proto TCP (6), length 52)
80.245.156.58.443 > tuxedo.lan.46142: Flags [.], cksum 0x6a19 (correct), seq 4757, ack 519, win 506, options [nop,nop,TS val 1592006417 ecr 1600266531], length 0
19:23:51.295290 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
tuxedo.lan.46142 > 80.245.156.58.443: Flags [R], cksum 0x12eb (correct), seq 1127280161, win 0, length 0
And a capture of a successful TLS handshake:
$ curl -v https://int.id.bund.de/idp/profile/SAML2/POST/SSO
* Trying 80.245.156.58:443...
* Connected to int.id.bund.de (80.245.156.58) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=DE; ST=Nordrhein-Westfalen; O=Informationstechnikzentrum Bund; CN=int.id.bund.de
* start date: Jan 10 00:00:00 2024 GMT
* expire date: Jan 9 23:59:59 2025 GMT
* subjectAltName: host "int.id.bund.de" matched cert's "int.id.bund.de"
* issuer: C=NL; O=GEANT Vereniging; CN=GEANT OV RSA CA 4
* SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /idp/profile/SAML2/POST/SSO HTTP/1.1
> Host: int.id.bund.de
> User-Agent: curl/7.81.0
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Thu, 28 Mar 2024 18:15:36 GMT
< Server: Apache
< Set-Cookie: INGRESSCOOKIE=5832166eb30b7d0aa56c661961dd03ca|864f5e10da27136412d8b1c00ef7b324; Path=/idp; Max-Age=3600; Secure; HttpOnly; SameSite=None
< Set-Cookie: JSESSIONID=69917BC01A7E8823814ACA00F1D16819; Path=/idp; Secure; HttpOnly; SameSite=None
< Set-Cookie: AL_SESS-S=ARU_nWWoMUG65LCRbcKihD8RPFBDCfwMOO6k4HstEOsCORNsApTbt6UO5T4rdDz9_cNP; Path=/; Secure; HttpOnly
< Content-Length: 0
< Cache-Control: no-store, no-cache, max-age=0
< X-Content-Type-Options: nosniff
< Content-Security-Policy: default-src 'self'; img-src 'self' data:; style-src 'unsafe-inline' 'self'; connect-src 'self' http://127.0.0.1:24727; upgrade-insecure-requests; frame-ancestors 'none'; object-src 'none'
< Strict-Transport-Security: max-age=31536000 ; includeSubDomains; preload
< Location: https://int.id.bund.de/idpError?idp.code=UnableToDecode&url
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Content-Type: text/html;charset=utf-8
<
* Connection #0 to host int.id.bund.de left intact
Firewall config
I use openWRT to connect to a Tailscale exit node. The ethernet port must have a strict
kill switchand must not have direct access to wan. The tailscale connection works very well despite this TLS issue and it works on other devices flawlessly if I use the tailscale client. I also have other tailscale exit nodes, so I guess it is not tailscale related.
I use a similar setup on a gl-inet
mangobut there I use a Wireguard exit node. It works without any problems.
/etc/config/firewall
config defaults
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'REJECT'
option synflood_protect '1'
config zone
option name 'lan'
list network 'lan'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
config zone
option name 'wan'
option output 'ACCEPT'
option forward 'REJECT'
option mtu_fix '1'
option input 'DROP'
option masq '1'
list network 'wan'
list network 'wan6'
list network 'wwan'
list network 'tethering'
config rule
option name 'Allow-DHCP-Renew'
option src 'wan'
option proto 'udp'
option dest_port '68'
option target 'ACCEPT'
option family 'ipv4'
config rule
option name 'Allow-IGMP'
option src 'wan'
option proto 'igmp'
option family 'ipv4'
option target 'ACCEPT'
config rule
option name 'Allow-DHCPv6'
option src 'wan'
option proto 'udp'
option dest_port '546'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-MLD'
option src 'wan'
option proto 'icmp'
option src_ip 'fe80::/10'
list icmp_type '130/0'
list icmp_type '131/0'
list icmp_type '132/0'
list icmp_type '143/0'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Input'
option src 'wan'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
list icmp_type 'router-solicitation'
list icmp_type 'neighbour-solicitation'
list icmp_type 'router-advertisement'
list icmp_type 'neighbour-advertisement'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-ICMPv6-Forward'
option src 'wan'
option dest '*'
option proto 'icmp'
list icmp_type 'echo-request'
list icmp_type 'echo-reply'
list icmp_type 'destination-unreachable'
list icmp_type 'packet-too-big'
list icmp_type 'time-exceeded'
list icmp_type 'bad-header'
list icmp_type 'unknown-header-type'
option limit '1000/sec'
option family 'ipv6'
option target 'ACCEPT'
config rule
option name 'Allow-IPSec-ESP'
option src 'wan'
option dest 'lan'
option proto 'esp'
option target 'ACCEPT'
config rule
option name 'Allow-ISAKMP'
option src 'wan'
option dest 'lan'
option dest_port '500'
option proto 'udp'
option target 'ACCEPT'
config rule
option name 'Support-UDP-Traceroute'
option src 'wan'
option dest_port '33434:33689'
option proto 'udp'
option family 'ipv4'
option target 'REJECT'
option enabled '0'
config include
option path '/etc/firewall.user'
config include 'nat6'
option path '/etc/firewall.nat6'
option reload '1'
config rule 'block_dns'
option name 'block_dns'
option src '*'
option device 'br-+'
option dest_port '53'
option target 'REJECT'
option enabled '0'
config rule 'process_mark'
option name 'process_mark'
option dest '*'
option proto 'all'
option extra '-m owner --gid-owner 65533'
option target 'MARK'
option set_xmark '0x8000/0xc000'
config rule 'wan_in_conn_mark'
option name 'wan_in_conn_mark'
option src 'wan'
option dest '*'
option set_xmark '0x8000/0xc000'
option target 'MARK'
option extra '-j CONNMARK --set-xmark 0x8000/0xc000'
option enabled '0'
config rule 'lan_in_conn_mark_restore'
option name 'lan_in_conn_mark_restore'
option src 'lan'
option dest '*'
option set_xmark '0x8000/0xc000'
option target 'MARK'
option extra '-m connmark --mark 0x8000/0xc000 -j CONNMARK --restore-mark'
option enabled '0'
config rule 'out_conn_mark_restore'
option name 'out_conn_mark_restore'
option dest '*'
option set_xmark '0x8000/0xc000'
option target 'MARK'
option extra '-m connmark --mark 0x8000/0xc000 -j CONNMARK --restore-mark'
option enabled '0'
config include 'swap_wan_in_conn_mark'
option type 'script'
option reload '1'
option path '/etc/firewall.swap_wan_in_conn_mark.sh'
option enabled '0'
config include 'gls2s'
option type 'script'
option path '/var/etc/gls2s.include'
option reload '1'
config include 'glblock'
option type 'script'
option path '/usr/bin/gl_block.sh'
option reload '1'
config zone
option name 'guest'
option network 'guest'
option output 'ACCEPT'
option input 'ACCEPT'
option forward 'REJECT'
config forwarding
option src 'guest'
option dest 'wan'
option enabled '1'
config rule
option name 'Allow-DHCP'
option src 'guest'
option target 'ACCEPT'
option proto 'udp'
option dest_port '67-68'
config rule
option name 'Allow-DNS'
option src 'guest'
option target 'ACCEPT'
option proto 'tcp udp'
option dest_port '53'
config include 'vpn_server_policy'
option type 'script'
option path '/etc/firewall.vpn_server_policy.sh'
option reload '1'
option enabled '1'
config zone
option name 'tailscale'
list device 'tailscale0'
option masq '1'
option input 'ACCEPT'
option output 'ACCEPT'
option forward 'ACCEPT'
config forwarding
option dest 'lan'
option src 'tailscale'
option enabled '1'
config forwarding
option dest 'tailscale'
option src 'lan'
config zone 'wgclient'
option name 'wgclient'
option forward 'DROP'
option output 'ACCEPT'
option mtu_fix '1'
option network 'wgclient'
option input 'DROP'
option masq '1'
option masq6 '1'
option enabled '0'
config forwarding 'lan2wgclient'
option src 'lan'
option dest 'wgclient'
option enabled '0'
config forwarding 'guest2wgclient'
option src 'guest'
option dest 'wgclient'
option enabled '0'
config forwarding
option dest 'lan'
option src 'wgclient'
config forwarding 'wgclient2wan'
option src 'wgclient'
option dest 'wan'
option enabled '0'
config zone
option name 'iot'
option forward 'REJECT'
list network 'IOT'
option output 'ACCEPT'
option input 'REJECT'
config rule
option name 'access cudy'
option dest 'wan'
list dest_ip '192.168.10.1'
option target 'ACCEPT'
list proto 'tcp'
option src 'lan'
option dest_port '80'
config rule 'sambasharewan'
option src 'wan'
option dest_port '137 138 139 445'
option dest_proto 'tcpudp'
option target 'DROP'
config rule 'sambasharelan'
option src 'lan'
option dest_port '137 138 139 445'
option dest_proto 'tcpudp'
option target 'ACCEPT'
config rule 'glnas_ser'
option src 'wan'
option dest_port '6000-6002'
option dest_proto 'tcp'
option target 'DROP'
config rule 'webdav_wan'
option src 'wan'
option dest_port '6008'
option dest_proto 'tcp'
option target 'DROP'
/etc/config/network
config interface 'loopback'
option device 'lo'
option proto 'static'
option ipaddr '127.0.0.1'
option netmask '255.0.0.0'
config globals 'globals'
option ula_prefix 'fdf3:8f9f:00ae::/48'
config device
option name 'br-lan'
option type 'bridge'
list ports 'eth1'
option macaddr '94:83:c4:3d:fa:90'
config device
option name 'eth1'
option macaddr '94:83:c4:3d:fa:90'
config interface 'lan'
option device 'br-lan'
option proto 'static'
option netmask '255.255.255.0'
option ip6assign '60'
option isolate '0'
option ipaddr '192.168.7.1'
config device
option name 'eth0'
option macaddr '94:83:c4:3d:fa:8f'
config interface 'wan'
option device 'eth0'
option proto 'dhcp'
option force_link '0'
option ipv6 '0'
option metric '1'
config interface 'wan6'
option proto 'dhcpv6'
option disabled '1'
option device '@wan'
config interface 'tethering6'
option proto 'dhcpv6'
option disabled '1'
option device '@tethering'
config interface 'wwan6'
option proto 'dhcpv6'
option disabled '1'
option device '@wwan'
config interface 'guest'
option force_link '1'
option type 'bridge'
option proto 'static'
option ipaddr '192.168.9.1'
option netmask '255.255.255.0'
option ip6assign '60'
option multicast_querier '1'
option igmp_snooping '0'
option isolate '0'
option bridge_empty '1'
option disabled '0'
config interface 'wwan'
option proto 'dhcp'
config interface 'modem_1_1_2_6'
option proto 'dhcpv6'
option disabled '1'
option device '@modem_1_1_2'
config rule 'policy_direct_rt'
option lookup 'main'
option suppress_prefixlength '0'
option priority '1100'
config rule 'policy_default_rt_vpn'
option mark '0x8000/0xc000'
option lookup '8000'
option priority '1101'
option invert '1'
config rule6 'policy_direct_rt6'
option lookup 'main'
option suppress_prefixlength '0'
option priority '1100'
config rule6 'policy_default_rt_vpn6'
option mark '0x8000/0xc000'
option lookup '8000'
option priority '1101'
option invert '1'
config interface 'wgclient'
option proto 'wgclient'
option config 'peer_9746'
option disabled '1'
config interface 'IOT'
option proto 'static'
option ipaddr '192.168.90.1'
option netmask '255.255.255.0'
config interface 'vpnLAN'
option proto 'static'
config interface 'novpnlan'
option proto 'static'
config interface 'modem_1_1_6'
option proto 'dhcpv6'
option disabled '1'
option device '@modem_1_1'
config interface 'tethering'
option proto 'dhcp'
option metric '3'
option device 'usb0'
