6in4: pinging works, but other traffic fails

I've got a new ISP that doesn't provide IPv6 support. I decided to solve that using HE tunnelbroker. I've successfully set up the tunnel, but apart from pinging over IPv6 all other traffic (HTTP, HTTPS, FTP, ...) fails.

I've been playing around with unfragmented ICMPv6 packages to try and diagnose the issue, but I'm unable to find any logic in my results. For example: I'm able to ping facebook com with unfragmented packets of up to 1232 bytes, but it starts failing from 1233 bytes. Meanwhile, I can reach tunnelbroker net with larger packets. I've also got a tcpdump of the traffic in my tunnel if that would help anybody to diagnose the issue.

I've already tried the following:

  • set the MTU to 1280 on both sides of the tunnel
  • create a tunnel on another server endpoint of HE
  • use tunnelbroker ch instead
  • downgrade from OpenWrt 19.07.1 to 18.06.3

Can anybody advice me in what I should test or try next?

Output of the pings can be found here: https://gist.github.com/wlcrs/2a1c85da2a53982ca42f9a2b5388360c

/etc/config/network (redacted):

config interface 'loopback'
        option ifname 'lo'
        option proto 'static'
        option ipaddr '127.0.0.1'
        option netmask '255.0.0.0'

config globals 'globals'
        option ula_prefix 'fd04:7421:xxx::/48'

config interface 'lan'
        option type 'bridge'
        option ifname 'eth0.1'
        option proto 'static'
        option ipaddr '192.168.1.1'
        option netmask '255.255.255.0'
        option ip6assign '60'

config interface 'wan'
        option ifname 'eth0.2'
        option proto 'pppoe'
        option username 'xxx@YYY'
        option password 'xxx'
        option ipv6 'auto'
        option keepalive '0'

config switch
        option name 'switch0'
        option reset '1'
        option enable_vlan '1'

config switch_vlan
        option device 'switch0'
        option vlan '1'
        option ports '1 2 3 4 5t'

config switch_vlan
        option device 'switch0'
        option vlan '2'
        option ports '0 5t'

config interface 'henet'
        option proto '6in4'
        option tunlink 'wan'
        option peeraddr '216.66.80.30'
        option ip6addr '2001:470:1f0a:xxx::2/64'
        option ip6prefix '2001:470:1f0b:xxx::/64'
        option tunnelid 'xxx'
        option username 'xxx'
        option password 'xxx'
        option mtu '1280'
        option auto '0'

~ # ifstatus wan
{
        "up": true,
        "pending": false,
        "available": true,
        "autostart": true,
        "dynamic": false,
        "uptime": 38051,
        "l3_device": "pppoe-wan",
        "proto": "pppoe",
        "device": "eth0.2",
        "updated": [
                "addresses",
                "routes"
        ],
        "metric": 0,
        "dns_metric": 0,
        "delegation": true,
        "ipv4-address": [
                {
                        "address": "62.235.202.156",
                        "mask": 32,
                        "ptpaddress": "62.235.202.1"
                }
        ],
        "ipv6-address": [

        ],
        "ipv6-prefix": [

        ],
        "ipv6-prefix-assignment": [

        ],
        "route": [
                {
                        "target": "0.0.0.0",
                        "mask": 0,
                        "nexthop": "62.235.202.1",
                        "source": "0.0.0.0\/0"
                }
        ],
        "dns-server": [
                "194.119.228.67",
                "193.74.208.135"
        ],
        "dns-search": [

        ],
        "inactive": {
                "ipv4-address": [

                ],
                "ipv6-address": [

                ],
                "route": [

                ],
                "dns-server": [

                ],
                "dns-search": [

                ]
        },
        "data": {

        }
}

~ # ifstatus henet
{
        "up": true,
        "pending": false,
        "available": true,
        "autostart": true,
        "dynamic": false,
        "uptime": 13,
        "l3_device": "6in4-henet",
        "proto": "6in4",
        "updated": [
                "addresses",
                "routes",
                "prefixes"
        ],
        "metric": 0,
        "dns_metric": 0,
        "delegation": true,
        "ipv4-address": [

        ],
        "ipv6-address": [
                {
                        "address": "2001:470:1f0a:85f::2",
                        "mask": 64
                }
        ],
        "ipv6-prefix": [
                {
                        "address": "2001:470:1f0b:85f::",
                        "mask": 64,
                        "class": "henet",
                        "assigned": {
                                "lan": {
                                        "address": "2001:470:1f0b:85f::",
                                        "mask": 64
                                }
                        }
                }
        ],
        "ipv6-prefix-assignment": [

        ],
        "route": [
                {
                        "target": "::",
                        "mask": 0,
                        "nexthop": "::",
                        "source": "2001:470:1f0a:85f::2\/64"
                },
                {
                        "target": "::",
                        "mask": 0,
                        "nexthop": "::",
                        "source": "2001:470:1f0b:85f::\/64"
                }
        ],
        "dns-server": [

        ],
        "dns-search": [

        ],
        "inactive": {
                "ipv4-address": [

                ],
                "ipv6-address": [

                ],
                "route": [

                ],
                "dns-server": [

                ],
                "dns-search": [

                ]
        },
        "data": {

        }
}
~ #

I also use HE's tunnel over a PPPoE connection, an I also had MTU issues... This is my current configuration:

  • My WAN interface has a MTU of 1492.
  • My 6in4 interface has a MTU of 1472.

Thank you for your reaction!

I've tried both 1472,1460 and 1452 as MTU, but that didn't solve the issue either. I thought that "lower is safer" in terms of MTU: If it doesn't work with an MTU of 1280, why could it work with an MTU of 1472/1460/1452?

15: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN qlen 3
    link/ppp
23: 6in4-henet@pppoe-wan: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN qlen 1
    link/sit 62.235.202.156 peer 216.66.80.30

I've also verified the MTU of my IPv4 connection:

thijs@ibcn055:~$ ping -4 -M do -s 1464 facebook.com
PING facebook.com (157.240.201.35) 1464(1492) bytes of data.
1472 bytes from edge-star-mini-shv-01-ams4.facebook.com (157.240.201.35): icmp_seq=1 ttl=53 time=16.3 ms
1472 bytes from edge-star-mini-shv-01-ams4.facebook.com (157.240.201.35): icmp_seq=2 ttl=53 time=19.9 ms
1472 bytes from edge-star-mini-shv-01-ams4.facebook.com (157.240.201.35): icmp_seq=3 ttl=53 time=15.6 ms
1472 bytes from edge-star-mini-shv-01-ams4.facebook.com (157.240.201.35): icmp_seq=4 ttl=53 time=17.3 ms
1472 bytes from edge-star-mini-shv-01-ams4.facebook.com (157.240.201.35): icmp_seq=5 ttl=53 time=15.6 ms
1472 bytes from edge-star-mini-shv-01-ams4.facebook.com (157.240.201.35): icmp_seq=6 ttl=53 time=19.1 ms
^C
--- facebook.com ping statistics ---
6 packets transmitted, 6 received, 0% packet loss, time 5005ms
rtt min/avg/max/mdev = 15.617/17.343/19.998/1.703 ms
thijs@ibcn055:~$ ping -4 -M do -s 1465 facebook.com
PING facebook.com (31.13.64.35) 1465(1493) bytes of data.
ping: sendmsg: Message too long
ping: sendmsg: Message too long
ping: sendmsg: Message too long
^C
--- facebook.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3006ms

There is a tab in your HE.net tunnel configuration which tells you the MTU of the tunnel. Make sure you are using the same on your side. I think it was 1480 back then when I was using it, but you need to verify yours.

I always changed the MTU on both sides: in the Advanced tab at HE (clicking on 'Update') to push the new MTU, and explicitely in my OpenWrt config.

When I don't configure the MTU in my OpenWrt install, then it stays at 1280 for the 6in4-henet interface.

Are you making those tests from the router or from a computer connected to the router?

I previously did my tests on my laptop as the ping6 utility on my OpenWrt 18.06.3 was broken. I flashed 19.07.1 again, where netutils-ping6 works.

With an MTU of 1472 - configured both on tunnelbroker.net and on my OpenWrt router -, I get the following results:

~ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP qlen 1000
    link/ether f4:ec:38:b7:df:58 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f6ec:38ff:feb7:df58/64 scope link
       valid_lft forever preferred_lft forever
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
5: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether f4:ec:38:b7:df:58 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global br-lan
       valid_lft forever preferred_lft forever
    inet6 2001:470:1f0b:xxx::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fdda:2082:9df1::1/60 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::f6ec:38ff:feb7:df58/64 scope link
       valid_lft forever preferred_lft forever
6: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether f4:ec:38:b7:df:58 brd ff:ff:ff:ff:ff:ff
11: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-lan state UP qlen 1000
    link/ether f4:ec:38:b7:df:58 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f6ec:38ff:feb7:df58/64 scope link
       valid_lft forever preferred_lft forever
14: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether f4:ec:38:b7:df:58 brd ff:ff:ff:ff:ff:ff
15: pppoe-wan_pppoe: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN qlen 3
    link/ppp
    inet 213.49.159.xxx peer 213.49.159.1/32 scope global pppoe-wan_pppoe
       valid_lft forever preferred_lft forever
22: 6in4-henet@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1472 qdisc noqueue state UNKNOWN qlen 1000
    link/sit 213.49.159.124 peer 216.66.80.30
    inet6 2001:470:1f0a:xxx::2/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::d531:9f7c/64 scope link
       valid_lft forever preferred_lft forever
~ # ping6 -M do -s 1424 tunnelbroker.net
PING tunnelbroker.net(tunnelbroker.net) 1424 data bytes
1432 bytes from tunnelbroker.net: icmp_seq=1 ttl=54 time=159 ms
1432 bytes from tunnelbroker.net: icmp_seq=2 ttl=54 time=159 ms
1432 bytes from tunnelbroker.net: icmp_seq=3 ttl=54 time=159 ms
^C
--- tunnelbroker.net ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 159.106/159.206/159.321/0.469 ms
~ # ping6 -M do -s 1425 tunnelbroker.net
PING tunnelbroker.net(tunnelbroker.net) 1425 data bytes
ping: local error: Message too long, mtu=1472
ping: local error: Message too long, mtu=1472
^C
--- tunnelbroker.net ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1049ms

~ # wget -6 http://tunnelbroker.net -O -
Downloading 'http://tunnelbroker.net'
Connecting to 2001:470:0:63::2:80
Connection error: Connection timed out
~ # traceroute6 tunnelbroker.net
traceroute to tunnelbroker.net (2001:470:0:63::2), 30 hops max, 64 byte packets
 1  tunnel571545.tunnel.tserv6.fra1.ipv6.he.net (2001:470:1f0a:85f::1)  23.296 ms  21.859 ms  21.926 ms
 2  10ge3-18.core1.fra1.he.net (2001:470:0:69::1)  16.597 ms  16.834 ms  16.557 ms
 3  100ge11-1.core1.fra2.he.net (2001:470:0:404::2)  16.926 ms  17.207 ms  17.529 ms
 4  *  e0-53.core1.ams2.he.net (2001:470:0:4b7::2)  31.605 ms  *
 5  100ge8-1.core1.lon3.he.net (2001:470:0:227::1)  26.972 ms  27.263 ms  27.093 ms
 6  100ge14-1.core1.lon2.he.net (2001:470:0:3ea::1)  44.709 ms  27.149 ms  49.253 ms
 7  100ge13-2.core1.nyc4.he.net (2001:470:0:2cf::2)  94.231 ms  94.091 ms  94.725 ms
 8  100ge8-1.core1.sjc2.he.net (2001:470:0:296::2)  158.358 ms  158.394 ms  158.210 ms
 9  10ge4-4.core1.sjc1.he.net (2001:470:0:55::1)  176.461 ms  100ge13-2.core1.sjc1.he.net (2001:470:0:1b1::1)  157.708 ms  162.598 ms
10  e0-50.core4.fmt1.he.net (2001:470:0:439::2)  175.700 ms  172.330 ms  176.326 ms
11  *  *  *
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *^C

When I do a tcpdump on my pppoe interface, I see a lot of TCP Retransmissions happening:

I'm a bit out of ideas here... Did you check that you have the correct IPv6 address (and my "ip6addr" parameter does not have a "/64" at the end, by the way)? Did you try with another endpoint?

Are you firewall the icmpv6 messages accidentally? Ipv6 should do path MTU discovery via icmp

Yes, I still have the default rules to allow ICMPv6 traffic. To rule out firewall problems, I've even disabled the firewall, checked if ip6tables was allowing everything, but alas: that is not the problem.

Thank you for all your input. Yes I've double-checked every configuration parameter. As there is some traffic that gets through, I'm 99,95% sure that everything is correct. I've also tried creating a tunnel on another HE endpoint, and I've even tried using tunnelbroker.ch instead, but that also doesn't solve the problem. I'm beginning to suspect that my ISP modem is messing with the traffic and causing issues :frowning_face:

The traceroute is fine to stop on step 10.
To rule out OpenWrt do you have a secondary router to try? Otherwise take a backup, restore to defaults and set up only the internet connection and the HE tunnel.