WAN IPv4 outbound not working correctly but 6in4 works just fine?

freddie · December 3, 2020, 3:46pm

I have been using OpenWrt for over two years (since 17.04) on my LinkSys WRT1200AC router, and am currently running 19.07.4 r11208-ce6496d796.

My ISP is IPv4 only and I like using IPv6 so have installed the 6in4 package and use a Hurricane Electric tunnel for IPv6 traffic. I have been using this setup for over two years without any issues.

Since Tuesday my WAN IPv4 access has been essentially broken - ping requests timing out, no access to IPv4-only web sites. This appeared to start gradually from about 1:20pm on Tuesday 1st December. At the same time, IPv4 on my LAN continued to work as normal, and IPv6 both on LAN on WAN (via the HE tunnel) work as normal.

WAN IPv4 obviously isn't completely broken as the HE tunnel is carrying traffic (such as this message). Also, I run an email server and a web server on my LAN and they are serving requests just fine. This issue appears to be affecting outbound IPv4 traffic only (not that that explains the HE tunnel connecting okay).

I upgraded OpenWrt in November (on the 14th) from 17.04 and things worked after the upgrade as normal. I haven't changed any settings in OpenWrt since 20th November, so am a bit flummoxed as to what is going on.

I have copied network config below. My ISP requires vlan tagging (vid 913):

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config globals 'globals'

config interface 'lan'
	option type 'bridge'
	option ifname 'eth0.1'
	option proto 'static'
	option netmask '255.255.255.0'
	option ip6assign '64'
	option ipaddr '10.0.8.1'

config interface 'wan'
	option proto 'static'
	option netmask '255.255.255.248'
	option ipaddr '185.161.134.132'
	option gateway '185.161.134.129'
	option ifname 'eth1.913'
        option dns '1.1.1.1'

config interface 'wan6'
	option proto '6in4'
	option peeraddr '216.66.80.26'
	list ip6prefix '2001:470:1f09:45e::/64'
	option ip6addr '2001:470:1f08:45e::2/64'

config switch
	option name 'switch0'
	option reset '1'
	option enable_vlan '1'

config switch_vlan
	option device 'switch0'
	option vlan '1'
	option ports '5t 3 2 1 0'
	option vid '1'

config switch_vlan
	option device 'switch0'
	option vlan '2'
	option ports '6t 4t'
	option vid '913'

I would be grateful for ideas regarding what has gone adrift.

Many thanks.

krazeh · December 3, 2020, 5:15pm

Have you spoken to your ISP? If not, I would speak to them first as it sounds more likely it's an issue on their side. I'd also give your router a reboot (if you've not already done so).

freddie · December 3, 2020, 5:36pm

Yes, I have done both. ISP say nothing has changed on their side and they can ping the public IP on my router. I have rebooted my router several times. I have also plugged the WAN ethernet cable into my laptop and can ping IPv4 hosts from it.

slh · December 3, 2020, 8:41pm

If your configuration has been working fine for a considerable time before, it's more likely that the he_net tunnel endpoint server is having trouble - I've seen that in the past, it can easily take a couple of days up to a week for this to get fixed.

freddie · December 3, 2020, 9:09pm

The HE tunnel is the only bit that is working as it should - so I don't understand how the tunnel endpoint server having trouble could be pertinent. I'm happy to be educated on this point, though, as I consider myself a relative newbie to OpenWrt.

vgaetera · December 4, 2020, 5:43am

You should isolate the issue, so try to back up and reset.
Then test with the default configs and just the minimum required modifications.

freddie · December 4, 2020, 5:04pm

I have re-flashed my router and used the default config. The only change I have made is to add in my WAN config details (as in the config I posted earlier) but left off the HE tunnel for the moment. I have to include the VLAN tagging as that is what my ISP requires. The connection doesn't work at all now. Even when I re-apply my original configuration (which partly worked until this morning when I reset my router) the connection doesn't work.

I think my router is functional, though, as I can connect to the ISP if I set the WAN to use DHCP - but at 20% normal speed. But I have a static IP address for a reason, so can't carry on for long with this configuration.

freddie · December 5, 2020, 10:40am

This is an incorrect statement - I didn't check carefully enough at the time. The connection is, in fact, performing exactly as before:

HE 6in4 tunnel works - so anything that uses IPv6 on my network has a working connection.
IPv4 through the WAN interface isn't working - apart from the HE tunnel.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.4, r11208-ce6496d796
 -----------------------------------------------------
root@gw:~# ping -c 3 -4 google.com
PING google.com (216.58.213.110): 56 data bytes

--- google.com ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
root@gw:~# ping -c 3 -6 google.com
PING google.com (2a00:1450:4009:819::200e): 56 data bytes
64 bytes from 2a00:1450:4009:819::200e: seq=0 ttl=121 time=49.829 ms
64 bytes from 2a00:1450:4009:819::200e: seq=1 ttl=121 time=13.231 ms
64 bytes from 2a00:1450:4009:819::200e: seq=2 ttl=121 time=18.073 ms

--- google.com ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 13.231/27.044/49.829 ms
root@gw:~#

So .... back to where I was before I reset the router.

vgaetera · December 5, 2020, 10:48am

Check the output:

mtr -4wbc10 example.org

freddie · December 5, 2020, 10:54am

Thanks for the reply. I see this:

root@gw:~# mtr -4wbc10 openwrt.org
Start: 2020-12-05T10:52:46+0000
HOST: gw                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 185.161.134.129 (185.161.134.129)  0.0%    10   10.1   6.2   2.5  10.2   2.5
  2.|-- ???                               100.0    10    0.0   0.0   0.0   0.0   0.0

vgaetera · December 5, 2020, 10:57am

Try to decrease MTU on the WAN interface, i.e. 1450, 1400, 1350, 1300.

Also check tracepath if possible.

freddie · December 5, 2020, 11:03am

Okay, thank you. No change with MTU 1450. I'll try a few more values and report back.

freddie · December 5, 2020, 11:35am

I tried using MTU values down to 1250. No real change - just some variation in the numbers reported by mtr.
This is MTU 1400:

root@gw:~# mtr -4wbc10 openwrt.org
Start: 2020-12-05T11:26:34+0000
HOST: gw                                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 185.161.134.129 (185.161.134.129)  0.0%    10   17.1  10.1   4.3  17.1   4.3
  2.|-- ???                               100.0    10    0.0   0.0   0.0   0.0   0.0

The tracepath showed a similar lack of variability - each time just reporting values for the first hop. Here's the output for MTU set to 1250:

root@gw:~# tracepath openwrt.org
 1?: [LOCALHOST]                                         pmtu 1250
 1:  185.161.134.129                                       4.664ms
 1:  185.161.134.129                                       4.235ms
 2:  no reply
 3:  no reply
 4:  no reply
 5:  no reply
 6:  no reply
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
13:  no reply
14:  no reply
15:  no reply
16:  no reply
17:  no reply
18:  no reply
19:  no reply
20:  no reply
21:  no reply
22:  no reply
23:  no reply
24:  no reply
25:  no reply
26:  no reply
27:  no reply
28:  no reply
29:  no reply
30:  no reply
31:  no reply
     Too many hops: pmtu 1250
     Resume: pmtu 1250

freddie · December 5, 2020, 11:44am

I should also add that I can ping my WAN IP address from my web server out in the wild.

freddie · December 6, 2020, 12:26pm

Setting my WAN connection to obtain it's IP address from DHCP creates a working connection, as I mentioned before, but speed is much reduced (I pay for 30 down, 5 up; the DHCP connection gives 3 down and 3 up).

The results of MTR and tracepath over the connection with the DHCP-obtained IPv4 address:

root@gw:~# mtr -4wbc10 openwrt.org
Start: 2020-12-06T12:20:47+0000
HOST: gw                                                             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 192.168.50.1 (192.168.50.1)                                     0.0%    10    7.5  14.3   6.9  51.0  13.3
  2.|-- 10.100.106.1 (10.100.106.1)                                     0.0%    10   22.1  30.2  16.7  67.5  18.5
  3.|-- vlan1.secure-web.router.enta.net (78.33.14.169)                 0.0%    10   45.0  31.6  14.7  61.8  15.8
  4.|-- 10.100.47.153 (10.100.47.153)                                   0.0%    10   59.6  42.7  26.4  69.4  13.5
  5.|-- bundle-ether102.telehouse-east4.core.enta.net (188.39.127.166)  0.0%    10   56.3  57.1  27.2 126.7  32.0
  6.|-- te5-3.amsterdam.core.enta.net (188.39.127.185)                  0.0%    10   33.1  39.0  29.7  53.6   9.1
  7.|-- 80.249.211.163 (80.249.211.163)                                 0.0%    10   59.7  58.3  36.0 122.0  26.7
  8.|-- 138.197.244.86 (138.197.244.86)                                 0.0%    10   52.9  67.4  47.1 139.3  28.8
  9.|-- ???                                                            100.0    10    0.0   0.0   0.0   0.0   0.0
 10.|-- ???                                                            100.0    10    0.0   0.0   0.0   0.0   0.0
 11.|-- ???                                                            100.0    10    0.0   0.0   0.0   0.0   0.0
 12.|-- wiki-01.infra.openwrt.org (139.59.209.225)                      0.0%    10   70.4  58.0  41.8 106.8  21.7
root@gw:~# tracepath openwrt.org
 1?: [LOCALHOST]                                         pmtu 1500
 1:  192.168.50.1                                         13.305ms
 1:  192.168.50.1                                         21.678ms
 2:  10.100.106.1                                         82.379ms
 3:  vlan1.secure-web.router.enta.net                     37.077ms
 4:  10.100.47.153                                       111.415ms
 5:  bundle-ether102.telehouse-east4.core.enta.net       132.960ms
 6:  te5-3.amsterdam.core.enta.net                        70.159ms
 7:  80.249.211.163                                       83.762ms
 8:  138.197.244.86                                       72.781ms asymm  7
 9:  138.197.250.138                                      88.629ms asymm  8
10:  no reply
11:  no reply
12:  wiki-01.infra.openwrt.org                           104.578ms reached
     Resume: pmtu 1500 hops 12 back 54

vgaetera · December 6, 2020, 2:54pm

Consider creating an IPv4 alias.
This way you can use dynamically and statically configured interfaces simultaneously.
Specific routing/failover policies can be configured with VPN Policy Routing and/or mwan3.
It should be possible to route 6in4 tunnel via the statically configured interface and route general IPv4 traffic via the other one.

eduperez · December 7, 2020, 6:52am

Have you tried to run your tests with the HE tunnel disabled?

What kind of internet connection do you have? I find it odd that you can configure a static IP public address, or a DHCP client on the same line, and do not obtain the same IP address (and the DHCP client obtains a private IP address inside another private network).

HE uses the 6in4 protocol, different to TCP or UDP, perhaps there is some firewall blocking these connections?

freddie · December 7, 2020, 8:14am

Thanks for the reply @eduperez

I live in a very rural area, and my internet connection is a radio link to a mast some 5km away. It is known as fixed wireless access (FWA). My connection is made to the radio at my end of the link. Most subscribers have their radio act as a router that supplies an IP address via DHCP - but I need a static routable IP address as I provide services (HTTP/S, IMAP/S) from my LAN; therefore my radio is in bridge mode. This is what the ISP has described to me. I'm assuming that my VLAN is what keeps my inbound traffic identifiable across my ISP's network.

I had a similar issue to this a couple of weeks ago which was related to VLAN tagging. Previous to that, I have had 6 months trouble-free connections.

My ISP wasn't surprised that I was able to connect using DHCP but at much-reduced throughput - so I haven't questioned that aspect as I don't normally use it (but have been very grateful for it over the weekend).

I did run tests with the HE tunnel disabled, and there was no change.

As I stated in an earlier reply - with my connection in the default state (i.e. static IP address) it is the HE tunnel that is the functioning part of the connection, along with inbound IPv4 requests to my LAN services. So I don't believe 6in4 is being blocked by a firewall in this scenario.

freddie · December 14, 2020, 5:48pm

Revisiting this. After reading up on some concepts and with help from @vgaetera and @eduperez (and apologies to @krazeh as he did suggest this initially) I managed to convince myself that my OpenWrt configuration was just fine, so it had to be a problem on the ISP side. After much ducking and diving, they finally admitted that occasionally their firewall appliances sometimes automatically blocked certain traffic patterns, and I had become a victim of this. They did not know how to unblock my IP address so gave me a new one, and everything started flowing straight away.

At least I have learned what to do if this happens again - but I sincerely hope it doesn't!

Thanks to all who contributed.