Firewall mysteriously blocking responses to outgoing connections

ioangogo · January 27, 2022, 3:45pm

Hi, I have been having some issues with my network recently, the main symptom is mainly new IPv4 connections timing out, this takes a while for me to notice as most of my web browsing is to websites that use IPv6. This also seems to only happen to Windows clients as it does not affect any of the phones or Linux computers on the network, just windows and intermittently happens, the inital pattern looked like it was every 24hrs, but it doesnt seem to fully follow that cycle.

The other day when it happened i connected to a known site that i ran and ran wireshark on the WAN side of my router and saw ICMP Destination Unreachable messages being returned by the router, Although i now dont have the slightest clue of what to google or where to look to further debug this issue.

I don't have the packet capturers to hand and they missed the connection setup so are not very useful, but if i catch it happening again i will capture them again and update this thread

lleachii · January 27, 2022, 4:08pm

What was the unreachable destination?
This usually means your ISP is telling you the IP is unreachable...which is rare

See: http://www.networksorcery.com/enp/protocol/icmp/msg3.htm

Have you called your ISP?

ioangogo · January 27, 2022, 4:12pm

Apologies, I don't think i explained it well, the ICMP messages where being made by my router and where sent in response to the replies from the website, so the outgoing connection from my machine had gotten to the site, but my router was blocking the return.

the website is ipv4.ioangogo.dev (This will raise a HSTS error because i haven't gotten around to fixing the server i use for testing).

lleachii · January 27, 2022, 4:26pm

That doesn't make scene either...but to clarify:

You're saying that you monitor the LAN port of your router with tcpdump - and you browse to a website... (e.g. https://1.1.1.1/ - I use this example because it is an IP and HTTPS site), you receive:

ICMP Destination unreachable from 192.168.1.1...and this message is regarding a packet from SRC 192.168.1.xxx to 1.1.1.1 443/tcp - correct?

???

This means the destination was reachable...now I'm confused...
I can reach it, albeit the cert error you mentioned

ioangogo · January 27, 2022, 4:34pm

So in this case im running tcp-dump on pppoe-wan so the ICMP Destination unreachable is SRC [my WAN address] to 1.1.1.1 443/tcp.

on the LAN side i only see the device connecting out and nothing returning

That was just a side note about the site. I should add here as it is related, if I disable and enable the network interface in windows the issue goes away.

lleachii · January 27, 2022, 4:51pm

Your WAN address is unreachable!?!?

In Windows...wth......

I think your Windows firewall may be closing the TCP connection...

What happens if you disable the Windows firewall on the client?

ioangogo · January 27, 2022, 4:57pm

Someone else on the network who had the same problem tired that and it still happens, however i'll also double check to confirm that

ioangogo · January 29, 2022, 8:36pm

Nothing, same problem still happens

Running wireshark on the laptop, and on br-lan &pppoe-wan doesn't show anything different. The windows client sends a syn and doesn't get the ack due to it being blocked by the router

I can't see any TCP packets with rst or fin flags coming from either host

ioangogo · January 31, 2022, 12:28am

Further information, it only seems to happen when the chosen ephemeral port is over 61000(not the exact number)

ioangogo · January 31, 2022, 10:51am

I now have a way to reproduce it

curl --local-port 61500 https://ipv4.ioangogo.dev --verbose

I have also since tried extending the net.ipv4.ip_local_port_range IANA reserved range and that didnt help(although from what i read that only affects connections from the device)

ioangogo · January 31, 2022, 1:01pm

So i ran my thing above and conntrack -L | grep 61500 and conntrack does see the packet and has an entry, but the retuned ACK is still blocked. I wish i could change the title but i cant

As this is seems to be a firewall issue, here is the contents of /etc/config/firewall. I cant see any rules that could be causing this, but anyway

config defaults
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option synflood_protect '1'
        option flow_offloading '1'
        option flow_offloading_hw '1'

config zone
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        list network 'lan'
        list network '6lan'
        option log '1'

config zone
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        list network 'wan'
        list network 'wan6'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Input'
        option src 'wan'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        list icmp_type 'router-solicitation'
        list icmp_type 'neighbour-solicitation'
        list icmp_type 'router-advertisement'
        list icmp_type 'neighbour-advertisement'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Support-UDP-Traceroute'
        option src 'wan'
        option proto 'udp'
        option dest_port '33434-33689'
        option target 'ACCEPT'
        option family 'ipv6'

config include
        option path '/etc/firewall.user'
        option reload '1'

config rule
        option src 'wan'
        option dest 'lan'
        list dest_ip '::11/-64'
        option target 'ACCEPT'
        option family 'ipv6'
        option name 'webserver'
        option dest_port '80 443'

config redirect
        option target 'DNAT'
        option name 'web'
        option src 'wan'
        option src_dport '80'
        option dest 'lan'
        option dest_ip '192.168.1.11'
        option dest_port '80'

config redirect
        option target 'DNAT'
        option name 'webs'
        option src 'wan'
        option src_dport '443'
        option dest 'lan'
        option dest_ip '192.168.1.11'
        option dest_port '443'

config rule
        option name 'MQTT'
        option family 'ipv6'
        option src 'wan'
        option dest 'lan'
        list dest_ip '::11/-64'
        option dest_port '8883'
        option target 'ACCEPT'

config redirect
        option target 'DNAT'
        option name 'mqtt'
        option src 'wan'
        option dest 'lan'
        option dest_ip '192.168.1.11'
        option dest_port '8883'
        option src_dport '8883'

config rule
        option name 'ssh'
        option family 'ipv6'
        option src 'wan'
        option dest 'lan'
        list dest_ip '::11/-64'
        option dest_port '22'
        option target 'ACCEPT'

config redirect
        option target 'DNAT'
        option name 'ssh'
        option src 'wan'
        option src_dport '22'
        option dest 'lan'
        option dest_ip '192.168.1.11'

config include 'miniupnpd'
        option type 'script'
        option path '/usr/share/miniupnpd/firewall.include'
        option family 'any'
        option reload '1'

config rule
        option name 'wireguard'
        option src 'wan'
        option dest 'lan'
        list dest_ip '::11/-64'
        option dest_port '51820'
        option target 'ACCEPT'
        option family 'ipv6'

config redirect
        option target 'DNAT'
        option name 'wg'
        option src 'wan'
        option src_dport '51820'
        option dest 'lan'
        option dest_ip '192.168.1.11'
        option dest_port '51820'

config rule
        option name 'UDP-Traceroute-Forward'
        option family 'ipv6'
        list proto 'udp'
        option src 'wan'
        option dest '*'
        option dest_port '33434-33689'
        option target 'ACCEPT'

config rule
        option name 'Syncthing'
        option family 'ipv6'
        option src 'wan'
        option dest 'lan'
        option dest_port '22000'
        option target 'ACCEPT'

config redirect
        option target 'DNAT'
        option name 'sip'
        list proto 'udp'
        option src 'wan'
        option src_dport '5060'
        option dest 'lan'
        option dest_ip '192.168.1.15'
        option dest_port '5060'
        option src_ip '217.10.64.0/20'

config redirect
        option target 'DNAT'
        option name 'sip'
        list proto 'udp'
        option src 'wan'
        option src_ip '217.116.112.0/20'
        option src_dport '5060'
        option dest 'lan'
        option dest_ip '192.168.1.15'
        option dest_port '5060'

config redirect
        option target 'DNAT'
        option name 'sip'
        list proto 'udp'
        option src 'wan'
        option src_ip '212.9.32.0/19'
        option src_dport '5060'
        option dest 'lan'
        option dest_ip '192.168.1.15'

dlakelan · January 31, 2022, 1:20pm

Changed. Let me know if you have a better suggestion for the title.

lleachii · January 31, 2022, 6:18pm

What are the results of ip route get <IP_of_DST>?

...maybe I missed something...but the OP claims to receive ICMP Destination Unreachable ...I'm not sure it's firewall-related yet.

dlakelan · January 31, 2022, 6:30pm

No, his router is sending those packets back to the remote host.

dlakelan · January 31, 2022, 6:31pm

also see above

lleachii · January 31, 2022, 6:34pm

I'm not sure why you reiterated this point, perhaps it was the semantics of my sentence. I understand that, and which is why I asked for an ip route get.

To clarify, I meant "...receive on tcpdump display".

But also, which is odd - it seems:

the client send traffic to <foo_IP> (assuming via default route)
the reply traffic hits WAN
WANs send an ICMP Destination Unreachable???

(this is what I wanted to clarify - that's not clear)

Who did WAN send the Unreachable to (remember it's not an IP that's Unreachable)?

Also recall - ICMP Unreachable messages included a part of the packet it references.

dlakelan · January 31, 2022, 6:35pm

mainly because it was actually not that obvious as I read back through the thread, so I wanted to make sure I was understanding it correctly

ioangogo · January 31, 2022, 7:43pm

I think i got confused when trying to clarify that eailer, so ill just share a screenshot from wireshark

the blurred sections are my IP address

Its my router sending the message back to ipv4.ioangogo.dev. the issue is not limited to that site, it affects all IPv4 connections that match the condition of having a ephemeral port number of over 61000, i am just using that as a known endpoint. This issue does not affect IPv6 connections to the same site with a ephemeral port number over 61000

lleachii · February 1, 2022, 3:04am

It's also interesting that some Windows devices didn't use ports above 61000...

Have you changed any other sysctrl or basic network settings or LAN or WAN?
Does this client route thru your Wireguard in any way?

ioangogo · February 1, 2022, 10:04am

Having a quick look over the files i don't think i have. One major difference is that I have jool setup. However that does not get affected by this.

I also have some static routes setup for the wireguard VPN(Its a inbound VPN for accessing the network), but that only affects 192.168.15.0/24 and [ISP /48 prefix]:15::/64.

no

This has happened with every windows device on the network, the reason i think it didn't look like it affected Linux is because most linux operating systems typically have net.ipv4.ip_local_port_range set to 32768–60999. Windows uses the full IANA reserved range 49152–65535