Coova-chilli and NAT leak

aport · February 29, 2020, 9:33pm

Running latest coova-chilli on OpenWrt 19.07.1, basically a standard setup going to hotspotsystem.com

The following tcpdump is running on my WAN interface during a client login. Almost every packet is properly put through NAT but occasionally some packets slip through with the source IP set to the private radius address.

In my case, my WAN provider (Verizon) sees the unroutable source IP and disconnects my LTE bearer which brings my internet down.

I've tried every combination of firewall rules I can think of but I must be doing something wrong because these packets still get through.

20:48:39.173708 IP (tos 0x0, ttl 53, id 59401, offset 0, flags [DF], proto TCP (6), length 1360)
195.228.254.149.443 > ZZZ.YYY.202.198.60371: Flags [P.], cksum 0x53da (correct), seq 2890:4198, ack 2890, win 278, options [nop,nop,TS val 2524332824 ecr 27519633], length 1308

20:48:39.316310 IP (tos 0x0, ttl 63, id 31724, offset 0, flags [DF], proto TCP (6), length 52)
ZZZ.YYY.202.198.60371 > 195.228.254.149.443: Flags [.], cksum 0xa7da (correct), seq 2890, ack 2890, win 369, options [nop,nop,TS val 27519696 ecr 2524332823], length 0

20:48:39.316450 IP (tos 0x0, ttl 63, id 55782, offset 0, flags [DF], proto TCP (6), length 52)
192.168.182.2 .60352 > 195.228.254.149.443: Flags [F.], cksum 0xd522 (correct), seq 0, ack 1, win 583, options [nop,nop,TS val 27519696 ecr 2524312590], length 0

20:48:39.355874 IP (tos 0x0, ttl 63, id 31725, offset 0, flags [DF], proto TCP (6), length 52)
ZZZ.YYY.202.198.60371 > 195.228.254.149.443: Flags [.], cksum 0xa2ae (correct), seq 2890, ack 4198, win 380, options [nop,nop,TS val 27519700 ecr 2524332824], length 0

20:48:39.679381 IP (tos 0x0, ttl 63, id 10255, offset 0, flags [DF], proto TCP (6), length 52)
192.168.182.2 .60357 > 195.228.254.149.443: Flags [F.], cksum 0x5616 (correct), seq 0, ack 1, win 347, options [nop,nop,TS val 27519731 ecr 2524315427], length 0

20:48:40.521621 IP (tos 0x0, ttl 63, id 55783, offset 0, flags [DF], proto TCP (6), length 52)
192.168.182.2 .60352 > 195.228.254.149.443: Flags [F.], cksum 0xd4aa (correct), seq 0, ack 1, win 583, options [nop,nop,TS val 27519816 ecr 2524312590], length 0

Has anyone else seen this issue?

trendy · February 29, 2020, 9:51pm

What is the network and firewall configuration?
uci export network; uci export firewall

aport · February 29, 2020, 9:57pm

Firewall and network are default out-of-the-box settings

trendy · February 29, 2020, 9:58pm

Okay, let's see them.
Also which interface was used for the tcpdump? WAN or any?

aport · February 29, 2020, 10:03pm

My WAN interface

The command used was tcpdump -n -vv -i wwan0

Since the WAN zone is masqueraded I'm not sure how packets can go out without the source IP being re-written... but there they are

trendy · February 29, 2020, 10:04pm

Ok if you don't want to provide the configurations, I'll let you here keep guessing why it doesn't work.

aport · February 29, 2020, 10:11pm

The firewall config is literally default.

The network config is literally default except for my WAN:

config interface wan
        option proto wwan

aport · February 29, 2020, 10:14pm

package firewall

config defaults
	option syn_flood '1'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'lan'
	option network 'lan'
	option input 'ACCEPT'
	option output 'ACCEPT'
	option forward 'REJECT'

config zone
	option name 'wan'
	option network 'wan'
	option input 'REJECT'
	option output 'ACCEPT'
	option forward 'REJECT'
	option masq '1'
	option mtu_fix '1'

config forwarding
	option src 'lan'
	option dest 'wan'

config rule
	option name 'Allow-DHCP-Renew'
	option src 'wan'
	option proto 'udp'
	option dest_port '68'
	option target 'ACCEPT'
	option family 'ipv4'

config rule
	option name 'Allow-Ping'
	option src 'wan'
	option proto 'icmp'
	option icmp_type 'echo-request'
	option family 'ipv4'
	option target 'ACCEPT'

config rule
	option name 'Allow-DHCPv6'
	option src 'wan'
	option proto 'udp'
	option src_ip 'fe80::/10'
	option src_port '547'
	option dest_ip 'fe80::/10'
	option dest_port '546'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Input'
	option src 'wan'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	list icmp_type 'router-solicitation'
	list icmp_type 'neighbour-solicitation'
	list icmp_type 'router-advertisement'
	list icmp_type 'neighbour-advertisement'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config rule
	option name 'Allow-ICMPv6-Forward'
	option src 'wan'
	option dest '*'
	option proto 'icmp'
	list icmp_type 'echo-request'
	list icmp_type 'echo-reply'
	list icmp_type 'destination-unreachable'
	list icmp_type 'packet-too-big'
	list icmp_type 'time-exceeded'
	list icmp_type 'bad-header'
	list icmp_type 'unknown-header-type'
	option limit '1000/sec'
	option family 'ipv6'
	option target 'ACCEPT'

config include
	option path '/etc/firewall.user'

package network

config interface 'loopback'
	option ifname 'lo'
	option proto 'static'
	option ipaddr '127.0.0.1'
	option netmask '255.0.0.0'

config interface 'lan'
	option type 'bridge'
	option proto 'static'
	option ipaddr '192.168.1.1'
	option netmask '255.255.255.0'
	option _orig_ifname 'eth0.1 wlan0 wlan1'
	option _orig_bridge 'true'
	option ifname 'eth0.1'

config interface 'wan'
	option proto 'wwan'

config switch
	option name 'rtl8366s'
	option reset '1'
	option enable_vlan '1'
	option blinkrate '2'

config switch_vlan
	option device 'rtl8366s'
	option vlan '1'
	option ports '0 1 2 3 5t'

config switch_port
	option device 'rtl8366s'
	option port '1'
	option led '6'

config switch_port
	option device 'rtl8366s'
	option port '2'
	option led '9'

config switch_port
	option device 'rtl8366s'
	option port '5'
	option led '2'

trendy · February 29, 2020, 10:17pm

Alright, and where is the 192.168.182.2 located? On the router? In the lan?

What is the output of iptables-save -c

aport · February 29, 2020, 10:27pm

It's a wireless client connected to coova-chilli. 192.168.182.0/24 is the network of the tunnel interface created and managed by chilli.

*nat
:PREROUTING ACCEPT [1551065:143479788]
:INPUT ACCEPT [329663:27601331]
:OUTPUT ACCEPT [483715:32717658]
:POSTROUTING ACCEPT [1274:337276]
:postrouting_lan_rule - [0:0]
:postrouting_rule - [0:0]
:postrouting_wan_rule - [0:0]
:prerouting_lan_rule - [0:0]
:prerouting_rule - [0:0]
:prerouting_wan_rule - [0:0]
:zone_lan_postrouting - [0:0]
:zone_lan_prerouting - [0:0]
:zone_wan_postrouting - [0:0]
:zone_wan_prerouting - [0:0]
[1551065:143479788] -A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
[1021004:105569161] -A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
[163738:8623554] -A PREROUTING -i wwan0 -m comment --comment "!fw3" -j zone_wan_prerouting
[1498994:108239764] -A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
[882:282902] -A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
[1497720:107902488] -A POSTROUTING -o wwan0 -m comment --comment "!fw3" -j zone_wan_postrouting
[882:282902] -A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
[1021004:105569161] -A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
[1497720:107902488] -A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
[1497720:107902488] -A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
[163738:8623554] -A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
COMMIT
# Completed on Sat Feb 29 14:21:47 2020
# Generated by iptables-save v1.6.2 on Sat Feb 29 14:21:47 2020
*mangle
:PREROUTING ACCEPT [368313216:268871553003]
:INPUT ACCEPT [1129589:113244802]
:FORWARD ACCEPT [366987698:268720107242]
:OUTPUT ACCEPT [2053001:201630288]
:POSTROUTING ACCEPT [368997148:268919349161]
[391491:24194583] -A FORWARD -o wwan0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Sat Feb 29 14:21:47 2020
# Generated by iptables-save v1.6.2 on Sat Feb 29 14:21:47 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:forwarding_lan_rule - [0:0]
:forwarding_rule - [0:0]
:forwarding_wan_rule - [0:0]
:input_lan_rule - [0:0]
:input_rule - [0:0]
:input_wan_rule - [0:0]
:output_lan_rule - [0:0]
:output_rule - [0:0]
:output_wan_rule - [0:0]
:reject - [0:0]
:syn_flood - [0:0]
:zone_lan_dest_ACCEPT - [0:0]
:zone_lan_dest_REJECT - [0:0]
:zone_lan_forward - [0:0]
:zone_lan_input - [0:0]
:zone_lan_output - [0:0]
:zone_lan_src_ACCEPT - [0:0]
:zone_wan_dest_ACCEPT - [0:0]
:zone_wan_dest_REJECT - [0:0]
:zone_wan_forward - [0:0]
:zone_wan_input - [0:0]
:zone_wan_output - [0:0]
:zone_wan_src_REJECT - [0:0]
[8:5832] -A INPUT -i br-lan -j DROP
[3:4500] -A INPUT -d 192.168.182.1/32 -i tun0 -p icmp -j ACCEPT
[0:0] -A INPUT -d 192.168.182.1/32 -i tun0 -p udp -m udp --dport 53 -j ACCEPT
[0:0] -A INPUT -d 192.168.182.1/32 -i tun0 -p udp -m udp --dport 67:68 -j ACCEPT
[0:0] -A INPUT -d 255.255.255.255/32 -i tun0 -p udp -m udp --dport 67:68 -j ACCEPT
[0:0] -A INPUT -d 192.168.182.1/32 -i tun0 -p tcp -m tcp --dport 4990 -j ACCEPT
[58:5851] -A INPUT -d 192.168.182.1/32 -i tun0 -p tcp -m tcp --dport 3990 -j ACCEPT
[0:0] -A INPUT -d 192.168.182.1/32 -i tun0 -j DROP
[614:125107] -A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT
[1128977:113119799] -A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule
[523695:71810272] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[136691:5594656] -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood
[197230:18929073] -A INPUT -i br-lan -m comment --comment "!fw3" -j zone_lan_input
[272786:13181669] -A INPUT -i wwan0 -m comment --comment "!fw3" -j zone_wan_input
[94:13876] -A FORWARD -i tun0 -o wwan0 -j ACCEPT
[0:0] -A FORWARD -i tun0 ! -o wwan0 -j DROP
[102:78200] -A FORWARD -o tun0 -j ACCEPT
[0:0] -A FORWARD -i tun0 -j ACCEPT
[366987707:268720108014] -A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule
[366038961:268634612451] -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[714540:65905276] -A FORWARD -i br-lan -m comment --comment "!fw3" -j zone_lan_forward
[0:0] -A FORWARD -i wwan0 -m comment --comment "!fw3" -j zone_wan_forward
[285:18452] -A FORWARD -m comment --comment "!fw3" -j reject
[614:125107] -A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT
[2052389:201505893] -A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule
[1559335:168091927] -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT
[1541:499501] -A OUTPUT -o br-lan -m comment --comment "!fw3" -j zone_lan_output
[491163:32821393] -A OUTPUT -o wwan0 -m comment --comment "!fw3" -j zone_wan_output
[249196:10504306] -A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset
[10398:1793033] -A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp-port-unreachable
[136691:5594656] -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN
[0:0] -A syn_flood -m comment --comment "!fw3" -j DROP
[1541:499501] -A zone_lan_dest_ACCEPT -o br-lan -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_lan_dest_REJECT -o br-lan -m comment --comment "!fw3" -j reject
[714540:65905276] -A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule
[714540:65905276] -A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT
[0:0] -A zone_lan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_REJECT
[197230:18929073] -A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule
[0:0] -A zone_lan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[197230:18929073] -A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT
[1541:499501] -A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule
[1541:499501] -A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT
[197230:18929073] -A zone_lan_src_ACCEPT -i br-lan -m conntrack --ctstate NEW,UNTRACKED -m comment --comment "!fw3" -j ACCEPT
[43278:2373853] -A zone_wan_dest_ACCEPT -o wwan0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
[1396346:115924651] -A zone_wan_dest_ACCEPT -o wwan0 -m comment --comment "!fw3" -j ACCEPT
[0:0] -A zone_wan_dest_REJECT -o wwan0 -m comment --comment "!fw3" -j reject
[0:0] -A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule
[0:0] -A zone_wan_forward -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port forwards" -j ACCEPT
[0:0] -A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT
[272786:13181669] -A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule
[308:103084] -A zone_wan_input -p udp -m udp --dport 68 -m comment --comment "!fw3: Allow-DHCP-Renew" -j ACCEPT
[13169:799698] -A zone_wan_input -p icmp -m icmp --icmp-type 8 -m comment --comment "!fw3: Allow-Ping" -j ACCEPT
[0:0] -A zone_wan_input -m conntrack --ctstate DNAT -m comment --comment "!fw3: Accept port redirections" -j ACCEPT
[259309:12278887] -A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT
[491163:32821393] -A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule
[491163:32821393] -A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT
[259309:12278887] -A zone_wan_src_REJECT -i wwan0 -m comment --comment "!fw3" -j reject
COMMIT
# Completed on Sat Feb 29 14:21:47 2020

trendy · February 29, 2020, 11:56pm

Config looks good to me. The only suspicious thing I notice is this:
[43278:2373853] -A zone_wan_dest_ACCEPT -o wwan0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
This is either lan->wan traffic or device output.

Which router is it?
Does it handle the traffic or the cpu maxes out?
Is power supply adequate?

Sometimes erratic behaviour is fixed with a reset and reconfiguration, so keep a backup handy just in case. Also you might want to give a try t0 19.07.2 which will be released anytime soon.

aport · March 1, 2020, 3:13am

It's a WNDR3700, so a bit old but still going strong. It handles the traffic just fine. In this particular scenario the bandwidth required is very low as it's just RADIUS auth packets.

I even explicitly added an SNAT rule for traffic from 192.168.182.0/24 to set the source address to the WAN IP. The rule is matching and still some packets are going out the WAN with the source address at 192.168.182.2

I'm so confused

reinerotto · March 1, 2020, 7:49am

I can think of chilli interfering. I have installed/used coova-chilli quite often on openwrt, in various, different hotspot systems, but never with the openwrt-firewall, but with plain, simple, org iptables rules.
KISS
The org (non-openwrt coova-chilli, from standard LINUX) modifies the iptables, and there might be a glitch in the adaption for openwrt.
So you should check coova-chilli setup in this regard, as it has to be modified from orginal, to openwrt-special firewall handling.
There is even a periodic re-config of coova, which might interfere. So I suggest, you run a long lasting tcp-dump session, and check the timing of the bad packets. Any cycle time, or really sporadic ?

Looks like on openwrt the cycle to reconfigure is set to 1h.

aport · March 1, 2020, 10:51pm

I believed I solved the issue.

I noticed that the un-NATted packets were always TCP FIN... it looks like when the chilli auth times out (hotspotsystem is reallllly slow) the conntrack gets cleared, and there's a very small window of time where the FIN can get sent after this and bypass the MASQUERADE.

I changed the default chilli firewall rules to only forward NEW,RELATED,ESTABLISHED connections from the tunnel interface. So far so good...

reinerotto · March 2, 2020, 6:47am

Very interesting. Could you please clarify in detail, which chilli-rule(s) you changed ?
(old -> new)

adayannick · September 14, 2023, 10:45am

Please how did you do practically?

Thanks

frollic · September 14, 2023, 10:48am

hopefully he moved over to opennds, since coova's dead.

adayannick · September 14, 2023, 11:26am

How is openNDS dealing with radius authentication?

frollic · September 14, 2023, 11:49am

the way you script it, if I remember correctly
https://opennds.readthedocs.io/en/stable/

adayannick · September 14, 2023, 12:34pm

I would be grateful to have tutorial or an example. I am a newbie.

thanks in advance