Did I just get DDOS'd?

Trying to work out what just happened to my network. About 30 mins ago all WAN connectivity dropped. Logged into the router and the System Log was showing hundreds of these messages:

Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.381376] net_ratelimit: 155 callbacks suppressed
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.381379] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.403920] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.423707] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.432854] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.449749] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.456881] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.658017] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.664896] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.681666] nf_conntrack: nf_conntrack: table full, dropping packet
Thu Mar 24 14:04:14 2022 kern.warn kernel: [1471832.688280] nf_conntrack: nf_conntrack: table full, dropping packet

Checked the current conntrack count and it was indeed hitting the 16384 limit. Odd.
So I temporarily increased the conntrack limit to 32384 and connectivity was restored.
Almost immedately the conntrack connections shot up to about 25000 but then, a moment later began to drain, and within about 30 seconds was back down to 500 which is the normal level and has stayed there ever since.

This has never ever happened before. Is it a potential DDOS? If so, any way of telling which service on the router was being hit?

Is there any way to protect against this? There are a few threads discussing the 'right' level to have conntrack max connections set at, as I have 32GB RAM in this router in theory there should be no issue setting it far far higher, but this seems like a band-aid rather than a fix.

  1. Run tcpdump on WAN to see the traffic.

Thanks, but I meant after the fact, as it's not currently ongoing.

Would the firewall logs capture it?

Not unless you enabled logging on some rule that blocked it - or use something like softflowd. Otherwise, it would just count hits of packets/bits the the rules/chains.

1 Like

I wouldn't immediately think of a DDoS being in progress, these limits are set with low-end devices in mind and a bit low for today's networks (WAN speed, number of connected clients, modern web with resources pulled from everywhere). Especially with faster connections (and correspondingly fast hardware) you might want to bump the settings a bit.

5 Likes

Some guidelines -

1 Like

Perhaps the number of internal clients, or the number of simultaneous connections they are opening, is too high for your current configuration.

How big is your network? I guess it must be pretty big, to justify 32GB of RAM on a router! How many WAN interfaces do you have?

The size of the hardware is just down to some spares I had available. I had an 8-core Intel C2758 32GB system that died in my NAS which I then replaced, but it turned out the failure on the C2758 was a known manufacturing fault, which the vendor then replaced free-of-charge so I ended up with the spare hardware!

The network is not big at all, I have one WAN interface, plus 3 VPN point to point tunnels and about 50 clients. Which is why I suspect some sort of DDOS rather than anything on my own network. The typical level of active connections is never normally above ~700.

People have discussed the 16k setting in OpenWrt...I cannot recall the link, but will edit if I find it...it also cover posters who want to set it over 64k (more ports than as single IP) or over 48k (as some kernels reserve ports)...and the memory keeps track of SRC and DST ports...

I've seen this setting for years...but I do a agree 50 clients could easily break that in any given moment in time.

I agree with @slh and @eduperez - this is not immediate evidence of a DDoS attack.

I've seen the thread you're referring to, it's a rather opinion driven matter and there doesn't seem to be a clear consensus on what is 'right'. Netfilter/iptables has been around in forms since 1998 and a lot of the reference material is wildly out of date for modern systems. The best reference I can find in terms of the impact of the max connections/hashtable size is here: https://wiki.khnet.info/index.php/Conntrack_tuning

But even that is not specific about what is recommended, rather it focuses on the inter-relation between the hash-table size and the max connections limit. It suggests that best performace is found when CONNTRACK_MAX = HASHSIZE (or '# of buckets'), on the logic that the hash lookup is fast, but the bucket linked list iteration is slow. It doesn't really make any recommendations beyond this, however it does state:

Now suppose your firewalling-only box has 512MB of RAM (a decent amount of memory considering today's memory prices). You have to spare a bit of memory for a few applications (syslog, etc.): 128MB should really be big enough for a firewall in console mode, for example. The rest can be dedicated to conntrack entries. Then you could set both CONNTRACK_MAX and HASHSIZE approximately to:
(512 - 128) * 1024^2 / 308 =~ 1307315 (instead of 32768 for CONNTRACK_MAX, and 4096 for HASHSIZE by default). Since Linux 2.4.21 (thus Linux 2.6 as well), hash algorithm is happy with "power of 2" sizes (it used to be a prime number before). So here we can set CONNTRACK_MAX and HASHSIZE to 1048576 (2^20), for example. This way, you can store about 32 times more conntrack entries than the default, and get better performance for conntrack entry access.

So it would seem on any system with more than 512MB of RAM available for firewall duties, setting CONNTRACK_MAX to very large number like 1,048,576 is absolutely fine, provided you also adjust the HASHSIZE.

If you just adjusted CONNTRACK_MAX without increasing the hashtable size then you'd incur a performance hit as you'd be storing CONNTRACK_MAX/HASHSIZE entries in each linked list.

Isn't this a misunderstanding though? Conntrack keeps track of the source and destination IP address in addition to the ports, so it's entirely possible to have more conntrack entries than ports, especially if under a DoS attack because every connection to a valid service would be ESTABLISHED and then have to wait to timeout.

Example:

root@router:~# cat /proc/net/nf_conntrack
ipv4     2 tcp      6 98 ESTABLISHED src=192.168.1.251 dst=13.107.21.200 sport=52259 dport=443 packets=50 bytes=61498 src=13.107.21.200 dst=[REDACTED] sport=443 dport=52259 packets=56 bytes=82924 [ASSURED] mark=0 zone=0 use=2

Your exact point was noted in the thread, and elaborated on - so I'm unsure of any misunderstanding. Connecttrack also handles non-NAT too...so memory could be used in a more exponential manner than a linear 0-65k (i.e. from a port point-of-view).

Sure, also recall the router and client reserve special ports for themselves; but yes there is a 5-way combination (IP-IP-Port-Port).

Lastly, it also tracks non-TCP and non-UDP depending on packet - it seems you're only considering TCP and UDP.

Basically, I agree. :smile:

I wasn't trying to disagree with you, but glad we're on the same page! :grin:

I think a lot of the discourse on netfilter conntrack refers to examples from client devices where it would be impossible to have more outgoing connections than ports available (over time, because of course there is still the timeout period for stale connections). In a router/firewall scenario or a device accepting incoming connections there is no limit (or maybe a theoretical soft limit at the entire IPv4 address space multiplied by the number of ports).