Torrents causing conntrack table to overflow

Mushoz · November 4, 2020, 9:37pm

So apparently when this issue happens I am getting spammed by invalid packets from multiple IPs, which showed up when I made conntrack log invalid packets to the log. I did a packet capture with tcpdump and waited for the log to show a message that an invalid packet from said IP was found. I then stopped the capture, and pasted the capture below. I am not entirely sure which of these packets are invalid and why (I am not even sure if they'd even show up in TCPDump?), but maybe this is useful?

root@OpenWrt:~# tcpdump -i any -xx -n -vvv src 197.253.66.127
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
22:25:50.068032 IP (tos 0x48, ttl 117, id 21000, offset 0, flags [DF], proto TCP (6), length 60)
    197.253.66.127.2257 > myExternalP.60781: Flags [S.], cksum 0xb5dd (correct), seq 4069701044, ack 2884318623, win 0, options [mss 1460,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop], length 0
	0x0000:  0000 0200 0000 8038 bc0b cd2e 0000 0800
	0x0010:  4548 003c 5208 4000 7506 9c5d c5fd 427f
	0x0020:  5658 b839 08d1 ed6d f292 b5b4 abeb 359f
	0x0030:  a012 0000 b5dd 0000 0204 05b4 0402 0101
	0x0040:  0101 0101 0101 0101 0101 0101
22:25:50.068128 IP (tos 0x48, ttl 116, id 21000, offset 0, flags [DF], proto TCP (6), length 60)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0x022e (correct), seq 4069701044, ack 2884318623, win 0, options [mss 1460,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 003c 5208 4000 7406 e9ad c5fd 427f
	0x0020:  c0a8 0199 08d1 ed6d f292 b5b4 abeb 359f
	0x0030:  a012 0000 022e 0000 0204 05b4 0402 0101
	0x0040:  0101 0101 0101 0101 0101 0101
22:25:50.068150 IP (tos 0x48, ttl 116, id 21000, offset 0, flags [DF], proto TCP (6), length 60)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0x022e (correct), seq 4069701044, ack 2884318623, win 0, options [mss 1460,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 003c 5208 4000 7406 e9ad c5fd 427f
	0x0020:  c0a8 0199 08d1 ed6d f292 b5b4 abeb 359f
	0x0030:  a012 0000 022e 0000 0204 05b4 0402 0101
	0x0040:  0101 0101 0101 0101 0101 0101
22:25:50.068159 ethertype IPv4, IP (tos 0x48, ttl 116, id 21000, offset 0, flags [DF], proto TCP (6), length 60)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0x022e (correct), seq 4069701044, ack 2884318623, win 0, options [mss 1460,sackOK,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop,nop], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 8100
	0x0010:  000b 0800 4548 003c 5208 4000 7406 e9ad
	0x0020:  c5fd 427f c0a8 0199 08d1 ed6d f292 b5b4
	0x0030:  abeb 359f a012 0000 022e 0000 0204 05b4
	0x0040:  0402 0101 0101 0101 0101 0101 0101 0101
22:25:50.175442 IP (tos 0x48, ttl 117, id 21001, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > myExternalIP.60781: Flags [S.], cksum 0xa0f7 (correct), seq 1385346484, ack 2884318623, win 0, options [mss 1460], length 0
	0x0000:  0000 0200 0000 0000 0000 0000 0000 0800
	0x0010:  4548 002c 5209 4000 7506 9c6c c5fd 427f
	0x0020:  5658 b839 08d1 ed6d 5292 b5b4 abeb 359f
	0x0030:  6012 0000 a0f7 0000 0204 05b4
22:25:50.222372 IP (tos 0x48, ttl 116, id 21001, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0xed47 (correct), seq 1385346484, ack 2884318623, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 002c 5209 4000 7406 e9bc c5fd 427f
	0x0020:  c0a8 0199 08d1 ed6d 5292 b5b4 abeb 359f
	0x0030:  6012 0000 ed47 0000 0204 05b4
22:25:50.222412 IP (tos 0x48, ttl 116, id 21001, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0xed47 (correct), seq 1385346484, ack 2884318623, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 002c 5209 4000 7406 e9bc c5fd 427f
	0x0020:  c0a8 0199 08d1 ed6d 5292 b5b4 abeb 359f
	0x0030:  6012 0000 ed47 0000 0204 05b4
22:25:50.222422 ethertype IPv4, IP (tos 0x48, ttl 116, id 21001, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0xed47 (correct), seq 1385346484, ack 2884318623, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 8100
	0x0010:  000b 0800 4548 002c 5209 4000 7406 e9bc
	0x0020:  c5fd 427f c0a8 0199 08d1 ed6d 5292 b5b4
	0x0030:  abeb 359f 6012 0000 ed47 0000 0204 05b4
22:26:00.320946 IP (tos 0x48, ttl 117, id 2062, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > myExternalIP.48593: Flags [S.], cksum 0xb8e8 (correct), seq 1625666996, ack 1419875905, win 0, options [mss 1460], length 0
	0x0000:  0000 0200 0000 8038 bc0b cd2e 0000 0800
	0x0010:  4548 002c 080e 4000 7506 e667 c5fd 427f
	0x0020:  5658 b839 08d1 bdd1 60e5 b5b4 54a1 9641
	0x0030:  6012 0000 b8e8 0000 0204 05b4
22:26:00.367723 IP (tos 0x48, ttl 116, id 2062, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.48593: Flags [S.], cksum 0x0539 (correct), seq 1625666996, ack 1419875905, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 002c 080e 4000 7406 33b8 c5fd 427f
	0x0020:  c0a8 0199 08d1 bdd1 60e5 b5b4 54a1 9641
	0x0030:  6012 0000 0539 0000 0204 05b4
22:26:00.367772 IP (tos 0x48, ttl 116, id 2062, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.48593: Flags [S.], cksum 0x0539 (correct), seq 1625666996, ack 1419875905, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 002c 080e 4000 7406 33b8 c5fd 427f
	0x0020:  c0a8 0199 08d1 bdd1 60e5 b5b4 54a1 9641
	0x0030:  6012 0000 0539 0000 0204 05b4
22:26:00.367788 ethertype IPv4, IP (tos 0x48, ttl 116, id 2062, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.48593: Flags [S.], cksum 0x0539 (correct), seq 1625666996, ack 1419875905, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 8100
	0x0010:  000b 0800 4548 002c 080e 4000 7406 33b8
	0x0020:  c5fd 427f c0a8 0199 08d1 bdd1 60e5 b5b4
	0x0030:  54a1 9641 6012 0000 0539 0000 0204 05b4
22:26:00.582190 IP (tos 0x48, ttl 117, id 21007, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > myExternalIP.60781: Flags [S.], cksum 0xc0f8 (correct), seq 848475572, ack 2884318622, win 0, options [mss 1460], length 0
	0x0000:  0000 0200 0000 0000 0000 0000 0000 0800
	0x0010:  4548 002c 520f 4000 7506 9c66 c5fd 427f
	0x0020:  5658 b839 08d1 ed6d 3292 b5b4 abeb 359e
	0x0030:  6012 0000 c0f8 0000 0204 05b4
22:26:00.634011 IP (tos 0x48, ttl 116, id 21007, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0x0d49 (correct), seq 848475572, ack 2884318622, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 002c 520f 4000 7406 e9b6 c5fd 427f
	0x0020:  c0a8 0199 08d1 ed6d 3292 b5b4 abeb 359e
	0x0030:  6012 0000 0d49 0000 0204 05b4
22:26:00.634054 IP (tos 0x48, ttl 116, id 21007, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0x0d49 (correct), seq 848475572, ack 2884318622, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 0800
	0x0010:  4548 002c 520f 4000 7406 e9b6 c5fd 427f
	0x0020:  c0a8 0199 08d1 ed6d 3292 b5b4 abeb 359e
	0x0030:  6012 0000 0d49 0000 0204 05b4
22:26:00.634065 ethertype IPv4, IP (tos 0x48, ttl 116, id 21007, offset 0, flags [DF], proto TCP (6), length 44)
    197.253.66.127.2257 > 192.168.1.153.60781: Flags [S.], cksum 0x0d49 (correct), seq 848475572, ack 2884318622, win 0, options [mss 1460], length 0
	0x0000:  0004 0001 0006 66be b77d 32e4 0000 8100
	0x0010:  000b 0800 4548 002c 520f 4000 7406 e9b6
	0x0020:  c5fd 427f c0a8 0199 08d1 ed6d 3292 b5b4
	0x0030:  abeb 359e 6012 0000 0d49 0000 0204 05b4

This was in the log showing I was getting invalid packets:

root@OpenWrt:~# logread -f | grep -i 197.253.66.127
Wed Nov  4 22:24:35 2020 kern.notice kernel: [96393.775968] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=2050 DF PROTO=TCP SPT=2257 DPT=48593 SEQ=283489716 ACK=1419875906 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:24:45 2020 kern.notice kernel: [96404.069522] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=2056 DF PROTO=TCP SPT=2257 DPT=48593 SEQ=2699408820 ACK=1419875905 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:24:55 2020 kern.notice kernel: [96414.511083] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=2058 DF PROTO=TCP SPT=2257 DPT=48593 SEQ=3504715188 ACK=1419875905 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:25:17 2020 kern.notice kernel: [96436.310428] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=2060 DF PROTO=TCP SPT=2257 DPT=48593 SEQ=2430973364 ACK=1419875905 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:25:50 2020 kern.notice kernel: [96468.781560] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=21001 DF PROTO=TCP SPT=2257 DPT=60781 SEQ=1385346484 ACK=2884318623 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:26:00 2020 kern.notice kernel: [96478.927027] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=2062 DF PROTO=TCP SPT=2257 DPT=48593 SEQ=1625666996 ACK=1419875905 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:26:00 2020 kern.notice kernel: [96479.193313] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=21007 DF PROTO=TCP SPT=2257 DPT=60781 SEQ=848475572 ACK=2884318622 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:26:11 2020 kern.notice kernel: [96489.597619] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=21009 DF PROTO=TCP SPT=2257 DPT=60781 SEQ=1653781940 ACK=2884318622 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:26:32 2020 kern.notice kernel: [96511.359737] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=21011 DF PROTO=TCP SPT=2257 DPT=60781 SEQ=4069701044 ACK=2884318622 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)
Wed Nov  4 22:27:15 2020 kern.notice kernel: [96554.033940] nf_ct_tcp: invalid packet ignored in state ESTABLISHED IN= OUT= SRC=197.253.66.127 DST=myExternalIP LEN=44 TOS=0x08 PREC=0x40 TTL=117 ID=21013 DF PROTO=TCP SPT=2257 DPT=60781 SEQ=43169204 ACK=2884318622 WINDOW=0 RES=0x00 ACK SYN URGP=0 OPT (020405B4)

Interestingly, I closed the torrent client and and lo and behold: Connections from said IP were part of connections still lingering around in the conntrack table:

root@OpenWrt:~# cat /proc/net/nf_conntrack | grep -i 197.253.66.127 | wc -l
11

While just 11 connections isn't a whole lot, I was getting invalid packets from a ton of different IPs, so they will add up. Seeing as how the exact same IP was generating invalid packets AND showed up with lingering connections in my conntrack table make me believe that these issues are related. I am not sure where to go from here though.

trendy · November 4, 2020, 10:25pm

These are SYNACK packets. Most likely they are coming as response to a SYN that the torrent client sent.

Mushoz · November 4, 2020, 10:33pm

But why are they considered as invalid?

trendy · November 4, 2020, 10:55pm

I cannot say for sure. One would have to see the whole picture.
Just by watching some SYNACKs arrive, sure it looks strange. You should not only filter src for this host, but also dst, to see what are the responses from the torrent client.

lleachii · November 4, 2020, 11:37pm

Ya kno...I have placed some interesting rules in my firewall, just to prevent quirky things...and two rules may help you.

Try this:

config rule
	option proto 'tcp'
	option name 'Block_In_Not_SYN'
	option src '*'
	option target 'DROP'
	option extra '! --syn -m conntrack --ctstate NEW'

config rule
	option name 'Block_FWD_Not_SYN'
	option proto 'tcp'
	option src '*'
	option dest '*'
	option target 'DROP'
	option extra '! --syn -m conntrack --ctstate NEW'

Place at the top of your rules, above the rule that opens the torrent port

(Please be aware of what blocking initial TCP packets that don't begin with SYN actually mean. To date, I haven't seen any legitimate traffic hindered by this rule.)

Also, do you have this checked?

Edit - see:

https://www.frozentux.net/iptables-tutorial/chunkyhtml/x6249.html

kousu · January 4, 2021, 11:59pm

I was having this problem last week (version: OpenWrt 19.07.5, r11257-5090152ae3). I seem to have fixed it! I don't have my exact logs saved anymore, and I don't really want to disrupt the house's internet just to reproduce them, but I will try to report as much as I can from notes and memory.

Previously, in my /etc/sysctl.conf I had set:

root@router:~# cat /etc/sysctl.conf 
net.netfilter.nf_conntrack_max=512

which is admittedly a pretty tight throttle. My router was crashing under load and I was worried it was running out of RAM. This throttle seemed to be okay -- it seems each member of my house uses, on average, 50 connections at a time, and once I put it in we didn't cross it for a couple weeks.

It was fine until someone turned on transmission-gtk. I noticed after about half an hour when my web browsing started lagging or failing or missing parts of pages. Luckily, I happened to have a LuCI tab open, and that connection was still working, and I have luci-app-statistics installed, so I was able to see on http://router.lan/cgi-bin/luci/admin/statistics/graph/conntrack that over the last hour the conntrack pool had steadily climbed until it plateaued at about 500, which was just about the time I started noticing problems.

I wish I had a screenshot of the conntrack graph, because it made the problem immediately obvious. It was something like this:

After some patience and multiple tries, I was able to establish an ssh connection into the router (a LAN connection is still a connection that needs a slot in conntrack) and, without rebooting or even breaking any active sessions in the house, do

$ ssh root@router.lan
root@router:~# sysctl -w net.netfilter.nf_conntrack_max=1024

and I could immediately use the web again. During that time I did

root@router:~# opkg update && opkg install conntrack

so that I could investigate with

root@router:~#  conntrack -L | grep ASSURED | wc -l

which is pretty similar to @Mushoz's check (I wish I'd googled better! I don't know how I missed /proc/net/nf_conntrack; conntrack is not a small package and I had to find space for it. Thanks for sharing the tip, @Mushoz).

Within half an hour we hit the limit of 1024 again and web browsing broke again.

While it was broken, from slicing up the conntrack table with different combinations of grep and wc, I learned:

nearly all (>90%) of connections in conntrack were from the laptop running transmission-gtk; every one of them was TCP and marked both "ESTABLISHED" and "ASSURED"
on the laptop running transmission-gtk, netstat reported much less, on the order of 10s of connections
I know from watching transmission-gtk's Peers tab that it is definitely using mixed TCP/UDP. About half of the connections it has are over UDP. But they aren't showing up, or at least not showing up for long, in conntrack.

So it seems that for some reason, ghost TCP connections are piling up on the router, even though the laptop isn't using them anymore.

I googled for advice and found these articles:

They both suggested taking a second look at net.netfilter.nf_conntrack_tcp_timeout_established. By default on Linux, that's set to 5 days ("432000") (!), and even on OpenWRT it's 2 hours ("7440") (https://dev.archive.openwrt.org/ticket/12976). Because a NAT proxies connections, connections going through the NAT count against it.

Now, the connection entries look like

tcp      6 5544 ESTABLISHED src=192.168.9.124 dst=91.22.43.1 sport=51489 dport=443 packets=589 bytes=43953 src=91.22.43.1 dst=87.11.33.22 sport=443 dport=51489 packets=882 bytes=70095 [ASSURED] mark=0 use=1

After reloading the list over and over, I realized that third entry, the "5544", was a timer, because it was counting down. And that made : all of the timers were under 7440, none above it, and many near it, exactly the value of the setting that Akamai's long-suffering network engineer suggested to look at.

So, while there's many other settings, but this turned out to be the relevant one. I now have this in my /etc/sysctl.conf:

# https://ixnfo.com/en/tuning-nf_conntrack.html
# https://developer.akamai.com/blog/2012/09/27/linux-tcpip-tuning-scalability
net.netfilter.nf_conntrack_max=8192 
net.netfilter.nf_conntrack_tcp_timeout_established=180

I've been running with this for 5 days now. You can see the improvement here:

(uh I'm not really sure why that graph is missing part of it's data, )

For most of this week, Torrents have been running without trouble. The torrenting laptop happens to be the only older one still using 2.4GHz, so you can see its traffic pretty clearly here:

(mind you: counterintuitively the green part TX on that graph means downloading and the blue RX means uploading, because collectd generated it from the perspective of the router which has the opposite perspective on direction)

So simply throttling the timeout harder fixed it. We're not even breaking my original throttle of 512, even with the torrents running, but I'm leaving it at 8096 for now to give us some head-room. I bet could probably even turn the timeout down to 30 seconds without causing any problems while saving even more router memory.

By the way, I'm pretty sure this isn't a hard timeout; judging from conntrack -L, the timer gets reset often. Guessing, again, I'd say that timeout is only timing the time between packets. When data goes through it it resets it, like a deadman's switch. I can't imagine why you would ever want to keep a TCP connection open for 2 hours (much less five days!!) without any data on it anyway; and if you do...there's TCP keepalives for that.

So, I haven't looked any deeper so I'm can't say exactly what is going on; guessing, I'd say something is blocking FIN or RST packets, or maybe torrent clients just don't care that much about following all the rules for reliable connections. If you keep tcpdumping your connections, @Mushoz, maybe try running tcpdump on both your OpenWRT's WAN side, LAN side, and your torrenting laptop/desktop all at the same time, and open them up in parallel Wireshark sessions to look for if there's a FIN packet on the WAN side that doesn't make it through to the the desktop.

For future-proofing, here's my full /etc/sysctl.conf

$ ssh root@router.lan cat /etc/sysctl.conf
root@router.lan's password: 
# Defaults are configured in /etc/sysctl.d/* and can be customized in this file

# curtail the memory used on the NAT by aggressively killing idle connections
# we don't need them here. we're just using facebook or w/e
# see also
# https://ixnfo.com/en/tuning-nf_conntrack.html
# https://developer.akamai.com/blog/2012/09/27/linux-tcpip-tuning-scalability
net.netfilter.nf_conntrack_max=1024
net.netfilter.nf_conntrack_max=8192
net.netfilter.nf_conntrack_udp_timeout=27
net.netfilter.nf_conntrack_udp_timeout_stream=61

#net.netfilter.nf_conntrack_tcp_timeout_close=5
#net.netfilter.nf_conntrack_tcp_timeout_close_wait=5
#net.netfilter.nf_conntrack_tcp_timeout_fin_wait=5
#net.netfilter.nf_conntrack_tcp_timeout_last_ack=2
#net.netfilter.nf_conntrack_tcp_timeout_max_retrans=6
#net.netfilter.nf_conntrack_tcp_timeout_syn_recv=5
#net.netfilter.nf_conntrack_tcp_timeout_syn_sent=45
#net.netfilter.nf_conntrack_tcp_timeout_time_wait=30
#net.netfilter.nf_conntrack_tcp_timeout_unacknowledged=100
# 
net.netfilter.nf_conntrack_tcp_timeout_established=180

You can see I was trying to tune conntrack before to control this stuff, but I ended up replacing all of the timeouts on the different parts of TCP with the single timeout net.netfilter.nf_conntrack_tcp_timeout_established=180. And that's working way better. I also have an aggressive UDP timeout, which maybe explains why UDP conntracks were never a problem -- and this UDP throttle doesn't seem to have impacted the actual bandwidth the torrents can manage, nor Facebook/Discord/Telegram/Zoom/WebEx video chat, so I think it's fine.

trendy · January 5, 2021, 9:51am

That is a pretty tight number, taking in consideration that the default is 16.000

Which router is that?

How many members, how many devices?

The torrent client is called transmission, transmission-gtk is the particular version with the graphical interface based in gtk.

kousu · January 6, 2021, 1:06am

It's low but it's working for us! And my conntrack graphs are showing it's plenty, for now. I was choosing to detect memory leaks earlier than to have the entire thing come crashing down and leave me no logs to work with. Anyway you can see in my final config that it's back up to ~8000, which seems gargantuan to me but okay.
https://openwrt.org/toh/tp-link/archer-c50 v4; it's a 8/64 model, so it's in the midrage for OpenWRT, according to the specs warning. I don't know what the crash was because I haven't invested in getting perfect monitoring, and when it crashes I lose all my logs, especially the dmesg. Besides setting net.netfilter.nf_conntrack_max=512, I installed SQM and upgraded to 19.07.05 from 19.07.04, and added @daily reboot to my crontab, and I haven't had a crash since, so maybe some combination of those things fixed it. But now I'm speculating, I'm sorry I can't give more accurate insight here
It's just a household. I don't really want to give out more details than that about my people, but not large.
Sure, but transmission-gtk is a different app than transmission-daemon. They share a base but they're compiled into different products, and most distros package them separately. I just wanted to be clear in case the details turn out to matter.

Anyway, if anyone else having this issue can try out the fix I dug up, I would love to know if it fixes the problem for you.

neil1 · January 18, 2021, 5:18pm

torrents cause SQM latency to jump very high
and number of connections also jump to like 40-50%

this spike is when i opened the torrent client

moeller0 · January 18, 2021, 6:19pm

Well the torrent client opened ~800 connections in short order and these 800 flows take a while to handle, either by servicing them or by "aging" enough to merit a drop. IMHO the best you can try to do is to move torrent packets into a lower priority tier (and use layer_cake.qos), but as so often that is a bit tricky for ingress. The second best would be to try to move the torrent client to its own computer and configure cake for per-internal-IP fairness so that the fall-out of the over-zealous torrent client is mostly restricted to that machine... But no matter what any thing that opens many connections is a short time is going to make cake work harder...