qBittorent.
Clearly something special is going on for bittorrent type traffic, and it's happening inside your modem. I didn't see in the online information any info about what qBittorrent does in terms of marking packets. I don't think this is just about queueing issues in your modem, it's probably more like some kind of throttling or maybe CPU limitations when there are a lot of connections, since bittorrent makes tens or hundreds of connections simultaneously.
also I think modern bittorrent uses UDP? And that might be part of the issue as well.
Thanks for your help, however I want to ensure I am also seeing no issues with normal traffic.
I ran the same fast.com test and pinged an IP from OpenWrt:
64 bytes from 8.8.8.8: icmp_req=1 ttl=116 time=8.03 ms
64 bytes from 8.8.8.8: icmp_req=2 ttl=116 time=20.9 ms
64 bytes from 8.8.8.8: icmp_req=3 ttl=116 time=8.13 ms
64 bytes from 8.8.8.8: icmp_req=4 ttl=116 time=7.92 ms
64 bytes from 8.8.8.8: icmp_req=5 ttl=116 time=7.87 ms
64 bytes from 8.8.8.8: icmp_req=6 ttl=116 time=8.58 ms
64 bytes from 8.8.8.8: icmp_req=7 ttl=116 time=11.3 ms
64 bytes from 8.8.8.8: icmp_req=8 ttl=116 time=7.91 ms
64 bytes from 8.8.8.8: icmp_req=9 ttl=116 time=8.13 ms
64 bytes from 8.8.8.8: icmp_req=10 ttl=116 time=9.63 ms
64 bytes from 8.8.8.8: icmp_req=11 ttl=116 time=8.37 ms
64 bytes from 8.8.8.8: icmp_req=12 ttl=116 time=8.10 ms
64 bytes from 8.8.8.8: icmp_req=13 ttl=116 time=7.86 ms
64 bytes from 8.8.8.8: icmp_req=14 ttl=116 time=8.37 ms
64 bytes from 8.8.8.8: icmp_req=15 ttl=116 time=7.88 ms
64 bytes from 8.8.8.8: icmp_req=16 ttl=116 time=8.80 ms
64 bytes from 8.8.8.8: icmp_req=17 ttl=116 time=8.89 ms
64 bytes from 8.8.8.8: icmp_req=18 ttl=116 time=7.63 ms
64 bytes from 8.8.8.8: icmp_req=19 ttl=116 time=7.64 ms
64 bytes from 8.8.8.8: icmp_req=20 ttl=116 time=8.44 ms
64 bytes from 8.8.8.8: icmp_req=21 ttl=116 time=8.62 ms
64 bytes from 8.8.8.8: icmp_req=22 ttl=116 time=9.14 ms
64 bytes from 8.8.8.8: icmp_req=23 ttl=116 time=7.83 ms
64 bytes from 8.8.8.8: icmp_req=24 ttl=116 time=9.41 ms
64 bytes from 8.8.8.8: icmp_req=25 ttl=116 time=8.63 ms
64 bytes from 8.8.8.8: icmp_req=26 ttl=116 time=8.15 ms
64 bytes from 8.8.8.8: icmp_req=27 ttl=116 time=7.60 ms
64 bytes from 8.8.8.8: icmp_req=28 ttl=116 time=7.88 ms
64 bytes from 8.8.8.8: icmp_req=29 ttl=116 time=54.1 ms
64 bytes from 8.8.8.8: icmp_req=30 ttl=116 time=9.61 ms
64 bytes from 8.8.8.8: icmp_req=31 ttl=116 time=8.12 ms
64 bytes from 8.8.8.8: icmp_req=32 ttl=116 time=9.93 ms
64 bytes from 8.8.8.8: icmp_req=33 ttl=116 time=7.59 ms
64 bytes from 8.8.8.8: icmp_req=34 ttl=116 time=8.89 ms
64 bytes from 8.8.8.8: icmp_req=35 ttl=116 time=8.34 ms
64 bytes from 8.8.8.8: icmp_req=36 ttl=116 time=8.81 ms
64 bytes from 8.8.8.8: icmp_req=37 ttl=116 time=8.38 ms
64 bytes from 8.8.8.8: icmp_req=38 ttl=116 time=7.63 ms
64 bytes from 8.8.8.8: icmp_req=39 ttl=116 time=7.86 ms
64 bytes from 8.8.8.8: icmp_req=40 ttl=116 time=8.34 ms
I see these random spikes upwards still. Now I know you'll say it's just a one off but I monitor my connection consistently with a ping monitor and I see these kind of spikes throughout the day but only when the connection is under some kind of load.
What exactly is happening here, I just wonder if it's the same symptom but the torrents just make it more prevalent with the increased number of connections.
I know that occasional spikes were associated with particular driver/hardware issues in the r7800 which is nearly identical internally to the ZyXEL. I don't know if that's been really fixed or not.
Fair point, I feel as if I have got the router wrong every time 
Archer C7 seems to run out of out CPU when doing torrents. X86 with Intel NIC is apparently creating lag caused by flow offloading (?)
I wish I could just find a router that was good to go, do you have any suggestions?
you can disable those packet offloads, and in fact I think SQM does so automatically. If you have an x86 I would use that, and choose a decent switch with VLANs and QoS: I use a ZyXEL gs1900-24e and a few TP-Link sg108e...
With my whole family at home, zooming, gaming, youtubing etc we can have easily 8 to 10 devices active at once, including 2 or 3 lines of VOIP calls and kids gaming on minecraft etc, as well as at least 3 machines using an NFS server for home directories. it all works well. Of course I'm lucky enough to get gigabit fiber from ATT, but the thread on extreme gaming stability shows that with appropriate settings in customized scripts, it's possible to get rock steady game latency even at 700kbps. Since you have well over 3000kbps each direction, you should be able to get very good latency, less than 15ms round trip time even with regular SQM, and probably better than that if you mark VOIP packets etc and use diffserv4
HOWEVER that will be just latency on exit of your router heading towards the modem. If the modem is borked and fluttering around with insufficient CPU or intentional packet shaping issues to discourage torrenting, etc then all bets are off.
EDIT:
When it comes to latency issues, there are two very very sensitive applications: Gaming and VOIP. I think SQM doesn't quite get to the state where gaming and VOIP are comfortable at speeds less than about 3000kbps but this is more or less "to be expected". However, above that speed, with SQM and diffserv4 I'd be surprised if at least VOIP wasn't very comfortable. Again, assuming your modem doesn't bork things.
1 Like
Nope, SQM doesn't, but cake automatically splits GSO type meta packets on links slower than 1 Gbps. SQM does have some example function for how to disable offloads, but most often it is not needed.
For VoIP I am not so sure, if egress VoIP packets are dscp marked into the high priority tier. VoIP takes ~ 100 Kbps per direction and cake's 1/4 of the total for the highest tin will give sufficient bandwidth for a single VoIP flow down to a net 400 Kbps link. And that tier is sufficiently prioritized to actually give ViOP packets a short circuit!.
For gaming I agree, at low rates things get very iffy for fatter flows than VoIP. But my hunch is using diffserv4's 2nd highest priority tin (which gets up to 50% of the total link) exclusively for gaming packets (and conpletely sparing the highest tin) should go a long way! Sure not as refined and bespoke as you fine script in the other thread, but for games with not total-crap dejitter buffers it could be pretty okay (as long as the link is not totally sluggish). But judging from the other thread de-jitter buffers seems to be super sensitive in many games posing the question with what type of links game developers actually test their games' playability, but I digress...
Sufficient bandwidth yes, but at say 800kbps a 1500 byte packet will be 15ms to serialize, and a 250byte voip packet takes an additional 2.5 so now we're down to 17.5ms and the inter-arrival time for voip packets is 20ms. So if you want to accomodate 1 bulk packet and get at most 1/2 an interarrival time jitter (10ms), you need (1500+250)*8/R = 10, or R = 1400kbps
I'm guessing a gigabit link between two high end gaming machines on the same desk 
That doesn't really matter that much as it is IMHO well within scope for a normal de-jitter buffer. And any service running over the open internet without sufficient jitter resistance is really asking for it...
I have been using VoIP over a 6/1Mbps ADSL link SQM shaped to ~5/0.9 and what could I say/sing:
99 problems but the VoIP ain't one
if you have VoIP problems I feel bad for you son
I got 99 problems but the VoIP ain't one
But as my reaction time is already on the high end for checkers, so I never tested reaction-gated on-line gaming over that link.
That impression is hard to shake, is it?
True that jitter buffers will help a lot, but then if they're too big the audio is fine but the conversation sucks "no you go ahead, no I'm ok, you go ahead". My impression is about 60ms of jitter buffer is the max for comfortable VOIP given the existing delays in the other parts of the system already.
I'm sure there's some math I could do here... in 60 ms we will send 250*3 bytes of voip packets, and then we allow 1x 1500 byte packet... so our bandwidth requirement is (1500+250*3)*8/60 = 300kbps upstream. I guess maybe it's ok so long as the queue system sufficiently avoids droppping packets out of the standing queue of VOIP. In general VOIP suffers pretty badly at about 1% packet loss, so we need 99% of the time this happens.
Then add that you have a family of say 4 (or a small business) and do business conference calls, and want 100% clarity when there are a total of up to say 6 VOIP streams going...
(1500+250*3*6)*8/60 = 800kbps, but again you want less than 1% packet loss... and here you have up to 6*3 = 18 packet standing queue for the VOIP stream while only 1 packet standing queue for the bulk upload... so really I still think anyone who does more than just 1 VOIP stream on a network where people might load the Netflix front page, which tries to grab multiple video previews at once, really wants 1500Kbps to 3000kbps upstream.
If you figure that the queue system will keep 99.7% of the time less than 3 bulk packets ahead of your batch of voip, then for ONE voip stream to have very good audio with a 60ms jitter buffer we need:
(1500*3+250*3)*8/60 = 700kbps
It's really that requirement for better than 99%tile reliability that hurts.
(But notice, this is consistent with your 1 voip stream clarity at 900kbps upload. so the numbers make sense at first glance).
Yeah, the acceptable numbers obviously depend on things like total RTT of the call (which seeing how ISPs occasionally route traffic can be quite unexpected).
This is where CoDel works quite will with its interval it effortlessly absorbs small bursts without switching to drops, but it still reigns in wildly unresponsive flows eventually (cake's inbuilt blue component is a lot better at the reigning in part).
Well, my VoIP provider only allows two concurrent calls with my current plan (one in and one out ) and my old one (while still at 6/1 Mbps ADSL) allowed up to three (but with just one phone we never used more than one anyway).
Except fq_codel or cake will treat the 6 VoIP flows a six flows with their independent intervals, that helps a lot and VoIP traffic being paced naturally has a decent chance of being treated as sparse.
That or an ISP who special cases its own VoIP packets on ingress... that can allow single stream VoIP down to ridiculous low rates, like 385/50 (with shitty codes). I note that ADSL actually allows to high priority ATM cells to interleave or bypass lower priority cells, with cells measuring 53 bytes, that results in much less serialization issue even if a 1500 full MTU packet sits before our VoIP packet in the modem's buffers. And PTM as used in VDSL2 also allows preempting ethernet frames to expedite highe priority data, but I am not sure that this functionality exists outside of the spec documents...
As I said with a little help of the ISP one can push the minimal required rates for acceptable single stream VoIP down a bit more, but that essentially requires to not run one's own traffic shaper (at least not for VoIP packets)
No doubt, the ISP has an easier job of it all with ability to prioritize their own voip. Used to drive me crazy on a Spectrum DOCSIS line... I knew I had sufficient upstream in theory but I was pretty sure that my voip packets which were going to an internet voip provider were getting dropped by the ISP during high traffic times. I would see 4% packet loss...
The solution was to switch to GigE from ATT 
These days with COVID everyone doing zoom calls etc, it would not be insane to see 20Mbps up + down of interactive traffic for a family of 4. Video is more packet drop tolerant though. Anyway I think we can summarize as follows:
- For VOIP you can do it down to around 700kbp upload reliably using SQM and diffserv4, with more tuning and stuff maybe you can go down to 400kbps or so and still get reliable performance.
- For gaming + voip you probably want 1500-3000kbps upload for SQM, or you can do it down to about 900kbps with highly tuned scripts...
- Zoom / Meets / Webex / Jitsi calls need probably close to 1Mbps per call to be comfortable.
- You will get better results when you have extra "slop" in the system. If you want reliability and quality that feels fluid get 2 to 3 times as much bandwidth as the minimum theoretical calcs suggest. Unfortunately this is upload bandwidth and the ISPs still think they're in the "content delivery" business, so they prioritize download.
- I almost forgot, if you want high quality gaming or voip, you can do well by getting a moderate speed dedicated line. A single ADSL 10000/800 kbps line dedicated to gaming will easily handle 2 or 3 consoles when there's no competition with bulk downloads etc.
Thanks for all your help and patience so far with me.
I've got 19.07.5 running on my X86 box now.
Somebody advised I should increase the maximum number of connections from 16384 as my hardware is more than capable, how do I do this?
Second, been reading around here and I've followed the instructions here: How to make ethtool setting persistent on br-lan. Apparently these interruptions can also cause lag spikes on Intel NICs, of which I am using in my X86 box.
What else can I do to optimise this? I read something here about distributing IRQs: [SOLVED] Router (Netgear R7800) introduced latency spikes >100ms. Not sure how I do this, will this also optimise performance?
Many thanks.
Also @dlakelan I saw your post here, mention lag spikes: Ultimate SQM settings: Layer_cake + DSCP marks.
Perhaps I should investigate doing this on my X86 box?
How much bandwidth are you shaping again? All of that was to shape a gigabit fiber while serving NFS and running a squid proxy. For regular routing of less than say 500Mbps symmetric you should be good to go out of the box.
I am on about 50Mb down, 7Mb up with VDSL.
start with your x86 out of the box, just configure SQM and see how it goes. You don't mention what kind of traffic you are interested in here. Are you doing multiple VOIP calls at once, or lots of video chats, or playing e-sports games for a living or what?
Lots of video chats and calls over WhatsApp, etc.
I use Zoom for work which I have found impacted by poor latency, especially when using video.
@dlakelan I hope you can help me configure the priority system with diffserv5, I believe you have experience of this?
All I want to do is put torrents in low priority so they don't make the Internet slow when I have a large download going on.
diffserv5 is some special thing, but diffserv4 is standard.
Your router will be wired correct? So suppose you have eth0 on LAN and eth1 on WAN. You should put SQM with egress control (Upload direction) only on each one. Make upload of the WAN port be your upstream ISP speed (7Mbps) , make upload of the LAN port be your downstream speed (50Mbps). If you do that, and you set your torrent client to use a consistent port, you can use a simple iptables rule to tag packets CS1 and diffserv will put them in the bulk tin.
I'll get you that rule later today. First work on this.
1 Like