Check my QOS config. (Some SQM issues.)

(For the people who don't wanna read the whole story just check out the SQM config at the bottom... I like to write the whole story k :frowning: )

Hi. I've been using LEDE for a couple of months now on my 841N V9, I initially flashed a community build on my router since if I flashed the normal stock one It didn't leave any space for SQM or UPNP. Now after a while like around 24hours of up time whenever I opened the Luci page I was met with CPU spikes, then someone mentioned this image builder in the thread and I build my own image via that (Might get Linux later on and build a image using that as well instead of the online tool). I just solved the CPU spike issue with a crontab that restarts my router every morning (I would do this regardless if the router was running stable... helps me sleep at night).

I have a relatively slow ADSL2+ AnnexM 4mbit download / 1mbit upload link.

I am currently met with these issues in SQM (and my main issue has been consistent with all the other firmware I've tried Gargoyle, DD-WRT). The issue is that when gaming my pings are relatively fine but it spikes up and down like every once in a minute or so which gets very annoying. I've tried to do something about my ISP but alas no solution, they even made me downgrade to 4mb from 8mb because "My line can't support it" which I call BS on because they should just disable AnnexM for my line which would make it a lot stable like it used to be.

My second but weird issue is that when I queue for a game of CSGO and my brother is in the same party as me the pings in the game menu rises up and doesn't come down even though the constant ping I'm doing to googles DNS server via CMD is perfectly fine. I have to disable SQM for this time period to get a match, However, Jump in the game and this isn't much of an issue.

My SQM Config:

config queue 'eth1'
    option interface 'pppoe-wan'
    option debug_logging '0'
    option verbosity '5'
    option qdisc 'cake'
    option script 'piece_of_cake.qos'
    option linklayer 'atm'
    option overhead '40'
    option download '3600'
    option upload '900'
    option qdisc_advanced '1'
    option squash_dscp '1'
    option squash_ingress '1'
    option ingress_ecn 'ECN'
    option egress_ecn 'NOECN'
    option qdisc_really_really_advanced '1'
    option iqdisc_opts 'nat dual-dsthost'
    option eqdisc_opts 'nat dual-srchost'
    option enabled '1'

Oh and my Line sync rate on my modem is 4608/1095 (I'll leave a screenshot of my line stats below just in case)

image

Your line is almost certainly capable, you have 24.6dB SNR downstream and 12.8dB SNR upstream margin. Unless your line is really poor, you would expect that to be closer to 6dB if your line was syncing at its limit, highly unlikely it was struggling on 8Mbit.

Its not like you are even seeing any benefits of Annex M, I used to get 1Mbit upload on Annex A and I could only hit 5800Kbit downstream due to being a long line. Annex M is only useful on really short lines, it just reduces your downstream on longer ones and may even be causing the problem as your upstream errors are much higher than your downstream.

To both those points, I would reply with... "IKR!!!" But alas my dumb ISP doesn't understand this shit... Coming up with the final response after a lot of headaches "We don't support 8mb in this area... why did you even get 8mbit?" Followed by "We can't disable AnnexM for your line." and when I asked them Why? they responded with "We just can't and if you want to get this fixed disconnect your line and apply for a new one... Which I don't wanna do as I don't pay for my internet (17 Y/O kid living with his parents) and it would just be an overall hassle.

Now to the part where I could just upgrade to 8mbit on the existing AnnexM line... the issue here is that when I do get an 8mbit line the SNR drops to 9-12db for both the upstream and downstream but in peak hours it just get's horrendously bad and I get disconnections where all of this began. right now those stats are what I'd consider the "Peak Best Stats" of the day... The worst stats on this line make the SNR go down to as low to 16-18snr which is not bad but considering that if I upgrade to 8mbit that drop won't let the line be stable.

I think my main issue with SQM the jerky ping spikes every now and then are due to my line and just the pure distance in between my house and the exchange but there isn't a thing I can do about it.

Point to be noted when I was on 8mbit without AnnexM and just the normal ADSL2+ that my ISP provides I had SNR between a stable 12-18db for download and 7-11db for the upload where 12db and 7db being at the worst time of the day and 18 and 11db being the best time of the day respectively for download and upload.

At this point every time I try to complain about this, I'm met with a lot of BS so I can't really try anymore.

Intersting, could you please post the output of
tc -d qdisc
tc -s qdisc

and it would be interesting if you could try to set-up and use flent (https://flent.org) to run a rrul test for several minutes, as that should show periodic latency increases quite nicely.
When I had similar issues it turned out that this was a wifi issue where some periodic wifi process (either in my client or the AP) made latency shoot up even for tests that where purely running from a wired client, diabling the wifi and repeating the wired test conclusively fingered wifi as the culprit. Please note that disabling wifi here is not the remedy but rather a means to diagnose and locate the potential root cause.

P.S.: with typical ADSL links where the ISP has no additional shaper, it might be possible to increase the uplink bandwidth to almost 100% of the reported sync rate of the modem (just leave a slither so that PPPoE LCP packets always find a "slot".)

I'll just real quickly post a output of the following for you.

tc -d qdisc

qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev wlan0 root refcnt 2
qdisc cake 800d: dev pppoe-wan root refcnt 2 bandwidth 900Kbit besteffort dual-srchost nat rtt 100.0ms raw total_overhead 22 hard_header_len 22
linklayer atm overhead 40 mtu 2047 tsize 512
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
qdisc cake 800e: dev ifb4pppoe-wan root refcnt 2 bandwidth 3600Kbit besteffort dual-dsthost nat wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
linklayer atm overhead 40 mtu 2047 tsize 512

tc -s qdisc

qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 23850766643 bytes 22347219 pkt (dropped 0, overlimits 0 requeues 7)
backlog 0b 0p requeues 7
maxpacket 1392 drop_overlimit 0 new_flow_count 14 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 3910917972 bytes 23738500 pkt (dropped 0, overlimits 0 requeues 14)
backlog 0b 0p requeues 14
maxpacket 1472 drop_overlimit 0 new_flow_count 22 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev wlan0 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 800d: dev pppoe-wan root refcnt 2 bandwidth 900Kbit besteffort dual-srchost nat rtt 100.0ms raw total_overhead 22 hard_header_len 22
Sent 551873365 bytes 3397385 pkt (dropped 860, overlimits 1898539 requeues 0)
backlog 0b 0p requeues 0
memory used: 199104b of 4Mb
capacity estimate: 900Kbit
Tin 0
thresh 900Kbit
target 20.2ms
interval 115.2ms
pk_delay 4.3ms
av_delay 325us
sp_delay 18us
pkts 3398245
bytes 552667676
way_inds 605701
way_miss 51927
way_cols 0
drops 860
marks 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1696

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
Sent 5434532045 bytes 4531005 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 800e: dev ifb4pppoe-wan root refcnt 2 bandwidth 3600Kbit besteffort dual-dsthost nat wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 5469907983 bytes 3976622 pkt (dropped 554383, overlimits 8134872 requeues 0)
backlog 0b 0p requeues 0
memory used: 251968b of 4Mb
capacity estimate: 3600Kbit
Tin 0
thresh 3600Kbit
target 5.0ms
interval 100.0ms
pk_delay 19.4ms
av_delay 12.1ms
sp_delay 101us
pkts 4531005
bytes 6303425309
way_inds 342095
way_miss 62312
way_cols 0
drops 554383
marks 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1696

I'll give flent a go later tonight.
I'm using Ethernet to connect to the router so that turns away the wifi being the issue here...

Could you can to explain more on that ADSL Links part? how exactly would I leave a slither? Would be nice to read up on this.

Not necessarily, as long as the wifi radio is still powered up/active you still might incurs processing costs even if you are not actively using the wifi yourself. If I remember correctly, I fixed my problem by turning of wifi of an nominally unrelated device which periodically was wreaking havoc on the air waves...

The tc outputs look nominal, so nothing much to talk about.

Ah, with the correct link layer account as you already set it up I would try to set the upload bandwidth to say 0.99 * 1095 = 1084.05 (or make it 1080 to keep it nice and round), unless your ISP operates a shaper as well, this will not increase your bufferbloat, but will give you ~10% more bandwidth; sure only 180Kbps more, but with your link I believe every bit will help :wink:
Please note, that on the ingress/downloading side this will not work, as you need to set you shaper low enough to avoid "back-spill" into the unmanaged buffers of your ISPs DSLAM/MSAN (if those buffers were well managed you would never see bufferbloat on ingress so I think it is a given that they are at least under-managed), typically something like 85-90% of the sync rate will work okay (if the proper link layer accounting has been selected).
BTW, how did you deduce your 40 bytes overhead?

Mhm, I'll definitely try to test with flent once I get the time to do so. Once I end up testing it should I turn the wireless off and test it to check If it's the wifi radio or not? Also, what would be the fix if it does end up being the wifi radio?

Glad that looks alright... I take it that then SQM is running fine?
Also, does the config look optimal? Last thing but without using Per-host isolation (As in the SQM guide) Torrents hogs all the speeds but say I'm running a torrent with doing a VoiceChat session on discord... the pings aren't as spiky on the same pc and websites while slower still end up loading smoothly... although when using Per-host isolation discord lags a lot and I get a lot of cuts when doing voice chat and loading even web pages slows down a bit. I take it this is how it's supposed to be to stop torrents from hogging all the bandwidth from all of your networks and instead it distributes the bandwidth on per local IP basis?

Alright seems simple enough... but what would happen if my ISP does operate a shaper? I have no way of knowing... would I just see more bufferbloat and high latency after setting my uplink to 99%? Or Is there another way to tell this? I initially started off with 900 when I was on gargoyle and then I just kept lowering the uplink number until the pings were fine (Not spiking that much during an upload test). is setting my uplink to 99% really going to be fine?

Lastly, I got the 40bytes overhead by using the atm overhead script. Now... It told me the that the overhead was 40bytes so I just put in 40bytes. Now, do note that I just left the script running and came back to it almost half a day later, so idk if someone used the internet during that time. Is that going to be fine or does that affect the overhead that it deduces for my line?

If the overhead is correct I can just set my uplink to 99% right? Latency/Bufferbloat should remain as it is? I'll try this right now infact while people use the internet to give me a good idea of how it is.

Yes, that is something I would do, tests with and without the radio active (as well as tests over wifi). Make sure to test for at least 5 minutes or even 10, otherwise slower but periodic events are harder to assess.

No idea yet, that really depends on the root cause of the issue, so the first step would be to figure out whether wifi is to blame at all...

Well, from looking at the output it seems so, but you still have some residual issues so I would say it is not working fine.

Well do the torrents run on the same computer as discord (no idea what that is)? The idea with per-host isolation would be to move torrenting to its own IP address/host, so that the other hosts are less affected. But if you already torrent from a different host this does not seem to work that well for you.

Yes, the test would be to set downloading to Around 50% of sync and uploading to 99% and see whether the measured latency increase under wan saturation stays acceptable or not (acceptable means you need to decide how much added latency you are willing to tolerate). Once you have a decent bufferbloat measurement method (if in doubt start with dslreport's speedtest, here are some ideas how to configure it: https://forum.openwrt.org/t/sqm-qos-recommended-settings-for-the-dslreports-speedtest-bufferbloat-testing/2803) you can then iteratively try to get adjust the bandwidth until you find the highest bandwidth that still results in acceptable latency under load increases. Case in point my ISP shapes a 10048 uplink to around 9600, which I heuristically deduced with the method above (and this shaper setting was later corroborated by users that got information about the true shaper setting).

Well, one thing to notice is that the ATM encapsulation alone adds ~10% overhead, so unless you used/configured the correct link later under gargoyle that setting of 90% is similar to 100% with sqm and linklayer adjustments for ATM. But the nice thing is you do not need to trust my words, just empirically figure it out and select the tradeoff between latency under load increase and bandwidth sacrifice you are comfortable with. All I am trying to do is elucidate your options and the mechanisms, you make the final choice.

Well, the only atm overhead script I know would have produced two plots, if you post these I am happy to comment upon these; that said 40 bytes looks quite reasonable.

What I would ike to ask you, is to post the outcome of these experiments to this thread please.

Best Regards

Yes I'm taking about when the torrents are running on the same computer as discord (Discord is a Text/Voice Chat client for gamers). When I'm in a voice with someone their voice sometimes cuts out and the pings in the client as shown in a graph goes up and down alot... If i remove the advanced string options for Per-host isolation this goes away however.. a torrent running on my machine will screw up everyone else's share of bandwidth.

I'll do a couple of tests later tonight with the down-link/up-link values along with flent (If i can get it to run on windows by installing the python module/package...). I have read that post and have already set up the appropriate settings for dslreports however you can take a look at the amount of down-link/up-link streams for my slow connection... the post suggested 16/16 but that may just be too much for my line so instead I used 8/2

In gargoyle if i used pppoe it seemed to automatically adjust for ATM encapsulation as i had to input values closer to my sync rate for 4mbit... though it didn't tell exactly how many bytes was it adjusting for.

Well... about that, I lost the files since i did this like a little over a month ago and my pc broke down which resulted in my files being lost. I can redo the test with the script and it indeed seemed to put out two plots.
I used the script that was in the SQM guide.
This one (https://github.com/moeller0/ATM_overhead_detector)

I'll definitely post any tests I do on this thread.

Thank you for being with me so far :slight_smile:

This is to be expected, as all per-host isolation does is to "limit then pain" caused by the under-controlled torrent flows to the IP-address running the torrent client. Nut have you tried to configure the torrent client to use only restricted bandwidth and a restricted number of concurrent flows?
Also, if discord uses typical VoIP dscp markings you might get a better experience if you switch from piece_of_cake to layer_cake (even though with your link you are heavily constraint by total available bandwidth so this might not help as much, but I think this is still worth trying).

When I had this issue I initially installed virtualbox and created an ubuntu VM in which I installed and ran flent... with your available bandwidth your computer would need to be truly ancient for the VM not to be easily able to saturate your link...

Ah, I know basically nothing of gargoyle...

Ah, good, if the fitted stair function was a good match for the data the 40 bytes are a decent estimate.

What exactly is the difference between piece_of_cake and layer_cake? Which one should I be using? A quick google search told me (and I stumbled on a reply of yours on a post) and what I got from reading that was that layer_cake just has different classes to put packets in instead of just one class that piece_of_cake does... and that using layer_cake uses up more CPU since it has to classify packets in either of those 3 classes. But layer_cake uses dscp markings to put each packet in the said class. You also put in brackets "(well “cake diffserv3”)" Does that mean that it's the same as piece_of_cake but with 3 classes instead of just a single one?
Lastly what if I don't know if my applications use dscp markings? what does layer_cake do then? Is it worth it to just use layer_cake instead of piece_of_cake just in case some applications do make use of it? My router has no problem handling my slow connection so I have CPU power to spare.

If I'm wrong about anything in my understanding of the difference between the two please do tell me.

Edit:
After reading up some more on layer_cake and diffserv it seems by the output that I got that it only applies the so-called Priorities to the uplink... the downlink still only has one class per say.

And with that, I found out that the Voip client Discord does use dscp markings but It won't make much of a difference since I limit my torrents upload speed to 5kbytes anyway... so that didn't bother me as much as the downlink. Running a torrent everything is slow on my computer but, on my phone webpages load just fine and the torrents speed does go down. I suppose the only workaround is to limit the downlink from my torrent client.

The flent tests will have to wait then... It'll take a day or two for me to download, install and setup the virtual machine and afterward do the tests.

Well, I might do the tests again sometime soon. Does my line need to be idle when doing this? or does it not matter?

Lastly, I did the tests and uplink set to 1050 seems good enough... I'll probably have to fine tune this later since I tested this at 3am here and at peak times the results might end up being different.

Downlink = 3600, Uplink = 1050, qdisc = cake/piece_of_cake

Downlink = 3600, Uplink = 1050, qdisc = cake/layer_cake

My config now:

config queue 'eth1'
option interface 'pppoe-wan'
option debug_logging '0'
option verbosity '5'
option qdisc 'cake'
option linklayer 'atm'
option overhead '40'
option qdisc_advanced '1'
option squash_dscp '1'
option squash_ingress '1'
option ingress_ecn 'ECN'
option egress_ecn 'NOECN'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'nat dual-dsthost'
option eqdisc_opts 'nat dual-srchost'
option enabled '1'
option download '3600'
option upload '1050'
option script 'layer_cake.qos'

Output of tc -d qdisc:

qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc cake 8010: dev pppoe-wan root refcnt 2 bandwidth 1050Kbit diffserv3 dual-srchost nat rtt 100.0ms raw total_overhead 22 hard_header_len 22
linklayer atm overhead 40 mtu 2047 tsize 512
qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
qdisc cake 8011: dev ifb4pppoe-wan root refcnt 2 bandwidth 3600Kbit besteffort dual-dsthost nat wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
linklayer atm overhead 40 mtu 2047 tsize 512

Output of tc -s qdisc:

qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 246022165 bytes 228086 pkt (dropped 0, overlimits 0 requeues 5)
backlog 0b 0p requeues 5
maxpacket 1486 drop_overlimit 0 new_flow_count 8 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth1 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 42372284 bytes 204900 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8010: dev pppoe-wan root refcnt 2 bandwidth 1050Kbit diffserv3 dual-srchost nat rtt 100.0ms raw total_overhead 22 hard_header_len 22
Sent 4960005 bytes 8863 pkt (dropped 630, overlimits 5978 requeues 0)
backlog 0b 0p requeues 0
memory used: 35712b of 4Mb
capacity estimate: 1050Kbit
Bulk Best Effort Voice
thresh 65624bit 1050Kbit 262496bit
target 276.8ms 17.3ms 69.2ms
interval 553.7ms 112.3ms 138.4ms
pk_delay 0us 31.9ms 2.1ms
av_delay 0us 11.0ms 114us
sp_delay 0us 162us 64us
pkts 0 9457 36
bytes 0 6016136 6148
way_inds 0 0 0
way_miss 0 125 6
way_cols 0 0 0
drops 0 630 0
marks 0 0 0
sp_flows 0 0 0
bk_flows 0 0 1
un_flows 0 0 0
max_len 0 1696 212

qdisc ingress ffff: dev pppoe-wan parent ffff:fff1 ----------------
Sent 13010099 bytes 11993 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8011: dev ifb4pppoe-wan root refcnt 2 bandwidth 3600Kbit besteffort dual-dsthost nat wash rtt 100.0ms raw total_overhead 14 hard_header_len 14
Sent 14311272 bytes 11579 pkt (dropped 414, overlimits 16149 requeues 0)
backlog 0b 0p requeues 0
memory used: 35712b of 4Mb
capacity estimate: 3600Kbit
Tin 0
thresh 3600Kbit
target 5.0ms
interval 100.0ms
pk_delay 1.5ms
av_delay 130us
sp_delay 16us
pkts 11993
bytes 15012780
way_inds 0
way_miss 134
way_cols 0
drops 414
marks 0
sp_flows 1
bk_flows 1
un_flows 0
max_n 1696

piece_of_cake will treat all flows (of a gicen internal IP-address if configured properly with the dual-xxxhost isolation options) equally, while layer_cake will (by default) use three different priority tiers, so if discord happens to use the right dscps it might automagically work better with layer_cake. That should be a) easy to convince yourself of using packet captures and b) also be noticeable in less choppy discord during torrenting.

That is a choice you need to make, I would recommend to use the one that works best for your typical use cases...

Yes, the biggest difference between piece_of_cake and layer_cake is that the former requests besteffort (flow fairness) while the latter by default requests the diffserv3 dscp to priority mapping scheme.

Just test both during typical usage and select the better one.

Almost, for the downlink you need to tell sqm-scipts that it should evaluate the DSPCs in the incoming packets (if you tell it to ignore those it will configure the ingress cake to besteffort to save a few CPU cycles, it is much easier to tell cake to not honor the dscps than to actually remap the dscps before cake gets hold of them, but I digress).

Nah, set sqm "Ignore DSCP on ingress:" to "Allow" (yes that is dubious wording right there) and you will have a diffserve3 three tier shaper on ingress as well. The bigger question to answer first though is IMHO, does your ISP actually send you sane DSCPs? That should be easy to confirm with a packet capture on your wan interface while using both torrents and discord.

Please note that I a not saying it does not work directly under windows (I really do not know) I was just telling you the work-around I used.

I guess any additional traffic will introduce some noise, so I try to measure over night, but unless you satureate your link constantly you should be able to overcome this by simply increasing the number of samples per ICMP size.

Oops, try to select a few servers close by? The idle RTT is already terrible and that hides the latency under load increase a bit, that said the bufferbloat plots look sane. I am puzzled a bit about the low upload, with 1050 I would expect around: 1050 * 48/53 * (1500-8-20-20) / (1500-8+40) = 901.28 Kbps goodput. Now both tests report that they where capped to a single upload stream which might explain that a bit...

The t statistics look okay.

First, Sorry for being inactive for the past 3 days... Was a bit busy. (Thank you for helping me out so far ^_^)

Well, Now I've got a solid understanding between layer_cake and piece_of_cake and DSCP's. My next three questions are going to be...

a) easy to convince yourself of using packet captures

That should be easy to confirm with a packet capture on your wan interface while using both torrents and discord.

How can I actually end up checking this?

it is much easier to tell cake to not honor the dscps than to actually remap the dscps before cake gets hold of them, but I digress

A bit more explanation on this? I didn't quite get what you're trying to say here.

The bigger question to answer first though is IMHO, does your ISP actually send you sane DSCPs?

What if my ISP doesn't let pass these DSCP's on to my local network. What happens then If I use layer_cake? What If I only have just discord that uses DSCP's? What about the impact on other applications?

and lastly,

Oops, try to select a few servers close by? The idle RTT is already terrible and that hides the latency under load increase a bit, that said the bufferbloat plots look sane. I am puzzled a bit about the low upload, with 1050 I would expect around: 1050 * 48/53 * (1500-8-20-20) / (1500-8+40) = 901.28 Kbps goodput. Now both tests report that they were capped to a single upload stream which might explain that a bit…

well, I live in Pakistan and I can't really get lower pings to EU servers. The other server locations being any Asian servers but, The routine is just terrible from my ISP to these servers and the site just ends up selecting a mix of NA/EU/Asian servers. I checked the upload/download speed on a local speed test server from my ISP in the next city over and some other ISP's in the next city over and the speeds looked fine. Just an impact from the terrible routing I guess... or I may need to increase the upstream streams but that shouldn't be likely... Since it is upload.
Unless... My ISP limits a single connection to the servers and multiple streams could help me test out my whole bandwidth.

Some good news though, I've seen some Fibre cables being put around the area... Since my ISP is the only one in the city It is most likely there (A friend told me) So hopefully, I can cross my fingers get VDSL/Fibre and then all these issues will go away. (Ping will also lower by 20-30ms so that should be a plus)

Lastly, I've gotten the virtual machine set up and will try to do some flent tests tomorrow. From what I can gather it requires me setting up a server also? I may be wrong but if that is so then my ISP's shitty routing to some servers (not most game servers that I play on) shouldn't affect the tests then.

Well, you just create packet captures, say from your router:
tcpdump -s 1600 -i pppoe-wan -w pppoe-wan.cap
just make sure that there is enough storage space ob your router (you might need to use a usb flash drive and execute the commend from there; you also might need to install tcpdump first "opkg update ; opkg install tcpdump"). Then copy pppoe-wan.cap to your desktop computer and open it with wireshark (you might want to start with actually using wireshark to do the packet capture on the windows machine first, less hassle). Then look into the individual packets and llok for TOS/DSCP.
Obviously you should disable the DSCP suashing option in the GUI:
"Squash DSCP on inbound packets (ingress):" "DO NOT SQUASH"
otherwise all dscps should be exactly 0.

That should get you started...

On ingress interfaces we traditionally do not have access to the user-friendly iptables commends, but only the less forgiving tc, so we would need to find a way in tc to remap CSPs to zero before cake sees the packet. Currently we simply do not do this but rather use iptables for the remapping, but only after sqm actually used the DSCPs to select one of the priority tiers. So the way to ignore the DSCPs on ingress simply is to select a single-tier qdisc.

In that case all will be 0 (which is the default) and layer_cake will behave exactly as piece_of_cake (albeit potentially with a slightly higher CPU cost). But note on egress you should have full control over the dscps so layer_cake will behave different for egress. Now, if you select ignore DSPS for ingress, then you will not pay the additional CPU cost on ingress.

Nothing, layer_cake will pretty much behave like piece_of_cake in that case.

Well, if it uses those DSCPs that cake sorts into the higher priority tiers, this should isolate discord from your concurrent torrenting.

Well, other applications that still use the default DCSP of zero will see a bit more atency under load, if there is concurrent discord traffic. Under saturating conditions all you can do is shift the "pain" around.

Ah that pesky speed-of-light limit strikes again ;).

I did not want to finger you ISP here at all, they might or might not do weird things; it is just that a single TCP-flow typically will not be able to use 100% of a link (partly due to TCPs probe and back-off mechanisms, this is one of the reasons why most speedtests use multiple flows.)

Something to look forward to then.

Yes, flent will measure the full path between the end host and the servers, bad transit/peering will certainly also affect the flent results.

I might test this out a bit later, Any other way I can do packet captures other than the router? Since I already had an issue fitting SQM and UPnP on there with the firmware being stable.

Since my cpu usage isn't really a problem for my low-speed connection I'll just use layer cake with priority tiers on both upload and download. Since the only real downside was the higher cpu usage and that isn't really an issue for me.

As said above, It will help for those applications that use dscp and just in case my ISP does pass on the dscp tags otherwise It will just work like piece_of_cake on the download side.

Lastly, The flent tests seem to be okay... I might make some more tests in a few days and report back. For now, it seems this is the best I can get.

Sure, you could actually install wireshark on your windows computer, it allows AFAIK to create a capture on the local machine, you just need to specify the correct interface (but please keep in mind if cake is set to squash the dscps all you will see is DSCP0).

Well there is the chance that the DSCPs your ISP sends do not make any sense for you, in that case running layer_cake on ingress might actually be worse than piece_of_cake. But this should be obvious from the cake statistics to some degree.

Sure take your time, if the internet works well enough, no need to keep fixing non-issues :wink:

Best Regards

I'll be sure to disable DSCP squashing for cake.

hmm, I can definitely see this as some sort of an issue... I'll just install Wireshark and see If I'm getting proper DSCP's passed on from my ISP.

Well, After this whole post I seem to have more upload speed and switching to layer_cake might be a tiny bit better for my slowness problem on the same computer. I'll see about that with Wireshark.

But after all, was set up, When playing CSGO with my brother being in the same lobby, Whenever the game tried to ping the game servers to find out the optimal server to put us in, It gave us higher pings to all the servers resulting in us not finding a match. I constantly did a background ping -t command to 2 of the servers we play and the pings was fine. He leaves the lobby however and everyone is fine and when the game does initial pings to all the servers in the lobby pings are also fine. layer_cake seemed to have no effect on this.

And lastly, The occasional ping spikes still happen and after this long post, I take it that It's just my ISP at fault here for bad downstream management from there side and the fact that I have really slow internet. (I figured this was the case but I initially made this post thinking that maybe there was something I could try out that I was not aware of.)

But alas, Thank you for the all the help :slight_smile: and I'll be using layer_cake instead of piece_of_cake for a while and If anything comes up I'll be sure to post back here.

Interssting, you might want to create a packetcapture while both of you hang out in the lobby that might reveal something (or at least do a tc -s qdisc before and after hanging in the lobby).

Since I am not much of a gamer, how does that work, do you need to be in the lobby at the same time to ender the same game on the same server, or can you d this sequentially, first your brother and then you? (While not a solution, it might be an acceptable work-around).

This is odd, and certainly makes a packet capture advisable; the symptoms as described might point to the "ping in lobby" process to be quite data intensive (maybe an attempt to not do a low bandwidth ping, but maybe a worst case traffic test to make sure that the server is not only reachable but will actually allow satisfactory game play. But without any data this is pure speculation on my part...

So it might be interesting to see whether these ping spikes also correlate with increases n error counters in the modem (if your modem is broadcom based http://dslstats.me.uk might be a method to continuously sample the modem stats to allow easier correlation with observed ping spikes).
Also you might want to install mtr (http://winmtr.net) to continously measure the RTT (aka ping) along the network path to your reference servers. Let this run for some time and post the results here.
In essence the idea is to find the hop that introduces packet-loss or RTT-increases as that will indicate a bottleneck. BUT please note that interpreting the results is not as straight forward as one would like; quite a number of hops along a network path mainly do "packet-pushing" (so are optimized for routing/switching and will take considerably longer to respond to packets terminating at the hop (or might even drop some of those), to keep that network elements CPU free to do its main work. So if from a given hop all RTTs increase (a lot) and/or from a given hop all further hops show large packet loss I would interpret this as an idicator that there is a network bottleneck at that hop. (Individual hops showing large packet loss or RTTs are less diagnostic, and often one sees with MTR that intermediate hosts have higher packet loss as well as higher max and mean RTTs than the endpoint, which really just indicates that that hop uses priretization/rate-limiting in processing mtr's probe packets, but I digress). But even that interpretation is a bit simplistic given all the "games" ISPs might play with their internal/external routing.

Best Regards

So, I've set up Wireshark and most of the dscp values seem to be CS0 while there are AF11, AF12 values present as well and discord seems to be using AF31 (Since the destination IP with the dscp tag is my local PC's IP I assume that is on the download side.)

I'll do that once we both get into a lobby again later, And no. We play what is called "compettive mode" where it puts us in a match based on our ranks in the game. We need to be in the same lobby to be in the same match. One can't simply join after the other one afterwards. Also do note that this only happens in the lobby while it pings it's game servers to decide which server it should put us in, If we manage to get ingame the pings are fine.

I'll set up the dslstats app but, I doubt it's something wrong in the hop since playing on local game servers in the country (provided by my isp) it's the same. I do still get ping spikes occasionally and whenever I check by playing an HD youtube video in the background and playing the game the pings seem to be fine but at peak times when my whole family is using the internet these ping spikes are worse. Though they still do occasionally happen in my own testing as you've seen with the flent tests (that initial spike) but imagine 4-5 people using the internet at a time, loading stream and opening web pages e.t.c.

in That case you might want to try diffserv4 instead of diffserv3 (edit /usr/lib/sqm/defaults.sh) as that will treat AF3X with higher priorety than the default.

Thanks for the description, it is clear that I have no clue about modern multiplayer FPS (I stopped caring for FPS after the original doom and quake).

But if you get the same spikes when going to local servers as with further away servers your uplink to the ISP looks like a decent candidate to cause problems, because after all packets leaving your home have to cross that link.

This indicates that at peak your bandwidth might be insufficient for gaming even if you get your "fair" share... At a certain point all sqm can do is shift the "pain" caused by too little bandwidth around :wink: