Hi all, i'm experimenting cake with layer cake and I have problem when loading youtube videos, some videos even won't load at all. piece of cake run smooth. Here's my config (my CPU is MT 7620):
tc -s qdisc
after a fresh reboot of the router , but after sqm with layer cake initiated and
tc -s qdisc
after testing with critical youtube videos.
Also try with setting download speed to 0 to disable ingress shaping for testing, with some devices sqm currently has issues with ingress shaping, your might be one of those...
So you are seeing giant packets on your egress, which can be a problem (cake tries to de-segment giants into normally sized packets automatically, so that might not be a problem, also your video issue most likely id ingress related, not egress). The kernel will keep any giant it receives intact so if you want to test without giant packets in the system you need to first make sure ethtool is installed:
opkg update ; opkg install ethtool
and then run the following for all ethN (eth0, eth1, ...) in your system:
ethtool -K ethN tso off gso off gro off
(the most relevant for the other interfaces is going to be "gro off" as this is the place where incoming MTU sized packets can be assembled into a giant in the first place).
Also it would be interesting to learn what kind of connection you actually have, from the uplink it looks like fiber, docsis-cable or potentially VDSL(2), the only one we can rule out is ADSL([1|2|2+] as your upload seems to be too high...
That said, if everything works well with piece_of_cake, but not layer_cake this would actually implicate your upstream more than your downstream, so I am a bit confused...
Also please post the results of a dslreports speedtest: https://www.dslreports.com/speedtest. After a free regitration you will be able to configure the testing parameters by clicking on the cog button. I would recommend the following settings:
No. download streams: 16
No. upload streams: 16
actually the number itself is not so important, but the different testing profiles this speedtest offers use different numbers of test streams making simple comarison difficult...
Hi-Res BufferBloat: (check this box)
this will give higher resolution latency probes during the speedtest, which is quite helpful in underatanding shaping issues in sqm-scripts
Upload duration: 30
Download duration: 30
longer tests are typically more reliable, 30 seconds is well above the typical 10seconds ISPs seem to optimise for, and I seem to recall 30s is the maximum dslreports happily offers (thet pay for bandwidth after all)
dodge compression: (check this box)
It would be most excellent if you could either post links to your test results or at least past in the test number ($TESTNUMBER) (that will allow others to reach your test results via https with the following address www.dslreports.com/speedtest/$TESTNUMBER
That looks relatively nice except for a few nasty latency spikes in both up- and download direction. If these are not artifacts of the speedtest's latency probes these might be related to your issue. BTW the tc -s qdisc lines above indicate that your luci-app-sqm and sqm-scripts might be older than required (newer sqm-scripts default to using cake's in-built overhead accounting method, but that really only has advantages for ATM networks...).
Now could you do the same test with piece_of_cake instead of layer_cake (or even better runn this test with both qos-scripts versions while also recreating your youtube problem in te background, it would be best to use a different computer for the speedtest than for youtube...)
Ah, i checked script version and it's really old (from June 2016). I've reinstalled new version and everything run fine until now. Thanks for remind me the version Here's my recent tc -s qdisc output:
You still seem to have GRO enabled on one of the interfaces? The 17472 should be much closer to 1500 otherwise, at 20Mbps that takes (17472*8) / (20 * 1000^2) = 0.0069888 or roughly 7ms, which actually might not be a big problem, but you will exercise cake's giant disassembly code paths which might not be as well tested as the boring MTU 1500 ones.... But as long everything just works with the newer version, I will just let it rest. (That said your old version, AFAICT does not have known bugs causing your observed behavior, so I fear that your problem might crop up again as the root cause still is unknown).
Ok, the problem still persist, it just take more times to stall. i'm using piece of cake for now until layer cake stable enough to use. Thanks for your quick support
The really weird thing is that piece_of_cake and layer_cake use the same besteffort diffserv-profile for ingress so there should be no real difference in shaping of the incoming video bursts... The big difference between the two is how they handle outgoing/egress traffic...
Hi again. Now my SQM script run pretty good but I have a question about eth offload. I created a file to turn off all offload on all interface at boot. I double check by ethtool to make sure it work but max lenght still very high. Is it normal ?
#!/bin/sh /etc/rc.common
START=99
start() {
local count=1
local eth
while true ; do
eth=`ls /sys/class/net | grep 'eth' | sed -n "$count"p`
if [ "$eth" == "" ]; then
break
fi
ethtool -K $eth tso off gso off gro off
if [ $? -eq 0 ]; then
logger -s -t disable_offload_init -p daemon.notice "$eth offload turned off"
else
logger -s -t disable_offload_init -p daemon.err "[ERR] $eth offload turn off FAILED: '$?'"
fi
count=$((count+1))
done
}
And tso, gso and gro turned off:
root@17CTuXuong:~# ethtool -k eth0
Features for eth0:
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: on
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
tx-tcp-segmentation: off
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
This is on egress only. The way to stop this is to disable GRO on all of the router's interfaces (wireed and wireless) as the kernel will keep an already assembled giant packet intact. That said, cake actually attempts to dissect the giants into reasonable sized "chunks" so that latency under load should still be fine. My recommendation would be to leave things as they are unless you notice unwanted behavior under load... (Really GRO and GSO are not bad things in themselves, they help the kernel to better scale to high packet rates, so disabling them without the need to do so will make your system less efficient).