root@LEDE:/etc# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
qdisc noqueue 0: dev br-lan root refcnt 2
qdisc noqueue 0: dev eth0.1 root refcnt 2
## qdisc cake 8040: dev eth0.2 root refcnt 2 bandwidth 4193Kbit besteffort triple-isolate nat rtt 100.0ms raw mpu 64
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
qdisc cake 8041: dev ifb4eth0.2 root refcnt 2 bandwidth 62900Kbit besteffort triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn
Sent 1125532334 bytes 1192697 pkt (dropped 0, overlimits 0 requeues 204)
backlog 0b 0p requeues 204
maxpacket 1514 drop_overlimit 0 new_flow_count 75836 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
## qdisc cake 8040: dev eth0.2 root refcnt 2 bandwidth 4193Kbit besteffort triple-isolate nat rtt 100.0ms raw mpu 64
Sent 78773 bytes 323 pkt (dropped 0, overlimits 83 requeues 0)
backlog 0b 0p requeues 0
memory used: 12Kb of 4Mb
capacity estimate: 4193Kbit
Tin 0
thresh 4193Kbit
target 5.0ms
interval 100.0ms
pk_delay 8.1ms
av_delay 715us
sp_delay 51us
pkts 323
bytes 78773
way_inds 0
way_miss 66
way_cols 0
drops 0
marks 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 1514
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 198017 bytes 431 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8041: dev ifb4eth0.2 root refcnt 2 bandwidth 62900Kbit besteffort triple-isolate nat wash rtt 100.0ms noatm overhead 18 via-ethernet mpu 64
Sent 202537 bytes 430 pkt (dropped 1, overlimits 69 requeues 0)
backlog 0b 0p requeues 0
memory used: 10Kb of 4Mb
capacity estimate: 62900Kbit
Tin 0
thresh 62900Kbit
target 5.0ms
interval 100.0ms
pk_delay 3.4ms
av_delay 163us
sp_delay 2us
pkts 431
bytes 204051
way_inds 0
way_miss 70
way_cols 0
drops 1
marks 0
sp_flows 2
bk_flows 1
un_flows 0
max_len 1514
## Heading
NAME="LEDE"
VERSION="17.01.4, Reboot"
ID="lede"
ID_LIKE="lede openwrt"
PRETTY_NAME="LEDE Reboot 17.01.4"
VERSION_ID="17.01.4"
HOME_URL="http://lede-project.org/"
BUG_URL="http://bugs.lede-project.org/"
SUPPORT_URL="http://forum.openwrt.org/"
BUILD_ID="r3560-79f57e422d"
LEDE_BOARD="bcm53xx/generic"
LEDE_ARCH="arm_cortex-a9"
LEDE_TAINTS="no-all"
LEDE_DEVICE_MANUFACTURER="LEDE"
LEDE_DEVICE_MANUFACTURER_URL="http://lede-project.org/"
LEDE_DEVICE_PRODUCT="Generic"
LEDE_DEVICE_REVISION="v0"
LEDE_RELEASE="LEDE Reboot 17.01.4 r3560-79f57e422d"Is this on the Archer C7 v4 in your other thread?
No, This same thing is happening now on my Netgear R7000
Cake should be undoing the existing 14 bytes and applying the new overhead. In essence:
+14 {linux kernel} 0 +18 { specified overhead either via keyword or llam tab in luci} for both eth0.2 (egress) and ifb4eth0.2 (ingress).
This is truly mysterious. Could you post the output of:
cat /etc/config/sqm
ls -all /usr/lib/sqm
If you can install a recent snapshot that would be great as the newer cake will be a bit more verbose about the accounted overhead... the "raw" keyword in your might mean that for whatever reason the calculated overhead results in 14, the question is why....
Best Regards
I'll did have recent snapshots, as of a few days ago installed, and the results were the same. Here is my current configuration...
config queue
option squash_dscp '1'
option squash_ingress '1'
option ingress_ecn 'ECN'
option tcMTU '2047'
option tcTSIZE '128'
option tcMPU '0'
option interface 'eth0.2'
option download '60000'
option upload '4000'
option debug_logging '1'
option verbosity '10'
option qdisc 'cake'
option script 'piece_of_cake.qos'
option qdisc_advanced '1'
option egress_ecn 'ECN'
option qdisc_really_really_advanced '1'
option iqdisc_opts 'nat dual-dsthost mpu 64'
option eqdisc_opts 'nat dual-srchost mpu 64'
option linklayer 'ethernet'
option overhead '18'
option linklayer_advanced '1'
option linklayer_adaptation_mechanism 'cake'
option enabled '1'It doesn't matter if I delete the instance, reboot, unplug power and then re-apply settings, use the docsis keyword, cut off medusas head. The surprising part is this is
v17.01.4 Stable.root@LEDE:~# ls -all /usr/lib/sqm
drwxr-xr-x 2 root root 1360 Dec 14 23:06 .
drwxr-xr-x 1 root root 1600 Dec 15 02:54 ..
-rw-r--r-- 1 root root 2744 Oct 17 00:47 defaults.sh
-rw-r--r-- 1 root root 22459 Oct 17 00:47 functions.sh
-rw-r--r-- 1 root root 2283 Oct 17 00:47 layer_cake.qos
-rw-r--r-- 1 root root 296 Oct 17 00:47 layer_cake.qos.help
-rw-r--r-- 1 root root 2214 Oct 17 00:47 piece_of_cake.qos
-rw-r--r-- 1 root root 316 Oct 17 00:47 piece_of_cake.qos.help
-rwxr--r-- 1 root root 3443 Oct 17 00:47 run.sh
-rw-r--r-- 1 root root 11347 Oct 17 00:47 simple.qos
-rw-r--r-- 1 root root 83 Oct 17 00:47 simple.qos.help
-rw-r--r-- 1 root root 4822 Oct 17 00:47 simplest.qos
-rw-r--r-- 1 root root 76 Oct 17 00:47 simplest.qos.help
-rw-r--r-- 1 root root 3626 Oct 17 00:47 simplest_tbf.qos
-rw-r--r-- 1 root root 146 Oct 17 00:47 simplest_tbf.qos.help
-rwxr--r-- 1 root root 1606 Oct 17 00:47 start-sqm
-rwxr--r-- 1 root root 1351 Oct 17 00:47 stop-sqm
-rwxr--r-- 1 root root 370 Oct 17 00:47 update-available-qdiscsThis is from just a few minutes ago running a recent snapshot installed for my R7000
Heading
tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 4Mb ecn
Sent 385100657 bytes 410367 pkt (dropped 0, overlimits 0 requeues 49)
backlog 0b 0p requeues 49
maxpacket 1514 drop_overlimit 0 new_flow_count 28082 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev eth0.1 root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8008: dev eth0.2 root refcnt 2 bandwidth 4Mbit besteffort dual-srchost nat rtt 100.0ms raw total_overhead 18 hard_header_len 18 mpu 64
Sent 33102697 bytes 160227 pkt (dropped 944, overlimits 84857 requeues 0)
backlog 0b 0p requeues 0
memory used: 131008b of 4Mb
capacity estimate: 4Mbit
Tin 0
thresh 4Mbit
target 5.0ms
interval 100.0ms
pk_delay 4.6ms
av_delay 2.4ms
sp_delay 3us
pkts 161171
bytes 34530553
way_inds 7
way_miss 944
way_cols 0
drops 944
marks 0
ack_drop 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 6056
qdisc ingress ffff: dev eth0.2 parent ffff:fff1 ----------------
Sent 349885253 bytes 251155 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 8009: dev ifb4eth0.2 root refcnt 2 bandwidth 60Mbit besteffort dual-dsthost nat wash rtt 100.0ms noatm overhead 18 via-ethernet total_overhead 18 hard_header_len 14 mpu 64
Sent 351942214 bytes 250145 pkt (dropped 1010, overlimits 222897 requeues 0)
backlog 0b 0p requeues 0
memory used: 538920b of 4Mb
capacity estimate: 60Mbit
Tin 0
thresh 60Mbit
target 5.0ms
interval 100.0ms
pk_delay 4.7ms
av_delay 214us
sp_delay 3us
pkts 251155
bytes 353470223
way_inds 12
way_miss 999
way_cols 0
drops 1010
marks 0
ack_drop 0
sp_flows 0
bk_flows 1
un_flows 0
max_len 23624
cat /etc/os-release
NAME="OpenWrt"
VERSION="SNAPSHOT"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt SNAPSHOT"
VERSION_ID="snapshot"
HOME_URL="http://lede-project.org/"
BUG_URL="http://bugs.lede-project.org/"
SUPPORT_URL="http://forum.openwrt.org/"
BUILD_ID="r5578-f704b64"
LEDE_BOARD="bcm53xx/generic"
LEDE_ARCH="arm_cortex-a9"
LEDE_TAINTS=""
LEDE_DEVICE_MANUFACTURER="OpenWrt"
LEDE_DEVICE_MANUFACTURER_URL="http://lede-project.org/"
LEDE_DEVICE_PRODUCT="Generic"
LEDE_DEVICE_REVISION="v0"
LEDE_RELEASE="OpenWrt SNAPSHOT r5578-f704b64"
So this looks slightly better, but note invalid difference in hard_header_len? This seems due to a small bug introduced during recent changes that is just about to be fixed. But at least the defied overhead no sticks for both ingress and egress...
I agree. Egress appears to have already recognized 18 bytes before the overhead is applied, while Ingress recognizes 14 bytes before overhead editing. Which IMHO is both a bug and accidentally correct surprisingly. Yet it shouldn’t do that.
At least now with this patch of stats, we can peer into the inner workings more closely. I like it ALOT 
I am curious about the eth0.2 interface adding 4 bytes to create hard_header_len 18, which I speculate comes from vlan. The total should now be 22 bytes to account for 14 ethernet, 4 vlan, 4 remaining for docsis. Should this be done for ifb4eth0.2 as well since that interface only sees 14 bytes, or just stating 18 bytes because it's not a true vlan device only a virtual device inside that vlan. if that makes sense to you.
This appears correct to my understanding regarding overhead since the vlan bytes are needed to be added as well.
tc -s qdisc | grep overhead
qdisc cake 802b: dev eth0.2 root refcnt 2 bandwidth 3400Kbit besteffort dual-srchost nat ack-filter rtt 100.0ms noatm overhead 22 via-ethernet total_overhead 22 hard_header_len 18 mpu 64
qdisc cake 802c: dev ifb4eth0.2 root refcnt 2 bandwidth 51Mbit besteffort dual-dsthost nat wash rtt 100.0ms noatm overhead 18 via-ethernet total_overhead 18 hard_header_len 14 mpu 64
Mmmh,
Unfortunately, you stumbled over an issue with cake's automatic accounting of kernel-added overhead; namely that it only works for pure Ethernet interfaces. Hard_header_len is alas not the amount of overhead the kernel added to the packet length (skb->Len) but rather the space the kernel reserves in the skb to allow the expected overheads to fill. So your observations about the reported hard_header_len seem reasonable, just cake's approach of effectively only adding skb->Len - hard_header_len to account for the requested overhead is misguided. There will be a fix for cake one way or the other*, but I can give no ETA.
Best Regards
*) The fix might be to simply disable the automatic accounting altogether....
Roger that. Sorry for the confusion. These low level concepts can be tricky especially when bugs are involved.