Cake WAN packet overhead for VDSL2 PPPoE PTM with VLAN Tagging

Hi everyone

I'm looking for some confirmation on this. As I'm sure you can imagine for someone new to this stuff it's been hard to understand. Here in the UK a VDSL2 connection via PPPoE and PTM is very popular (BT ISP etc). It also requires a 101 vlan tag.

When looking at the values here: https://man7.org/linux/man-pages/man8/tc-cake.8.html

pppoe-ptm is 30 but I also have to consider that ether-vlan should be applied (101) so thus add 4 bytes to get 34. This I think should be correct but I have some related questions:

1: Is there any penalty when setting the ptm flag for compensation? The manual above says it's better to just do the 64/65 stuff from the shaper bandwidth and not enable it aka noatm command. Is it less cpu intensive to just take care of the ptm compensation in bandwidth allowance (0.984)?

2: The mpu is something going over my head. I found this wiki entry: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#more_hints_tips_info

The author talks about having a Sky connection in the UK. His connection is not PPPoE so it uses the bridged-ptm overhead which I believe is 22. They then talk about setting a 72 mpu.

Now I might be out of the ballpark here but since both have a vlan tag, is it a decent guess to think that my mpu should be ~80? i.e. add 8 since this would be the difference between pppoe ptm and bridged-ptm to start with.

If this is wrong what mpu should I be setting?

Thank you for any assistance.

So PTM encoding uses a 65/64 scheme where there is one byte of overhead for every 64bytes of payload, but that 65th byte is always special even if the "payload" is empty. So the easiest way to account for that overhead is simply to reduce the shaper-rate from the desired gross rate by 100-100*64/65 = 1.54%-age points. If you do this once there is no additional run time cost to account fr PTM. If you set the ptm keyword cake will instead account the size of each packet by multiplying with roughly 65/64 = 1.015625, so there will be one additional calculation per packet (and some more quantization error as the size of all packets needs to be ab integer number of bytes). In all honesty, the additional calculation probably is not going to be noticeable and the imprecision is also going to be small (at most 1 byte per packet, with many packet being >> 100 or even 1000 bytes).
That said, personally, I simply reduce the gross shaper sufficiently, and spare cake the futile calculation, but again that is more a matter of taste...

Technically the concept of MPU is relatively easy (let's look at a toy example to is not 100% veridical but gets hopefully the idea across). The linux kernel does accounting for packets it processes for a number of reasons, but primarily to keep track of its own immediate needs, so an IPv4 packet with 1 byte payload would be accounted for as size 21 (header 20 bytes 1 byte payload), and as far as the kernel is concerned that is the size it needs to prepare buffers for. But if that packet ends on any "ethernet"-ish medium (or rather any medium that uses full L2 ethernet frames) the device driver to even the NIC is going to add the ethernet checksum (called frame check sequence or FCS) at the end, but with the twist that an ethernet L2 frame is at least 64 bytes in size (containing 6B dstMAC, 6B srcMAC, 2B ethertype, 46-1500B payload, 4B FCS), so our nominal 21 byte packet actually consumes 64 byte on the medium (on real ethernet we also need to add 20 more bytes for ethernet's L1 overhead, but let's ignore that for now). And the parameter MPU really just tells cake what is the minimum size of any packet on the relevant medium, so that in our example cake assumes 64 bytes instead of just 21.
If cake just assumes 21 bytes it would end up admitting too much effective data onto the link and fail to achieve its shaper rate (shaper rates are by necessity gross rates).

So here is the point, VDSL2 mostly encapsulates for L2 ethernet frames so inherits the 64 byte ethernet MPU, but now we need to add the VDSL2/PTM overhead:
1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 4 Byte
as well as that ISPs VLAN tag 4 byte
(-> 64 + 4 + 4 = 72) EDIT: this is incorrect

EDIT: as you pointed out elsewhere, the VLAN tag (and even Q-in-Q dual vlan tags) lives inside the 46byte "payload" area the above calculation is 4 bytes too large:
1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS), = 4 Byte
-> 64 + 4 = 68

Yes the full PPPoE overhead is:
PPPoE: 2 Byte PPP + 6 Byte PPPoE = 8 Byte

so your MPU would be 80... (EDIT: 68+8 = 76)

However, the PPPoE header lives within the ethernet payload, complicating things abit.
So you account for these in the total per-packet overhead already:

PPPoE-VDSL2-PTM: 6 (dest MAC) + 6 (src MAC) + 4 (VLAN) + 2 (ethertype) + 2 Byte PPP + 6 Byte PPPoE + 4 Byte Frame Check Sequence (FCS) + 1 Byte Start of Frame (S), 1 Byte End of Frame (Ck), 2 Byte TC-CRC (PTM-FCS)
6 + 6 + 4 + 2 + 2 + 6 + 4 + 1 + 1 + 2 = 34
(EDIT: this is still correct as VLAN tags are still part of the per packet overhead that the kernel does not account for in all cases)

Now let's look at our packet of size 21:
first we add the per-packet overhead to the IP size:
effective_size = 21 + 34 = 55 Bytes
then we compare with the mpu
55 < 72? -> effective_size = 72
so the MPU stays 72, as the PPPoE overhead does not sit outside the L2-ethernet frame.
EDIT:
55 < 68? -> effective_size = 68
so the MPU stays 68, as the PPPoE overhead does not sit outside the L2-ethernet frame.

However things can get trickier, if say you connect to a 116 Mbps VDLS2 link via 100 Mbps FastEthernet, then ethernet's own mpu of 84 bytes becomes relevant...

In reality most packets will be larger than mpu anyhow, so getting this wrong is not going to be all that obvious, but will e.g. show with bidirectionally saturating TCP traffic, as especially TCPs pure ACK packets are likely to require the mpu-correction. Bi-diectionally saturation as you need both loads of packets requiring mpu-correction as well as the shaper actually engaging in that direction...

EDIT: corrections for wrong placement of VLAN tags inside an ethernet frame...

1 Like

Thank you for that. It was exactly what I needed to know.

1 Like