Choosing the CAKE overhead and MPU for PPoE & GPON

[dropping_state]

My FTTH Internet connection uses PPPoE on a GPON network.

The overhead might be computed like this:

• the ethernet overhead is 38 bytes
• the GEM header is only 5 bytes (PLI, Port-ID, PTI, CRC)
• 21 bytes of the ethernet header (inter packet gap, SFD, EOF) are replaced with the 5 bytes GEM header
• the PPPoE header is 8 bytes

This gives us an overhead of 31 bytes if we don't remove the EOF from the computation. Otherwise we get an overhead of 30 bytes.

Since the minimum IPv4 packet is 68 bytes, an MPU of 90 or 91 bytes appears to be right.

Based on this, the settings for cake would be
overhead 30 mpu 98 noatm
or
overhead 31 mpu 99 noatm

Can someone confirm this is correct, please?

[moeller0]

@dropping_state

• 21 bytes of the ethernet header (inter packet gap, SFD, EOF) are replaced with the 5 bytes GEM header

Make this 20 Bytes:
Preamble: 7 octets, Start of Frame delimiter (SFD) 1 octet, Inter packet/frame gap (IFG) 12 bytes:
7 + 1 + 12 = 20
The rest of the ethernet over head would be:
destination MAC 6 octets, source MAC 6 octets, optional VLAN tap (4 octets), ether type 2 octets, frame check sequence (FCS) 4 octets:
6 + 6 + (4) + 2 + 4 = 18 or 22 with single VLAN tag.

Relevant later an ethernet L2-frame from dstMAC to FCS is at least 64 octets.

Not sure what EOF is in the context of ethernet frames. But yes GPON maintains the ethernet L2-frame but replaces the L1 overhead with GEM+. But it is not clear what the + actually comes out to. This really depends on two issues:
a) what fraction of cost of GPONs request-grant traffic and time-slot framing is accountable to each packet (probably variable)
b) more important which additional overhead does the ISP's traffic shaper assume.

So in short, even after browsing through ITU GPON standards documents I can't reliably tell you what the effective per-packet overhead on GPN is any better than your calculations above that from 38-42 bytes of effective ethernet L1/L2-overhead we go down to 23-27 bytes for GPON/GEM.

@dropping_state

This gives us an overhead of 31 bytes if we don't remove the EOF from the computation. Otherwise we get an overhead of 30 bytes.

I assume you are looking at adding the 8 byte PPPoE overhead, where we end up with 31-35 bytes.
What do you mean by removing the EOF?

@dropping_state

Since the minimum IPv4 packet is 68 bytes, an MPU of 90 or 91 bytes appears to be right.

A minimum IPv4 packet results in an ethernet L2-frame of at least 64 bytes (see above).
To calculate the MPU we need to use the minimal L2 payload size (the 64 from above) and add our known L1 overhead, which for GPON is just the 5 byte GEM for an MPU of 69. The PPPoE overhead is part of the ethernet payload and hence does not add on top and does not affect the MPU. (It does however affect the minimal payload size, but that is fine, if the ethernet L2 frame does not contain any padding the MPU correction should not come into effect, if both MU\PU and per packet overhead have been configured correctly).

@dropping_state

Based on this, the settings for cake would be
overhead 30 mpu 98 noatm
or
overhead 31 mpu 99 noatm

I think the numbers would be:
overhead 31 mpu 69
or
overhead 35 mpu 74 (with VLAN)

BUT the problem is that it is unclear what effective per-packet-overhead is relevant for us. A competent ISP will configure its own shaper such that shaper overhead >= true overhead, so the 31-35 should be a lower bound.

However, it turns out getting the overhead exactly correct, while a fun game of detective, is not all that important compared to not set the overhead too small. So cake_overhead >= true_overhead is an acceptable goal as the cost of gently over estimating the per-packet-overhead is a small decrease in throughput notably mostly if the link is saturated with small packets (which is an uncommon load for most home networks). Given the uncertainties in correctly accounting for all GPON overhead, I would still stick to 42-44 bytes for per-packet-overhead.

@dropping_state

Can someone confirm this is correct, please?

Sorry, the above is the best I can offer. If you manage to get hold of your ISP's technicians they might be able to tell you more; should you find out more details, please post these here.

[dropping_state]

Thanks for the detailed and helpful answer.

Cheers!

[dropping_state]

The EOF is mentioned here in this image:

That's what made me think this is also part of the ethernet overhead.

[moeller0]

See e.g. page 2-5 of https://docs.broadcom.com/doc/12353256
Showing that EOF might be a concept of an ethernet NIC, but not a part of the actual data frame. This is different for Fiber Channel which includes an EOF field in the frame (but fiber channel over ethernet packs this FC EOF into the ethernet payload).

Not saying I am an expert in ethernet framing, but so far I have seen no indication that there exists an EOF field in an ethernet frame.

[dropping_state]

I was confused as well. Other diagrams didn't show the EOF for the ethernet frame either. This was the diagram which explained how the header of a GEM frame relates to the header of an ethernet frame.

[moeller0]

Yes, I found it in ITU G.984.3, but I am not sure that is actually correct, it is not that the ITU is staffed with ethernet experts (that would be the IEEE).

The bigger problem with all of this is that is it hard to generally measure the effective per packet overhead without additional information. The SQM details page in the OpenWrt wiki contains a small discussion on the interdependence of shaper-rate and per-packet-overhead, which shows some of the problems of trying empirically to deduce both of these. It gets considerably simpler if you know one of them for good, but typically we know both only approximately. The proposed test using smaller packet sizes while in theory elegant quickly runs into the issue that few home routers are prepared to the factor 10 increase in packet processing rate involved in those measurements, and the problem of getting high precision throughput tests done.