Enabling UDP GSO (generic segmentation offload) on PPPoE interface, not working, but works on physical ethernet interface

vincejv · July 5, 2024, 4:23am

I can't seem to enable generic-segmentation-offload on my pppoe interface in order to enable quic_gso on my nginx config for faster quic performance...

I tried to enable it through ethtool using the following command, but the status remains off

$ ethtool -K pppoe-wan gso on
Actual changes:
tx-generic-segmentation: off [requested on]
Could not change any device features

Without gso on nginx errors out with
2024/06/27 01:24:43 [alert] 7#7: *584 sendmsg() failed (5: I/O error) while sending frames, client: x.x.x.x, server: 0.0.0.0:443 based on searching online I/O error 5 indicates GSO is not supported on the interface https://github.com/envoyproxy/envoy/issues/21200

When nginx listens to a physical ethernet port with GSO on, i'm not encountering any errors and GSO is successfully turned on as checking with ethtool, any kernel patches i can apply with this?

brada4 · July 5, 2024, 5:07am

Random lines from nginx documentation:

The module is experimental, caveat emptor applies.
Default: quic_gso off;

vincejv · July 5, 2024, 6:04am

Envoy proxy GSO also errors out (due to interface refuses to turn GSO on) even if HTTP/3 is no longer experimental on their end, like this error reported here

github.com/envoyproxy/envoy

HTTP/3 serving fails with EIO on Linux

opened 12:35PM - 09 May 22 UTC

closed 02:26PM - 24 May 22 UTC

kanongil

bug area/quic

*Description*: The supposedly stable HTTP/3 server feature fails to deliver U…DP packets on one of my cloud servers. *Analysis*: The linux build enables [the GSO batch writer](https://github.com/envoyproxy/envoy/blob/83af7c0b864c69f87e8d3c6969675423a379af4c/source/common/quic/udp_gso_batch_writer.h#L3-L6) to optimize delivery of UDP packets. However, the [`UDP_SEGMENT` feature](https://github.com/torvalds/linux/commit/bec1f6f697362c5bc635dacd7ac8499d0a10a4e7) that this relies on, requires that [`NETIF_F_HW_CSUM` offloading](https://github.com/torvalds/linux/blob/c5eb0a61238dd6faf37f58c9ce61c9980aaffd7a/include/linux/skbuff.h#L58-L62) is enabled. If not enabled (either from an explicit disable or through missing hardware support), the kernel UDP logic returns [an `EIO` error](https://github.com/torvalds/linux/blob/c5eb0a61238dd6faf37f58c9ce61c9980aaffd7a/net/ipv4/udp.c#L931) for any write larger than the interface `MTU`, and envoy *gives up delivering the packet!* Note that this is a critical issue on one of my cloud servers, which is served by the `virtio_net` driver *without the [`VIRTIO_NET_F_CSUM` feature](https://github.com/torvalds/linux/blob/a7391ad3572431a354c927cf8896e86e50d7d0bf/drivers/net/virtio_net.c#L3493)*. I have no way to enable it, and my current workaround is a custom compile of `envoy` where I force `UDP_GSO_BATCH_WRITER_COMPILETIME_SUPPORT = 0`. *Repro steps*: 1. Using `envoy` binary from published v1.22.0 Linux x86_64 docker image with example HTTP/3 server from [here](https://github.com/envoyproxy/envoy/blob/v1.22.0/configs/envoyproxy_io_proxy_http3_downstream.yaml). 2. Ensure MTU is <= 1500 and transmit `tx-checksumming` is `off` on the primary network interface: `ethtool -K <if> tx off`. 3. Perform any HTTP/3 request from an external host. 4. Observe that no usable packets are returned from the server, and an error is printed to the server console. *Config*: https://github.com/envoyproxy/envoy/blob/v1.22.0/configs/envoyproxy_io_proxy_http3_downstream.yaml *Logs*: ``` [2022-05-05 17:19:10.518][1411][error][quic] [external/com_github_google_quiche/quiche/quic/core/quic_connection.cc:3874] Server: Write failed with error: 5 (Input/output error) ``` `strace` ``` recvmmsg(36, [{msg_hdr={msg_name={sa_family=AF_INET, sin_port=htons(36346), sin_addr=inet_addr("<snip>")}, msg_namelen=128->16, msg_iov=[{iov_base="\302\0\0\0\1\20\231v\35\272\230:&\260\232\34\354B\312\267\266\320\24\34eNH\270\22\301\330\3"..., iov_len=1500}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("eth0"), ipi_spec_dst=inet_addr("<snip>"), ipi_addr=inet_addr("<snip>")}}], msg_controllen=32, msg_flags=0}, msg_len=1200}], 16, MSG_TRUNC|MSG_WAITFORONE, NULL) = 1 sendmsg(36, {msg_name={sa_family=AF_INET, sin_port=htons(36346), sin_addr=inet_addr("<snip>")}, msg_namelen=16, msg_iov=[{iov_base="\316\0\0\0\1\24\34eNH\270\22\301\330\3\363\311\fi\236W\242\263\240G\307\10\231v\35\272\202"..., iov_len=3600}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=0, ipi_spec_dst=inet_addr("<snip>"), ipi_addr=inet_addr("0.0.0.0")}}, {cmsg_len=18, cmsg_level=SOL_UDP, cmsg_type=0x67}], msg_controllen=56, msg_flags=0}, 0) = -1 EIO (Input/output error) ```

brada4 · July 5, 2024, 6:07am

There is complex dependency tree in ethtool kK
Please post

ubus call system board
ethtool -k pppoe-wan | grep -v "\[fixed\]$"
ethtool -i pppoe-wan 
ethtool -k wan | grep -v "\[fixed\]$" 
...
(all lowlevel ifaces backing pppoe interface)

The only thing you can do - blindly cram all changeable options "on" until desired option changes to on from requested. if fixed options need to be changed they cannot.

vincejv · August 17, 2024, 5:21am

this patch fixed the issue https://github.com/openwrt/openwrt/commit/79fce424046fed521026df7c7f838441fb46add8

altough ethool gso flag still shows off, it's no longer throwing an EIO error when nginx tries to send quic/http3 packets quic_gso flag in nginx.conf switched to on

but the performance is very bad though when turning it on, as noted in the patch as system tries to calculate for the checksum through software...