Mwan3 doesn't work with hardware offloading

Hi Everyone. I’m trying to setup mwan3 multi-wan, and I realized https traffic fails intermittently (timed out because pkt dropped)

If I turn off hardware offloading and only use software offloading, problem solved like magic.

Is it a known issue? Should I update mwan3 wiki to recommend turning off hardware offloading?

test log
MPC➜  linuxconf git:(master) curl https://1.1.1.1/ -v
*   Trying 1.1.1.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O="Cloudflare, Inc."; CN=cloudflare-dns.com
*  start date: Jan  2 00:00:00 2025 GMT
*  expire date: Jan 21 23:59:59 2026 GMT
*  subjectAltName: host "1.1.1.1" matched cert's IP address!
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* Established connection to 1.1.1.1 (1.1.1.1 port 443) from 10.100.100.220 port 48598 
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://1.1.1.1/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 1.1.1.1]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.16.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: 1.1.1.1
> User-Agent: curl/8.16.0
> Accept: */*
> 
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 301 
< date: Sat, 04 Oct 2025 05:27:11 GMT
< content-length: 0
< location: https://one.one.one.one/
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=Ei%2FjnSJ%2FAjWsb3On402HE0YSo3aACFB9iKb4PI6YWpy6JfxpwYPkSXDn8PHnPOH3wEM%2BtH4fb7QZNqQPldsMVX0KMBBzhpnmaeP%2B%2F5Q3huwO%2B4gBoq2%2Fk3I%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 989244c6594fa385-SEA
< 
* Connection #0 to host 1.1.1.1:443 left intact
MPC➜  linuxconf git:(master) curl https://1.1.1.1/ -v
*   Trying 1.1.1.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
^C⏎                                                                                                                                                                                                                                    MPC➜  linuxconf git:(master) curl https://1.1.1.1/ -v
*   Trying 1.1.1.1:443...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
[hangs for minutes...]

Router info: Netgear R6850, MT7621, OpenWrt 23.05.0 r23497-6637af95aa

software offloading neither on v23….

First update to v24

Read here about limitations, probably add a line to… https://openwrt.org/docs/guide-user/perf_and_log/flow_offloading

in short - all offloading transports packets from interface A to interface B adjusting headers, checking routing table and arp table only every sysctlable 60 seconds. Software offload does forwarding between br-lan and wan, thus at least home side you have arp and routes and fdb working right.

2 Likes