# Software flow offloading implications

**URL:** <https://forum.openwrt.org/t/software-flow-offloading-implications/90957>\
**Category:** Installing and Using OpenWrt\
**Created:** [March 11, 2021, 2:26pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957 "2021-03-11T14:26:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![xorbug](https://forum.openwrt.org/letter_avatar/xorbug/32/5_5575768a8748004e209b776fc1b2916d.png) [@xorbug](https://forum.openwrt.org/u/xorbug)\
**Post date:** [March 11, 2021, 2:26pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/1 "2021-03-11T14:26:02Z")

</div>

Hi all,  
I was wondering, is there any reason to keep _software flow offloading_ disabled (aside not being able to use QoS/SQM)? Are there practical implications of any kind when enabling it?  
Thanks.

---

<div class="post-metadata">

**Author:** ![bobafetthotmail](https://forum.openwrt.org/user_avatar/forum.openwrt.org/bobafetthotmail/32/39_2.png) [@bobafetthotmail](https://forum.openwrt.org/u/bobafetthotmail)\
**Post date:** [March 11, 2021, 5:27pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/2 "2021-03-11T17:27:09Z")

</div>

I think you got it reversed. If you enable software flow offloading then you cannot use QoS and SQM. If it is disabled you can use them.

That's because "software flow offloading" means that the traffic is bypassing some of the advanced firewall features used by QoS and SQM to prioritize traffic.

It's meant to provide an "alternative" to the "hardware accelerated NAT" features offered in stock firmware that can't usually be done in OpenWrt. It's not as good as HW NAT, but it's at least halfway there

---

<div class="post-metadata">

**Author:** ![xorbug](https://forum.openwrt.org/letter_avatar/xorbug/32/5_5575768a8748004e209b776fc1b2916d.png) [@xorbug](https://forum.openwrt.org/u/xorbug)\
**Post date:** [March 11, 2021, 5:58pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/3 "2021-03-11T17:58:49Z")

</div>

> [@bobafetthotmail](#):
>
> I think you got it reversed. If you enable software flow offloading then you cannot use QoS and SQM. If it is disabled you can use them.

Yeh i see... so allow me to rephrase: any other reason to keep _software flow offloading_ disabled when not using QoS/SQM at all?  
And incidentally, any reason why it is disabled by default although there is no QoS/SQM by default (afaict)? Is it because it is still an experimental feature?

---

<div class="post-metadata">

**Author:** ![bobafetthotmail](https://forum.openwrt.org/user_avatar/forum.openwrt.org/bobafetthotmail/32/39_2.png) [@bobafetthotmail](https://forum.openwrt.org/u/bobafetthotmail)\
**Post date:** [March 11, 2021, 6:12pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/4 "2021-03-11T18:12:23Z")

</div>

Because if it's off by default, someone enables it and their configuration breaks they can figure out pretty quickly that it's not good for them.

While if you enable it by default you will get flooded by people complaining and opening bug reports about SQM/QoS not working or this or that other feature not working and you can't be sure it's software offloading issue or something else.

---

<div class="post-metadata">

**Author:** ![xorbug](https://forum.openwrt.org/letter_avatar/xorbug/32/5_5575768a8748004e209b776fc1b2916d.png) [@xorbug](https://forum.openwrt.org/u/xorbug)\
**Post date:** [March 12, 2021, 10:38pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/5 "2021-03-12T22:38:50Z")

</div>

Seems reasonable.  
Thank you.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dlakelan/32/2320_2.png) [@dlakelan](https://forum.openwrt.org/u/dlakelan)\
**Post date:** [March 12, 2021, 11:04pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/6 "2021-03-12T23:04:50Z")

</div>

@bobafetthotmail and @xorbug as far as I understand **software** offloading still works with SQM just fine. Only hardware offloading is incompatible.

---

<div class="post-metadata">

**Author:** ![dl12345](https://forum.openwrt.org/letter_avatar/dl12345/32/5_5575768a8748004e209b776fc1b2916d.png) [@dl12345](https://forum.openwrt.org/u/dl12345)\
**Post date:** [March 13, 2021, 12:23am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/7 "2021-03-13T00:23:11Z")

</div>

I'm not sure that it does. I was busy running some flent tests when I read your reply, so I enabled SFO for the one test and disabled it for the other. The first test shows SFO enabled and the second disabled.

 ![image](https://forum.openwrt.org/uploads/default/original/3X/2/4/24a7b0c8fe073d19b0fbbd9f32ba767a6e1cd8f6.png)

 ![image](https://forum.openwrt.org/uploads/default/original/3X/3/4/3475978d8631a8179a8543efebd450c4a7fd0ab0.jpeg)

---

<div class="post-metadata">

**Author:** ![dlakelan](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dlakelan/32/2320_2.png) [@dlakelan](https://forum.openwrt.org/u/dlakelan)\
**Post date:** [March 13, 2021, 12:28am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/8 "2021-03-13T00:28:50Z")

</div>

This was with SQM running I assume? Both have average bandwidth within epsilon of each other, and avg latency under load within epsilon of each other. The details seem different, but I'd expect VERY different graphs if SFO disabled SQM.

---

<div class="post-metadata">

**Author:** ![dl12345](https://forum.openwrt.org/letter_avatar/dl12345/32/5_5575768a8748004e209b776fc1b2916d.png) [@dl12345](https://forum.openwrt.org/u/dl12345)\
**Post date:** [March 13, 2021, 12:31am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/9 "2021-03-13T00:31:54Z")

</div>

Yes, this is SQM running with a diffserv8 profile in a gigabit link. Two identical tests run one after another. Some of the apparent difference is due to the scaling, but if you look at the two upload graphs, note how CS0 and CS2/CS3 are inverted between the two runs

---

<div class="post-metadata">

**Author:** ![dl12345](https://forum.openwrt.org/letter_avatar/dl12345/32/5_5575768a8748004e209b776fc1b2916d.png) [@dl12345](https://forum.openwrt.org/u/dl12345)\
**Post date:** [March 13, 2021, 12:38am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/10 "2021-03-13T00:38:51Z")

</div>

It definitely doesn't disable SQM, but it seems to change the behaviour appreciably, right?

---

<div class="post-metadata">

**Author:** ![dlakelan](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dlakelan/32/2320_2.png) [@dlakelan](https://forum.openwrt.org/u/dlakelan)\
**Post date:** [March 13, 2021, 12:43am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/11 "2021-03-13T00:43:14Z")

</div>

Yeah, I'm not sure what's up there. But if you turn on **hardware** offloading the qdiscs aren't involved **at all** and so SQM does nothing. If you turn on software offloading it may change exactly how cake handles thing potentially, but it also might be specific to diffserv8 which is I think an unusual config.

in general **software** offloading still uses the qdiscs, it mainly just bypasses the details of iptables. Basically the initial packets in a flow go through iptables, and then the remaining ones go through the "fast path" but they still hit the qdiscs.

---

<div class="post-metadata">

**Author:** ![dl12345](https://forum.openwrt.org/letter_avatar/dl12345/32/5_5575768a8748004e209b776fc1b2916d.png) [@dl12345](https://forum.openwrt.org/u/dl12345)\
**Post date:** [March 13, 2021, 12:44am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/12 "2021-03-13T00:44:01Z")

</div>

Let me do two runs with a diffserv4 profile and see what it looks like...

---

<div class="post-metadata">

**Author:** ![dl12345](https://forum.openwrt.org/letter_avatar/dl12345/32/5_5575768a8748004e209b776fc1b2916d.png) [@dl12345](https://forum.openwrt.org/u/dl12345)\
**Post date:** [March 13, 2021, 12:55am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/13 "2021-03-13T00:55:00Z")

</div>

Diffserv4 profile now: first is with SFO on, second with SFO off. Similar behaviour....

 ![image](https://forum.openwrt.org/uploads/default/original/3X/8/7/870db4a835f4d2cbcb97ae8ff0096b434efb82e8.png)

 ![image](https://forum.openwrt.org/uploads/default/original/3X/4/4/44f3333e9b3b85fc5b06193f2454b9b7edd7ec6f.jpeg)

---

<div class="post-metadata">

**Author:** ![dlakelan](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dlakelan/32/2320_2.png) [@dlakelan](https://forum.openwrt.org/u/dlakelan)\
**Post date:** [March 13, 2021, 1:10am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/14 "2021-03-13T01:10:38Z")

</div>

I agree it's weird how different tiers seem to priority-invert. I wonder what @moeller0 thinks is going on?

---

<div class="post-metadata">

**Author:** ![xorbug](https://forum.openwrt.org/letter_avatar/xorbug/32/5_5575768a8748004e209b776fc1b2916d.png) [@xorbug](https://forum.openwrt.org/u/xorbug)\
**Post date:** [March 13, 2021, 9:50am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/15 "2021-03-13T09:50:06Z")

</div>

> [@dlakelan](#):
>
> @bobafetthotmail and @xorbug as far as I understand **software** offloading still works with SQM just fine. Only hardware offloading is incompatible.

Ah good, I wasn't able to get that by reading around about SFO... also the "_Experimental feature. Not fully compatible with QoS/SQM._" warning in the LuCI firewall tab appears above the SFO checkbox (when HFO checkbox is not even visible) so maybe I got confused a bit by that...

@dl12345 thanks for taking the effort of sharing these benchmarks... very informative indeed.

> [@dl12345](#):
>
> The first test shows SFO enabled and the second disabled.

> [@dlakelan](#):
>
> The details seem different, but I'd expect VERY different graphs if SFO disabled SQM.

Speaking as a layman about combined operation of SFO+SQM including these graphs meaning (which i never met but i just imagine to be measurements for different classes of traffic), could it be at this point the other way round?  
I mean, since SFO disabled/enabled has so little impact (as if it's not operating at all, well, except the priority inversion, and the shapes too), maybe it's SQM that "disables" SFO no matter what the settings are and the cli/gui shows, and not the opposite? i.e. you are operating _without_ SFO all the time? May this be possible? Or those iverted priorities and different shapes are a clear evidence of SFO?  
(sorry in case this looks cocky, i see you know lots more than me on the subject, i don't mean to say you are not aware of what settings you are running or you cannot notice the differences by yourself... just asking to learn something 😉)

---

<div class="post-metadata">

**Author:** ![dlakelan](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dlakelan/32/2320_2.png) [@dlakelan](https://forum.openwrt.org/u/dlakelan)\
**Post date:** [March 13, 2021, 2:12pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/16 "2021-03-13T14:12:46Z")

</div>

> [@xorbug](#):
>
> maybe it's SQM that "disables" SFO no matter what the settings are and the cli/gui shows, and not the opposite?

This is a good thought but I don't think so. @moeller0 is there anything in the SQM scripts that disables software flow offload?

Reading further on the internet, it's clear that software offloads still send packets via `neigh_xmit` which I believe still sends things through the qdisc so SQM should work. The priority inversion is weird though.

---

<div class="post-metadata">

**Author:** ![moeller0](https://forum.openwrt.org/user_avatar/forum.openwrt.org/moeller0/32/1591_2.png) [@moeller0](https://forum.openwrt.org/u/moeller0)\
**Post date:** [March 13, 2021, 2:42pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/17 "2021-03-13T14:42:00Z")

</div>

> [@dlakelan](#):
>
> This is a good thought but I don't think so. @moeller0 is there anything in the SQM scripts that disables software flow offload?

No idea, but SQM does nothing actively to foil software offloading...

---

<div class="post-metadata">

**Author:** ![Sparks](https://forum.openwrt.org/user_avatar/forum.openwrt.org/sparks/32/8563_2.png) [@Sparks](https://forum.openwrt.org/u/Sparks)\
**Post date:** [March 13, 2021, 9:29pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/18 "2021-03-13T21:29:15Z")

</div>

For flows in the outbound direction , Im curious whether the sqm layer is before or after software flow offloading layer.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://forum.openwrt.org/user_avatar/forum.openwrt.org/dlakelan/32/2320_2.png) [@dlakelan](https://forum.openwrt.org/u/dlakelan)\
**Post date:** [March 13, 2021, 10:04pm UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/19 "2021-03-13T22:04:51Z")

</div>

Software offloading sends packets direct to the xmit path, and that means into the qdiscs, so SQM works both directions afaik.

---

<div class="post-metadata">

**Author:** ![xorbug](https://forum.openwrt.org/letter_avatar/xorbug/32/5_5575768a8748004e209b776fc1b2916d.png) [@xorbug](https://forum.openwrt.org/u/xorbug)\
**Post date:** [March 14, 2021, 11:36am UTC](https://forum.openwrt.org/t/software-flow-offloading-implications/90957/20 "2021-03-14T11:36:37Z")

</div>

> [@dl12345](#):
>
> if you look at the two upload graphs, note how CS0 and CS2/CS3 are inverted between the two runs

And in the end, which one was the expected/good result? the one with SFO on or SFO off?

> [@dlakelan](#):
>
> Reading further on the internet, it's clear that software offloads still send packets via `neigh_xmit` which I believe still sends things through the qdisc so SQM should work.

Ok i have done a couple tests myself with SFO and SQM (first time setting up SQM for me).  
Well, SQM is clearly working and the difference is like night and day on latency. Additionally i see a _FLOWOFFLOAD_ target in iptables, and it is being hit since the counter increases. I'd say they are both working fine together then.

[Next page](https://forum.openwrt.org/t/software-flow-offloading-implications/90957.md?page=2)
