Transparent cake box, where to apply sqm?

fernandodiacenco · November 7, 2020, 11:31pm

Hello,

I'm experimenting with a transparent Openwrt box, so far everything is working but I have a question on where to apply sqm in this case

My network looks like this

(each pfSense has its own public ip, does nat, etc)

I applied SQM on eth1, layer cake, no squash, ingress/egress at 100000 and everything works great so far

BUT, reading this thread something caught my eye

Correct, but ingress requires an IFB which incurrs some processing cost, so on case of using two interfaces, always instantiate sqm-scripts on egress (by setting the ingress bandwidth to 0, which denotes "do not shape" as "shape to 0" would end up with a non-functional link...)

So I tried setting sqm on both eth0 and eth1 with ingress at 0 (disable), egress at 100000, no squash, and everything works

My question is, is this correct? or I should just set it up on eth1 with ingress/egress and be happy?

Thanks, have a nice weekend.

moeller0 · November 8, 2020, 8:54am

So, both options work. ln your case I would use the SQM on eth0 and eth1 egress option as that will save you the processing cost of the IFB we typically use to instantiate a shaper for the ingress.
On normal full wifi routers the IFB method is safer as it will a) also cover traffic from the internet to the router (important if say you run a VPN server) and b) will not also shape Wifi to LAN traffic. Both complications do not arise in your configuration....
But please post the output of

tc -s qdisc
cat /etc/config/sqm

if you want some comments upon the rest of the options.

fernandodiacenco · November 8, 2020, 3:13pm

Oh, you're the guy from that thread! Thanks for answering!

I have two more questions if you won't mind

One is about ECN, as I understood ECN works by telling the sender to slow down when the receiver is reaching capacity, (which is a good thing?) In this setup, as I can see there is no ECN enabled, so nobody is telling the sender to slow down and is instead dropping packets? (is this a bad thing?)

The other question is, at home I have a similar setup, with a physical two nic quad core 4gb ram pc (no wifi) doing pppoe and with sqm enabled on the pppoe-wan interface (not the physical wan, there was like another virtual wan-pppoe to choose there), internet is vdsl at 40/20mbps

In this case, would it make sense to do the same, enable sqm on wan at ingress 0, egress 40000, lan at ingress 0, egress 20000 instead of having sqm at the pppoe interface?

Anyway, here is the output of tc -s qdisc

qdisc noqueue 0: dev lo root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc cake 8065: dev eth0 root refcnt 65 bandwidth 100Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
 Sent 596872950 bytes 2438681 pkt (dropped 1228, overlimits 878682 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 676928b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       6250Kbit      100Mbit       25Mbit
  target          5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms
  pk_delay          0us         75us         10us
  av_delay          0us          8us          5us
  sp_delay          0us          1us          2us
  backlog            0b           0b           0b
  pkts                0      2260802       179107
  bytes               0    589253260      9400956
  way_inds            0        52075            0
  way_miss            0       148569            1
  way_cols            0            0            0
  drops               0         1228            0
  marks               0            0            0
  ack_drop            0            0            0
  sp_flows            0            2            0
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514           60
  quantum           300         1514          762

qdisc cake 8068: dev eth1 root refcnt 65 bandwidth 100Mbit diffserv3 triple-isolate nonat nowash no-ack-filter split-gso rtt 100.0ms raw overhead 0
 Sent 4165203563 bytes 3160207 pkt (dropped 2509, overlimits 7714505 requeues 0)
 backlog 0b 0p requeues 0
 memory used: 1871232b of 5000000b
 capacity estimate: 100Mbit
 min/max network layer size:           42 /    1514
 min/max overhead-adjusted size:       42 /    1514
 average network hdr offset:           14

                   Bulk  Best Effort        Voice
  thresh       6250Kbit      100Mbit       25Mbit
  target          5.0ms        5.0ms        5.0ms
  interval      100.0ms      100.0ms      100.0ms
  pk_delay          0us        224us         10us
  av_delay          0us         18us          3us
  sp_delay          0us          3us          2us
  backlog            0b           0b           0b
  pkts                0      3162378          338
  bytes               0   4168837443        14196
  way_inds            0        20482            0
  way_miss            0         7703            1
  way_cols            0            0            0
  drops               0         2509            0
  marks               0          839            0
  ack_drop            0            0            0
  sp_flows            0            0            1
  bk_flows            0            1            0
  un_flows            0            0            0
  max_len             0         1514           42
  quantum           300         1514          762

qdisc mq 0: dev eth2 root
 Sent 51959424 bytes 114839 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth2 parent :8 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 22979872 bytes 36413 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 54 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :7 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 2809835 bytes 9087 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 485 drop_overlimit 0 new_flow_count 1 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :6 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 14075643 bytes 23293 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :5 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 1416487 bytes 7790 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :4 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 1267604 bytes 7051 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 54 drop_overlimit 0 new_flow_count 3 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :3 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 1366920 bytes 7216 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 3196392 bytes 9975 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth2 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
 Sent 4846671 bytes 14014 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
  maxpacket 66 drop_overlimit 0 new_flow_count 7 ecn_mark 0
  new_flows_len 1 old_flows_len 0
qdisc noqueue 0: dev br-BRIDGE root refcnt 2
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

the output of cat /etc/config/sqm

config queue
        option debug_logging '0'
        option verbosity '5'
        option linklayer 'none'
        option interface 'eth0'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option qdisc_really_really_advanced '0'
        option squash_ingress '0'
        option download '0'
        option enabled '1'
        option upload '100000'

config queue
        option debug_logging '0'
        option verbosity '5'
        option ingress_ecn 'ECN'
        option egress_ecn 'NOECN'
        option linklayer 'none'
        option interface 'eth1'
        option qdisc 'cake'
        option script 'layer_cake.qos'
        option qdisc_advanced '1'
        option squash_dscp '0'
        option squash_ingress '0'
        option qdisc_really_really_advanced '0'
        option enabled '1'
        option download '0'
        option upload '100000'

And the output of echo 'Thank you moeller0!'
Thank you moeller0!

moeller0 · November 8, 2020, 7:13pm

Mmmh, cake defaults to using ECN without a way to disable it, IIRC. The stats record this under the "marks" field. Whether that is good or bad really depends, on slow links I still believe that on egress dropping might be better as an marked packet still will block the link for a while, but on ingress that is different, as the packet is past the true botteneck once cake gets hold of it, that is why the GUI and the HTB/fq_codel based scripts default to ECN for downloads, noECEN for uploads, but for cake there is nothing to do. But you still need your clients and the upstream servers to actually use ECN, as only ECT(0) and ECT(1) carieying packets will be marked, everything else will be dropped. Not all client OS will by default try to negotiate ECN on a connection...

Sure, as long as there is not much traffic from the internet only towards the server you will be fine. If there is internet traffic terminating at the server cake on the interface towards the LAN will not see these and hence not properly include it in its calculations.

"raw overhead 0" is almost guaranteed to be wrong, on pppoe-wan I expect ~34 Bytes of per-packet-overhead.

Also maybe you want to try the per-internal-IP fairness modes?

fernandodiacenco · November 9, 2020, 5:42pm

You mean, traffic going to the router as if I was hosting something there like a website? But if traffic is passing trough it, like for something inside the lan, its fine, right?

Also maybe you want to try the per-internal-IP fairness modes?

I would love to, but for some reason I am not able to click on your link "per-internal-ip fairness modes"

I applied the transparent bridge today on production in two different states with an identical 100mbps link and a site-to-site ipsec between them, so far its going better than the fq_codel previously used on the pfsense(s), the only thing I noticed differently is a slight delay when first using the connection, like when first connecting to vpn, or first opening a website, there is a longer delay before an answer but feels very fast afterward. Testing and checking connection graphs I see no packet loss, low jitter, low delay, faster browsing, A+ in dslreports speedtest, improvements all around.
Thank you!

moeller0 · November 9, 2020, 6:13pm

Yes, of ftp, SFTP, even VPN can be an issue, as the router will see a higher per-packet-overhead, but you could account for that in the configuration for the LAN cake instance.

Yes.

Ooops, sorry, see https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#making_cake_sing_and_dance_on_a_tight_rope_without_a_safety_net_aka_advanced_features.

Mmmh, if this is related to cake you can expect to see this in the tc -s qdisc output, look for the peak delay (pk_delay) in comparison with the average delay (av_delay). I would say for a human to notice, it will take north of 100ms more delay so if cake is to blame (well possible) at least its counter should tel us.

fernandodiacenco · November 9, 2020, 8:17pm

Did some speedtests and ran tc -s qdisc | grep pk_delay to check

pk_delay 0us 4.4ms 61us
pk_delay 0us 8.4ms 51us

So this slow start is most likely not related to sqm, I believe its something on hyper-v, I suspect virtual machine queues (hyper-v always had a problem with that). I will spin up another non prod vm to test, will find the culprit eventually

So, about per internal ip fairness, if I understood correctly, in my transparent box I enable it on both interfaces like this

and be happy

BUT

at home since I do nat on the wan of the physical box, I enable like this on the LAN

But on the WAN I do

Since I applied sqm only on the egress of both nics but this one does nat, is that correct?

moeller0 · November 9, 2020, 9:35pm

Almost. If your run SQM on the router doing NAT you should add the nat keyword to both instances of sqm.

And the interface responsible for sending traffic into your LAN (so either ifb4pppoe-wan or eth1, assuming eth1 is the LAN interface) needs dual-dsthost while the interface sending traffic out into the internet needs dual-srchost.
This is the same for both of your setups.

fernandodiacenco · November 9, 2020, 11:34pm

Got it

So, just double checking

At home box with nat:

At work, transparent bridge not doing nat

Is this correct? Thanks.

moeller0 · November 10, 2020, 6:57am

Yes, that is how I would do it.