Help me update my HFSC shaper scripts for fw4/nftables

Well last night I was only able to try htop. And when I perform the speed test with the CPU monitor I see that it reaches 100%. So, it means that SQM activated. On my Linsys WRT 32X router. Are you running out of processor?


Do as suggested. And I don't change anything. Do you have a symmetrical connection? Apparently with SQM my router is 100% using the processor :frowning:

@NeMe_FuUuRyyyyY please post the output of:

cat /proc/interrupts
cat /proc/softirq

one core is maxed out the other idle, maybe the processing could be moved around a bit...

yes, it's quite a bit more sophisticated than my very simple filter. The very simple filter was a simple stopgap for people with 20:1 asymmetric DSL type lines.

I should probably think up some smarter method...

@Hudra

What if we separated acks into their own leaf class with an upper limit of like 10% of the upstream bandwidth, and then put sfq with depth 5 and head-drop below that. Probably smarter than my rate limiting code in nftables.

Why bother? To do it right you need to split the packet into individual flows (or you have to search the full queue for every ACK entering) at which point, just use cake as leaf qdisc...

1 Like

Yeah, I think the only reason to bother would be maybe to have a system that's less CPU intensive than running all the features of cake. But, probably not worth the hassle.

You do have a point, cake has a bad reputation for being resource hungry... I wonder how much of that is caused by the shaper component and how much by the rest, because it is possible to use cake as simple leaf qdisc... it is also possible to overrode cake's priority tier/dscp mappings

Yeah, absolutely if we let HFSC do the shaping and cake be the leaf qdisc with ack suppression and fair queue sort of stuff only it might be a really good idea. Or it might be a bit overpowered. Someone should do the tests comparing cake as leaf qdisc to fq_codel as leaf and see the CPU usage.

1 Like

cat /proc/interrupts

root@OpenWrt:~# cat /proc/interrupts
           CPU0       CPU1
 25:          0          0     GIC-0  27 Edge      gt
 26:      92586      42162     GIC-0  29 Edge      twd
 27:          0          0      MPIC   5 Level     armada_370_xp_per_cpu_tick
 29:      31626          0     GIC-0  34 Level     mv64xxx_i2c
 30:         58          0     GIC-0  44 Level     ttyS0
 40:          0          0     GIC-0  41 Level     f1020300.watchdog
 44:          0          0     GIC-0  96 Level     f1020300.watchdog
 45:      86160          0      MPIC   8 Level     eth0
 46:          0          0     GIC-0  50 Level     ehci_hcd:usb1
 47:          0          0     GIC-0  51 Level     f1090000.crypto
 48:          0          0     GIC-0  52 Level     f1090000.crypto
 49:          0          0     GIC-0  58 Level     ahci-mvebu[f10a8000.sata]
 50:         95          0     GIC-0 116 Level     marvell-nfc
 51:      69550          0     GIC-0  57 Level     mmc0
 52:          0          0     GIC-0  49 Level     xhci-hcd:usb2
 53:          2          0     GIC-0  54 Level     f1060800.xor
 54:          2          0     GIC-0  97 Level     f1060900.xor
 58:          0          0  mv88e6xxx-g1   3 Edge      mv88e6xxx-f1072004.mdio-m                                                                                                                                                             ii:00-g1-atu-prob
 60:          0          0  mv88e6xxx-g1   5 Edge      mv88e6xxx-f1072004.mdio-m                                                                                                                                                             ii:00-g1-vtu-prob
 62:          9          3  mv88e6xxx-g1   7 Edge      mv88e6xxx-f1072004.mdio-m                                                                                                                                                             ii:00-g2
 64:          0          0  mv88e6xxx-g2   0 Edge      mv88e6xxx-1:00
 65:          0          0  mv88e6xxx-g2   1 Edge      mv88e6xxx-1:01
 66:          0          0  mv88e6xxx-g2   2 Edge      mv88e6xxx-1:02
 67:          4          1  mv88e6xxx-g2   3 Edge      mv88e6xxx-1:03
 68:          5          2  mv88e6xxx-g2   4 Edge      mv88e6xxx-1:04
 79:          0          0  mv88e6xxx-g2  15 Edge      mv88e6xxx-f1072004.mdio-m                                                                                                                                                             ii:00-watchdog
 80:          0          0  f1018100.gpio  24 Edge      gpio-keys
 81:          0          0  f1018100.gpio  29 Edge      gpio-keys
 82:     127783          0     GIC-0  61 Level     mwlwifi
 83:      44313          0     GIC-0  65 Level     mwlwifi
IPI0:          0          1  CPU wakeup interrupts
IPI1:          0          0  Timer broadcast interrupts
IPI2:       2105       2049  Rescheduling interrupts
IPI3:       5524      59080  Function call interrupts
IPI4:          0          0  CPU stop interrupts
IPI5:          0          0  IRQ work interrupts
IPI6:          0          0  completion interrupts
Err:          0
root@OpenWrt:~#

cat /proc/softirq

root@OpenWrt:~# cat /proc/softirq
cat: can't open '/proc/softirq': No such file or directory
root@OpenWrt:~#
root@OpenWrt:~# cat /proc/softirq
cat: can't open '/proc/softirq': No such file or directory
root@OpenWrt:~# cat /proc/softirq
cat: can't open '/proc/softirq': No such file or directory
root@OpenWrt:~# cat /proc/softirq
cat: can't open '/proc/softirq': No such file or directory
root@OpenWrt:~#

Can you help me with the configuration?

Sorry a typo, please try:

cat /proc/softirqs 

but /proc/interrupts is already indicating much stuff on CPU0, exactly the CPU that saturated in your htop screenshots...

1 Like

Good perfect! Thanks for the help moeller0. And sorry for the delay, I'm working a lot and I'm late getting home, that's why it takes me a while to respond. Help is appreciated :slight_smile:

Just did a really quick test on an openwrt VM (only one core) and changed:

echo "adding fq_codel qdisc for non-game traffic"
for i in 12 13 14 15; do 
    tc qdisc add dev "$DEV" parent "1:$i" fq_codel memory_limit $((RATE*200/8)) interval "${INTVL}ms" target "${TARG}ms" quantum $((MTU * 2))
done

to

echo "adding cake qdisc for non-game traffic"
for i in 12 13 14 15; do 
    tc qdisc add dev "$DEV" parent "1:$i" cake bandwidth ${RATE}kbit ack-filter besteffort
done

Bandwidth was set to:

DOWNRATE=900000 
UPRATE=900000

With cake as leaf qdisc for the non gaming classes:

With fq_codel as leaf qdisc for the non gaming classes:

2 Likes

You maybe do not want to use cake's traffic shaper here...

1 Like

unlimited is what you should be using, I believe... if you attach cake to an existing shaper...

2 Likes

Makes sense. Still pretty much the same result with cake:

root@OpenWrt:~# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5ms interval 100ms memory_limit 32Mb ecn drop_batch 64 
 Sent 33826082638 bytes 22363571 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  maxpacket 0 drop_overlimit 0 new_flow_count 0 ecn_mark 0
  new_flows_len 0 old_flows_len 0
qdisc hfsc 1: dev eth1 root refcnt 2 default 13 
 Sent 151571533 bytes 2253787 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8184: dev eth1 parent 1:15 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 107879498 bytes 1604442 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 768b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           55 /      87
 min/max overhead-adjusted size:       55 /      87
 average network hdr offset:           14

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay          0us
  av_delay          0us
  sp_delay          0us
  backlog            0b
  pkts          1604442
  bytes       107879498
  way_inds            0
  way_miss            1
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len            87
  quantum          1514

qdisc bfifo 10: dev eth1 parent 1:11 limit 406200b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8183: dev eth1 parent 1:14 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 0b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:        65535 /       0
 min/max overhead-adjusted size:    65535 /       0
 average network hdr offset:            0

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay          0us
  av_delay          0us
  sp_delay          0us
  backlog            0b
  pkts                0
  bytes               0
  way_inds            0
  way_miss            0
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            0
  un_flows            0
  max_len             0
  quantum          1514

qdisc cake 8181: dev eth1 parent 1:12 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 727805 bytes 10829 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 4352b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           55 /     358
 min/max overhead-adjusted size:       55 /     358
 average network hdr offset:           14

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay          3us
  av_delay          0us
  sp_delay          0us
  backlog            0b
  pkts            10829
  bytes          727805
  way_inds            0
  way_miss            5
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            1
  bk_flows            1
  un_flows            0
  max_len           358
  quantum          1514

qdisc cake 8182: dev eth1 parent 1:13 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 42964132 bytes 638515 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 2534b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           43 /    1309
 min/max overhead-adjusted size:       43 /    1309
 average network hdr offset:           14

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay         15us
  av_delay          1us
  sp_delay          0us
  backlog            0b
  pkts           638515
  bytes        42964132
  way_inds            0
  way_miss           13
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len          1309
  quantum          1514

qdisc ingress ffff: dev eth1 parent ffff:fff1 ---------------- 
 Sent 7512141030 bytes 2675 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc hfsc 1: dev ifb-eth1 root refcnt 2 default 13 
 Sent 7844637471 bytes 5182092 pkt (dropped 34, overlimits 1514854 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8188: dev ifb-eth1 parent 1:15 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 5620420469 bytes 3712298 pkt (dropped 23, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 1028496b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:         1514 /    1515
 min/max overhead-adjusted size:     1514 /    1515
 average network hdr offset:           14

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay        5.7ms
  av_delay       5.54ms
  sp_delay       5.24ms
  backlog            0b
  pkts          3712321
  bytes      5620455291
  way_inds            0
  way_miss            1
  way_cols            0
  drops              23
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         65227
  quantum          1514

qdisc cake 8186: dev ifb-eth1 parent 1:13 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 2214701546 bytes 1463137 pkt (dropped 11, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 2009752b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           43 /    1515
 min/max overhead-adjusted size:       43 /    1515
 average network hdr offset:           14

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay       1.89ms
  av_delay       1.72ms
  sp_delay          1us
  backlog            0b
  pkts          1463148
  bytes      2214718200
  way_inds            2
  way_miss           31
  way_cols            0
  drops              11
  marks               0
  ack_drop            0
  sp_flows            2
  bk_flows            1
  un_flows            0
  max_len         65227
  quantum          1514

qdisc bfifo 10: dev ifb-eth1 parent 1:11 limit 406200b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc cake 8185: dev ifb-eth1 parent 1:12 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 9515456 bytes 6657 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 113016b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:           61 /    1515
 min/max overhead-adjusted size:       61 /    1515
 average network hdr offset:           14

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay        394us
  av_delay        186us
  sp_delay          1us
  backlog            0b
  pkts             6657
  bytes         9515456
  way_inds            0
  way_miss            7
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            1
  un_flows            0
  max_len         65227
  quantum          1514

qdisc cake 8187: dev ifb-eth1 parent 1:14 bandwidth unlimited besteffort triple-isolate nonat nowash ack-filter split-gso rtt 100ms raw overhead 0 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
 memory used: 0b of 15140Kb
 capacity estimate: 0bit
 min/max network layer size:        65535 /       0
 min/max overhead-adjusted size:    65535 /       0
 average network hdr offset:            0

                  Tin 0
  thresh           0bit
  target            5ms
  interval        100ms
  pk_delay          0us
  av_delay          0us
  sp_delay          0us
  backlog            0b
  pkts                0
  bytes               0
  way_inds            0
  way_miss            0
  way_cols            0
  drops               0
  marks               0
  ack_drop            0
  sp_flows            0
  bk_flows            0
  un_flows            0
  max_len             0
  quantum          1514

root@OpenWrt:~# 
1 Like

cat /proc/softirqs

root@OpenWrt:~# cat /proc/softirqs
                    CPU0       CPU1
          HI:          0          0
       TIMER:    5194728    2756639
      NET_TX:    6557712         63
      NET_RX:   20746565     918479
       BLOCK:          0          0
    IRQ_POLL:          0          0
     TASKLET:   63947507      47462
       SCHED:    7537011    7011871
     HRTIMER:          0          0
         RCU:     342399     277834
root@OpenWrt:~#

Same story here, too much work on CPU0, try to enable packet steering and irqbalance...

1 Like

Was wondering how many people are using this script and have felt a distinct effect when playing games?
How do people differentiate between the normal internet and using this script?
Is there gameplay evidence to show theres advantage to using these router based scripts?
Having tired running games with no sqm.
And with sqm enabled i fail to see and difference
Also having used this script also makes no meaning full difference.
The only thing i have felt is when using netem to add delay to ping.
Are these design for people with poor speed, bad connections.

Zia