Questions about SQM settings

[HelloShitty]

Like this:

config queue 'wan'                                                              
        option enabled '1'                                                      
        option interface 'eth0.12'                                              
        option download '75000'                                                 
        option upload '75000'                                                                                              
        option qdisc 'fq_codel'                                                                                  
        option script 'simplest.qos'                                            
        option qdisc_advanced '1'                                                                                         
        option qdisc_really_really_advanced '0'                                 
        option itarget 'auto'                                                   
        option etarget 'auto'                                                   
        option linklayer 'ethernet'                                             
        option squash_dscp '1'                                                  
        option squash_ingress '1'                                               
        option overhead '44'                                                    
        option debug_logging '1'                                                
        option verbosity '5'                                                    
        option iqdisc_opts 'flows'                                              
        option eqdisc_opts 'flows'

[moeller0]

Almost, simplest.qos/fq_codel will not accept the flows keyword, so leave iqdisc_opts and eqdisc_opts empty....

[HelloShitty]

I'm lost now. Didn't you say above to consider using

@moeller0

simplest.qos/fq_codel or even simplest_TBF.qos/fq_codel as these might be more CPU efficient when none of cake's bells and whistles are used.

But now you're telling me that they won't work with flows

Can you please clear what I shoul use?

[moeller0]

@HelloShitty

But now you're telling me that they won't work with flows

No, my intent was telling you that simplest.qos/fq_codel will by default use the same policy that in cake needs to be specifically requested with the flows keyword. And fq_codel will not accept flows as parameter so puting that in iqdisc_opts/eqdisc_opts will cause an error.

@HelloShitty

Can you please clear what I shoul use?

IMHO either simplest_TBF/fq_codel (to get the maximum throughput with AQM & traffic shaper) or layer_cake.qos/cake with the per-IP-isolation mode to get the most robust configuration where no singe machine can easily crowd out all the rest.
But it is not my place to tell you what to use really, I hope I explained the trade-offs of the different alternatives well enough so you can pick the policy you seem best for your network.

[HelloShitty]

I got it about the trade offs. And I don't want to over complicate because I also don't have the goal to understand it all deeply as you do! Goal is to get the best speed I can possibly can at the best bufferbloat score I can.

Out of the box I have Grade A, without any sqm script. With sqm scripts I can have A+ but at the cost of some speed loss.

I probably don't even need to get into those per IP modes etc, so first, I'll just make the script work with a set of correct settings. Then, maybe I can start tweaking, if needed, or for the sake of curiosity.

I spent this last week with the speed cur donwn to 75Mbps (which was the limit my router cpu started to getting too close to 100% load) and IPTV didn't improve, so I think IPTV is not being a victim of bufferbloat.

Knowing this, I'm not going to bother any more with bufferbloat regarding IPTV issues.
Just gonna try to get the best I can with my setup!

So, to make sure I got it right, check please, if these 2 options are correct in terms of accepte option by each discipline (is that what it's called cake and fq_codel?)

Option 1 fq_codel (no 'flows')

config queue 'wan'                                                              
        option enabled '1'                                                      
        option interface 'eth0.12'                                              
        option download '75000'                                                 
        option upload '75000'                                                                                                 
        option qdisc 'fq_codel'                                                                                
        option script 'simplest.qos'                                            
        option qdisc_advanced '1'                                                                                         
        option qdisc_really_really_advanced '0'                                 
        option itarget 'auto'                                                   
        option etarget 'auto'                                                   
        option linklayer 'ethernet'                                             
        option squash_dscp '1'                                                  
        option squash_ingress '1'                                               
        option overhead '44'                                                    
        option debug_logging '1'                                                
        option verbosity '5'                                                    
        option iqdisc_opts ''                                                   
        option eqdisc_opts ''

Option 2 cake

config queue 'wan'                                                              
        option enabled '1'                                                      
        option interface 'eth0.12'                                              
        option download '75000'                                                 
        option upload '75000'                                                   
        option qdisc 'cake'                                                                                              
        option script 'piece_of_cake.qos'                                                                            
        option qdisc_advanced '1'                                                                                           
        option qdisc_really_really_advanced '0'                                 
        option itarget 'auto'                                                   
        option etarget 'auto'                                                   
        option linklayer 'ethernet'                                             
        option squash_dscp '1'                                                  
        option squash_ingress '1'                                               
        option overhead '44'                                                    
        option debug_logging '1'                                                
        option verbosity '5'                                                    
        option iqdisc_opts 'flows'                                                   
        option eqdisc_opts 'flows'

I'm going to stick to Option 1 for now, because I still haven't read how to setup the script for that per IP option. I mean, I think this is not even the option I want. In the worst case scenario, I wanted to give priority for the down/up link to one of my IPs, and I think this is not the case, so let's leave it for later!

Tel me please, if these 2 options are correct in terms of settings, please.
For Option 1, I'll be running a few tests to see if I can increase the speed and how much I can increase until I get too close to 100% load on the cpu!

I have one more question though. About the ECN and NOECN options, can I ignore them in those 2 options?

Thank you in advance!

[moeller0]

@HelloShitty

Tel me please, if these 2 options are correct in terms of settings, please.

As far as I can tell, yes these seem to be functional.

@HelloShitty

For Option 1, I'll be running a few tests to see if I can increase the speed and how much I can increase until I get too close to 100% load on the cpu!

OK, but if you do that really try simplest_TBF.qos as that was created to gain a bit more throughput on R7800 routers (like yours?).

@HelloShitty

I have one more question though. About the ECN and NOECN options, can I ignore them in those 2 options?

cake will always use ECN in both directions (but each flow needs to negotiate the use of ECN, so you can always disable ECN in your clients). fq_codel however evaluates these settings. Whether to enable or disable them is a matter of preference (only on slower links disabling ECN on egress is potentially better).

@HelloShitty

I still haven't read how to setup the script for that per IP option.

Easy for piece_of_cake.qos/cake or layer_cake.qos/cake just add:

option iqdisc_opts 'nat dual-dsthost ingress'                                                   
option eqdisc_opts 'nat dual-srchost'

to your configuration to configure per-internal-IP-fairness.

@HelloShitty

In the worst case scenario, I wanted to give priority for the down/up link to one of my IPs, and I think this is not the case, so let's leave it for later!

In Layer_cake.qos you can also play with DSCPs to give priority. But honestly, unless you really need targeted unfairness (like 8 Mbps upload for OBS-streaming from one host on a 10Mbps link) per-internal-IP fairness has one thing going for it, no host will get starved. But again a question of policy you need to answer for your network.

[HelloShitty]

Ok, I've been trying the simplest_tbf.qos but I think somehow I get capped at aroun 250Mbps / 95Mbps and the cpu is not yet that close to 100%. But no matter if I increase more the limit, the result is not better.

I will try more tomorrow. As of now, this is what I get, wired, but with TV (IPTV at least also using the internet).

config queue 'wan'                                                                                                                                                                                                                            
        option enabled '1'                                                                                                                                                                                                                    
        option interface 'eth0.12'                                                                                                                                                                                                            
        option download '280000'                                                                                                                                                                                                              
        option upload '95000'                                                                                                                                                                                                                                                                                                                                                                                                                                
        option qdisc 'fq_codel'                                                                                                                                                                                                  
        option script 'simplest_tbf.qos'
	option qdisc_advanced '1'                                                                                                                                                                                                             
        option qdisc_really_really_advanced '0'                                                                                                                                                                                               
        option itarget 'auto'                                                                                                                                                                                                                 
        option etarget 'auto'                                                                                                                                                                                                                 
        option linklayer 'ethernet'                                                                                                                                                                                                           
        option squash_dscp '1'                                                                                                                                                                                                                
        option squash_ingress '1'                                                                                                                                                                                                             
        option overhead '44'
        option debug_logging '1'                                                                                                                                                                                                              
        option verbosity '5'                                                                                                                                                                                                                  
	option iqdisc_opts ''
        option eqdisc_opts ''

Bufferbloat Test Result: A+

[JonP]

@slh

@moeller0

Here is the problem, on a dual CPU router like yours, 48% idle might mean one CPU essentially fully busy at 100% while the other only has a 4% load -> 100-(100+4)/2 = 48% idl

Exactly this, you will see one core at 90+% and the other perhaps around 20%-30%.

There is no magic bullet, but x86_64, RPi4 and r4s can deal with these speeds (and sqm) easily - and the r7800 can stil serve as managed switch and AP. During (local) testing, my c1037u was totally bored while doing sqm at 1 GBit/s (symmetric) line speed - it didn't even have to clock up - extrapolating, I'd venture that even this ~10 year old x86_64 CPU would easily do 2.5 GBit/s.

Has the original poster tried the Packet Steering option? I actually don't know where this lives on the command line, in Luci its found under the Network -> Interfaces -> Global Network Options tabs, and then a checkbox. It enables packet handling to get spread across CPU cores. Another thing to try would be to install irqbalance, (need to enable it after package install, OWT website has a page on that) which does something similar. Sometimes, it's beneficial to do both.

Since you have a dual core CPU, and cake is a single threaded process, it should help if some of the rest of the load gets spread more evenly. You might find a higher limit threshold than you have now. How much, I couldn't say offhand. It isn't going to get you to 500mbit.

Otherwise, to fully utilize the speed of your link, you probably want to look into something with more CPU power to handle the load. This thread will point you in a good direction, with solutions that have already been suggested. https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and-need-a-router-read-this-first/90305

[moeller0]

@HelloShitty

I get capped at aroun 250Mbps / 95Mbps

If I understand correctly that is already noticeably higher than with piece_of_cake/cake, no?

@HelloShitty

and the cpu is not yet that close to 100%.

See @JonP's post on the peculiarities of detecting CPU overload (for individual CPUs).

@HelloShitty

But no matter if I increase more the limit, the result is not better.

That indicates that for the current setting you have achieved a maximum, some fine tweaking might still be possible...

@HelloShitty

As of now, this is what I get, wired, but with TV (IPTV at least also using the internet).

So you split out IPTV between the ONT and your router, so outside of SQM? Which is fine as long as:

peak IPTV rate + SQM shaper rate < true bottleneck rate

for each direction...

[HelloShitty]

@JonP

Has the original poster tried the Packet Steering option?

I haven't, no.

irqbalance is kinda familiar to me but not sure I ever did anything with it!

[HelloShitty]

@moeller0

If I understand correctly that is already noticeably higher than with piece_of_cake/cake, no?

Yes, quite better, yes.

@moeller0

See @JonP's post on the peculiarities of detecting CPU overload (for individual CPUs).

I know. We discussed this before. I am looking to induvidual cpu loads. It's only cpu0 that is dealing with sqm!

@moeller0

That indicates that for the current setting you have achieved a maximum, some fine tweaking might still be possible...

But why maximum if the cpu load still have some room?

@moeller0

So you split out IPTV between the ONT and your router, so outside of SQM? Which is fine as long as:

I'm not sure I understand what you mean, or better if what I understood is in fact what you mean, but I didn't split anything. I mean, I've only been changing settings in sqm config file. And my config file is exactly the one above with fq_codel settings!

[moeller0]

@HelloShitty

But why maximum if the cpu load still have some room?

Well, if increasing the shaper setting does not result in more throughput there clearly is some limit reached somewhere....

@HelloShitty

I'm not sure I understand what you mean,

So do the IPTV consumers (set-top boxes, computers, rokus, appleTVs, ...) sit behind or in front of your OpenWrt router? Both placements can work, but in front of the router (so not under control of SQM) requires more care and is generally not recommended. I simply have not understood how you set up your network in regards to the IPTV clients.

[HelloShitty]

@moeller0

So do the IPTV consumers (set-top boxes, computers, rokus, appleTVs, ...) sit behind or in front of your OpenWrt router?

The setup is ISP ONT -> Netgear R7800 -> Android Box + old LCD TV // RockPro64 // 2x laptops // 1x Desktop // 4x smartphones // 1x printer and I think it's all. The // symbol represents that all these devices are conencted to the router (meaning after the router in terms of placement in line).

[moeller0]

Ah, okay, then I was confused, please ignore my IPTV comments.

[JonP]

@HelloShitty

@moeller0

See @JonP's post on the peculiarities of detecting CPU overload (for individual CPUs).

I know. We discussed this before. I am looking to induvidual cpu loads. It's only cpu0 that is dealing with sqm!

@moeller0

That indicates that for the current setting you have achieved a maximum, some fine tweaking might still be possible...

But why maximum if the cpu load still have some room?

CPU load and SIRQ are two different things. You can reach the IRQ rate limits of your box and still be below 100% CPU use. Paying attention to idle time % and SIRQ % will tell you that. (idle time 0%, SIRQ 100%) That's probably what's happening. Watch SIRQ and idle time in top.

Another thing many don't know at first, is when using htop, you don't see the interrupts in the bar graphs on the default settings by default. Go into the settings (F2) and tick the box that says something like extended CPU info. Then you will have interrupts showing in purple on the bar graph.

[HelloShitty]

I just did another testing with the following settings

# cat /etc/config/sqm
config queue 'wan'                                                                                                                                                                                                                            
        option enabled '1'                                                                                                                                                                                                                    
        option interface 'eth0.12'                                                                                                                                                                                                            
        option download '280000'                                                                                                                                                                                                              
        option upload '95000'                                                                                                                                                                                                                                                                                                                                                                                                                               
        option qdisc 'fq_codel'                                                                                                                                                                                                   
        option script 'simplest_tbf.qos'
	option qdisc_advanced '1'                                                                                                                                                                                                             
        option qdisc_really_really_advanced '0'                                                                                                                                                                                               
        option itarget 'auto'                                                                                                                                                                                                                 
        option etarget 'auto'                                                                                                                                                                                                                 
        option linklayer 'ethernet'                                                                                                                                                                                                           
        option squash_dscp '1'                                                                                                                                                                                                                
        option squash_ingress '1'                                                                                                                                                                                                             
        option overhead '44'
        option debug_logging '1'                                                                                                                                                                                                              
        option verbosity '5'                                                                                                                                                                                                                  
	option iqdisc_opts ''
        option eqdisc_opts ''

and got the following reesults

Bufferbloat Test Result: A+

top showed a max of 71% idle and 46% sirq

I have one other question I think I know the answer but wanted confirmation.
I was told here to consider using my RockPro64 to do the SQM thing but I woul need to use the PCIe slot to add a new RJ45 port.

My question is if it is not possible/advantageous to use an USB 3 to RJ45 hub or something like that, allowing me to keep using the PCIe slot for my NVMe drive instead of using it for the extra RJ45 port!

I think USB 3 may be the bottleneck if we consider maximum speeds of my link but I'm not 100% sure on this.

[moeller0]

@HelloShitty

My question is if it is not possible/advantageous to use an USB 3 to RJ45 hub or something like that, allowing me to keep using the PCIe slot for my NVMe drive instead of using it for the extra RJ45 port!

Yes it is possible to do that, on raspberry pi 4B's people in this forum converged on the TP-Link UE300 USB3 gigabit ethernet dongle. It is not recommended as these have pretty substantial bufferbloat themselves, but if you use SQM to shape up- and downstream WAN traffic then using an USB dongle for WAN will avoid noticing the USB-bufferbloat. You are also limited basically to realtek ethernet dongles (while on PCIe you can select from realtek, broadcom, intel, ...) which in the past was sub-optimal, but current realtek devices seem quite OK.

The bigger question is why do you want to keep an NVMe device on a router... the router itself is not going to profit from that much (routing is IO and CPU intensive, but not storage wise unless you want to log everything to disk)...
It is tempting to centralize functionality onto one's router (given that this needs to run 24/7 anyway) but that has a number of caveats:
a) security, if your router falls to an external attack so do your converged services
b) isolation, if your router needs a reboot, your NAS might prefer to keep up and running, but if running on the router their uptime is now linked.

[HelloShitty]

Well, this RockPro64 is running a few services I don't want to stop or chage and removing the NVMe drive would force me to have to reinstall and re-configure all those services and that would take me quite some time. Another reason is that I invested quite some money on this drive and setup and I just don't want to ditch it and make the investement a complete loss. Like put the NVMe drive aside would represent a complete loss of the investment because I wouldn't be using it at all.

Anyway, about the speed result, do you think I still have room to increase speeds? Like, I think there is still room, but according to the results, looks like increasing more the limit in sqm config file is not going to give me higher download speeds while keeping the byfferbloat at A+, I guess!

[moeller0]

@HelloShitty

Well, this RockPro64 is running a few services I don't want to stop or chage and removing the NVMe drive would force me to have to reinstall and re-configure all those services and that would take me quite some time.

But if you want the rockpro to become your OpenWrt router you will need to restart it anyway, no?

@HelloShitty

Another reason is that I invested quite some money on this drive and setup and I just don't want to ditch it and make the investement a complete loss. Like put the NVMe drive aside would represent a complete loss of the investment because I wouldn't be using it at all.

Well, if the rock pro runs OpenWrt the NVMe device might technically be in use, but it will not help much for OpenWrt and you will likely not be able to keep the same services running you run right now.

Maybe you should keep that rockpro64 doing what it is doing now and consider a new or different device to take up the new router role instead?

@HelloShitty

Anyway, about the speed result, do you think I still have room to increase speeds?

For your R7800? the achieved 258.4/88.6 Mbps with simplest_TBF.qos/fq_codel appear close ot what that router is capable of. The theoretically best speetest results from your config would be around:

280 * (1500-20-20)/(1500+44) = 264.77 Mbps
95 * (1500-20-20)/(1500+44) = 89.83 Mbps

So you are getting reasonably close to the theoretical limits.

@HelloShitty

Like, I think there is still room, but according to the results, looks like increasing more the limit in sqm config file is not going to give me higher download speeds while keeping the byfferbloat at A+, I guess!

That however seems easier/quicker for you to figure out empirically than for me to speculate (given that I do not own a R7800, nor a link as fast as yours), no?

[HelloShitty]

@moeller0

But if you want the rockpro to become your OpenWrt router you will need to restart it anyway, no?

That's what I'm still considering. I'm nto sure I want to do it all from the beginning because at least, one of the services cannot be offline more than a week, and I don't know if I have time to migrate all things to a new drive in a week (very limited time uring the work days)
But yeah, that's something depending solely on my decisions.

@moeller0

Well, if the rock pro runs OpenWrt the NVMe device might technically be in use, but it will not help much for OpenWrt and you will likely not be able to keep the same services running you run right now.

Maybe you should keep that rockpro64 doing what it is doing now and consider a new or different device to take up the new router role instead?

Why you say I may not be able to keep running the same services I'm running now?
I lost myself here!

heheh, Another device woul get mylsef killed (by my wife). lol. I already have the RP64, the R7800, a Nintendo Switch OLED dock and an Android box on the living room furniture. Another device would drive her crazy! lol
And also, I really can't spend more money in another device. I think I'll stick to this one only, for now and maybe keep the internet traffic management on the R7800 even with the limitations and whenever I want full speed for a download, I disable SQM while the download is in progress, and re-enable SQM later!

@moeller0

280 * (1500-20-20)/(1500+44) = 264.77 Mbps
95 * (1500-20-20)/(1500+44) = 89.83 Mbps

Could you please explain the valaues you used?