R7800 SQM performance

immi803 · October 20, 2021, 7:31pm

We live in ignorance when we stop exploring
Saw great builds for r7800, thanks @KONG @hnyman ,
Apology for interrupting developers thread, appreciate for bearing with me
Coming to the point

glimpse : prior i had asus ax88u plugged in at this, 4 cpu 1gb ram but didn't satisfy as was getting latency spikes in ping sensitive games like CODm, so decided to take it off for a while, though didn't have any voice/video discrepancy

Tried Vanilla stable 21.02 with ath10-ct, wifi doesn't feel extra ordinary, shifted to Kong with default ath10, build 10-10, K 5.4

SETUP
1.unbound (default dns over https custom is uninstalled)
2. Sqm Cake
3. Vsftp
4. ksmbd
3. Hd-idle
4. Aria2

So nothing fancy, i would say, all basic and some basic queries below

FIRST
On my previous Asus router with asuswrt on it, eth* always showed fast_pfifo, but on openwrt showing it as mq, I'm fine with it as far it doesn't hurt anything seriously, i know fast_pfifo patch has been removed from openwrt later on, so i was expected pfifo

SECOND
I'm unbound fan, recursion & snappy and dnssec is fine for me, not interested in stubby etc
On Ax88u , I've used following well set unbound by MARTINEAU

Tried to optimize unbound by coping some of his values, i don't know if it's a good idea or not, EDNS was biggest concern, he set 1472 at ipv4 and default set in openwrt is 1232 (ref 2020 dns day), i tried 1472 but didn't feel it better than 1232, later on noticed in dhcp.conf , option ednspacket_max '1232'
So going for 1472 in DHCP conf and setting 1472 in unbound conf will make a difference? Also attached is conf of unbound I'm using and it certainly needs experts check as if everything's ok?
Unbound Conf

THIRD
Sqm Cake is such a promising idea , don't skip a byte, but only if it works
Isp speeds i get raw as 48.5 down/49.5 up, GPON FTTH , isp modem in bridge mode
In the past, thanks @moeller0 for guiding overhead of 39 for this connection and i believe it's the best value , he's expert
Setting 45/45, piece of cake, interface is pppoe-wan(not eth0.2, as read it recommended somewhere), Ethernet overhead 39, getting very high overlimits, especially egress, don't know why? It does give nice latency numbers in games but surprisingly voip(zoom) audio is not best, lagging and i don't know why it should

I've 2 PCs, 3 Android phones, 10 ip cameras, 2 smart tv , mentioning this is not important

FORTH (EDIT : TODAY'S KONG UPDATE OF PACKAGES INCLUDE CHRONY) Huge thanks @KONG
in Kong available packages, couldn't find chrony and with default sysntp, getting error 5 , always used chrony in the past and feels it's best time sync service and method, is there any way to get chrony in Kong build?

FIFTH & MOST IMPORTANT
My basic propose is to get most robust internet possible, I've read tweaking from ondemand to performance does make a difference but is it at the cost of stability? Further any recommended options for achieving this basic wish on r7800

Thanks to all network gurus here
Especially @KONG @hnyman @moeller0 @everyone

moeller0 · October 21, 2021, 12:46pm

So looking at that I see relative high peak delay (pk_delay, exponentially weighted moving average) for the ingress side, which might indicate that your r7800 might be throttling its CPUs, cake is relatively CPU intensive (and not necessarily raw throughput, but rather it needs CPU cycles with relative low deadlines/latencies, if it can not get them bufferbloat increases. Maybe run a speedtest and look at your CPU load (instructions for how one could do this in the FAQ section on https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details).

immi803 · October 21, 2021, 6:30pm

Sir your mention is right, hope you've noticed overlimits in ingress (as you mentioned egress may be cause)
Here's a glimpse of cpu idle below (a short clip)

Honestly I'm blank what is causing this , please guide

KONG · October 21, 2021, 8:18pm

Try:

config queue 'eth1'
        option interface 'pppoe-wan'
        option debug_logging '0'
        option verbosity '1'
        option linklayer 'none'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option download '42000'
        option upload '42000'
        option enabled '1'

For me the recommended settings for vdsl are not working. My advice skip the recommendations and try different settings. Everey ISP is different and what works for one might not work for the other.

moeller0 · October 21, 2021, 8:57pm

If this (which still results in the kernel's default 14 bytes of overhead) results in less bufferbloat than his original:
option linklayer 'ethernet'
option overhead '39'

then something is really broken with cake on the r7800...

This however, down from his 45 is going to have some effect, unless the issue is cake running out of CPU cycles.
For a discussion how overhead and shaper rate are interdependent and how for a fixed packet size one can make up for the wrong setting of one by the other, see https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm-details#sqmlink_layer_adaptation_tab.

immi803 · October 21, 2021, 9:37pm

I think issue is much more intricate than putting overhead to 0, over estimation doesn't hurt seriously as far as i know,
so many overlimits, is my quantum getting wrong values or what? Should i try some tcmtu? Or mpu?
I intentionally sent short video instead of writing so you can see what services are active and behaving how
Disabling unbound & checking it, didn't help, I'm out of ideas, reading & reading, getting nothing

In the short video clip, you can see speed test at cpu0 (pid xxxxx) and at cpu 1(pid 1), idle does sink below 90%, looks ok to me under speed test
Is it ok @moeller0

Should i try 4.x kernel? Your recommendation?
Things don't look normal to me, are you using it with 5.4? Should i suspect kernel or r7800 ?

KONG · October 22, 2021, 7:07am

In his case I don't think CPU is the problem. I have a 100Mbps line and have no issues. But in my case if I set it to high, then ISP qos/throttling/congestion or something like that kicks in, which I can see whenever I turn off qos, it ramps up to around 105, for a few seconds and then goes down with a bumpy ride and bad throughput and not the best latency. So for my ISP it is important to keep throughput below a certain threshold, otherwise it interferes with my shaping. Maybe I can show you some pics later on.

moeller0 · October 22, 2021, 7:49am

Ah, okay, so you were not really talking about the per-packet-overhead, but about setting the gross shaper rate lower than what a speedtest returns? With that I agree, that really is just to get a starting point (that just happens to work reasonably well on a number of links, but not on all) for searching SQM settings that result in robust and reliable low latencies under load.
And many ISPs use their own traffic shapers, although mostly set such that the goodput achievable in speedtests actually matches the contracted maximum rate (at least in Germany as that is the official interpretation of the national regulatory authority BNetzA). But a few might be willing to play tricks, with a dynamic shaper that allows a higher peak rate for a few seconds before scaling back to a lower permanent rate (on DOCSIS systems that is sometimes marketed as power-boost*), but I would be amazed if any of the big ISPs would play such games.

Yes, I agree, SQM's shaper needs to stay below the relevant bottleneck capacity, and in your case that is the rate available after the initial bump.

I would be delighted, thanks.

*) And that feature in itself is not really bad, as it gives users that have been inactive for some time a mild boost when starting to use the network which can help making the internet feel snappier, but it also has the side-effect of somewhat inflating throughput numbers of typically short speedtests (which is why we recommend to run a speedtest for >= 30 seconds instead of the traditional ~10).

immi803 · October 22, 2021, 10:46am

I think I've tried something non bookish and it worked
As I've symmetrical connection from isp 50/50
I set upload at 40mb with ack-filter and download at 45mb and made my network A-Symmetrical with my own hands
Further used dual-srchost and Dual-dsthost instead of triple isolate
This making of A-Symmetrical and adding ack filter so in upload did the trick for me
Using for last 8 hours and things looking solid

Though I'm not bothered for overlimits in ingress, these are just numbers and i don't have a solution for that

moeller0 · October 22, 2021, 11:48am

Well, this amount of required tuning is not uncommon, after all not all links are the same. Technically I agree 45/40 is not fully symmetric, but a far cry from the 10/1 or even 20/1 ratios some users have to suffer.

Whether ack filtering is a good thing or not depends on your link, but cake's ack-filter is pretty cautious so should not cause problems...

These are not really a problem, (unlike drops or marks, overlimits really just mean that incoming traffic into the qudisc was exccedings it fair capacity share, but that is sort of expected as e.g. TCP flows will try to fill the pipe and scale down their rate to do so, but AQMs then need to reign that in, resulting in the overimits counter increasing, at least that is my understanding).

immi803 · October 22, 2021, 12:15pm

Thank You for being so helpful.

Do you have good number for edns both for unbound as well as dnsmasq (/etc/config/dhcp.conf) ?

EDNS really confused me, as DHCP edns meaning same for unbound edns ? both should match? IPV4 dns google recommendation is 1452+20=1472, whereas 1232 recommended for ipv6, as i've only ipv4 native, should i change default 1232 (dns flag day recommendation 2020) to 1472?

Honestly i have not a good understanding what EDNS do?

moeller0 · October 22, 2021, 12:54pm

Well, nor have I , that is completely outside of my small area of relative expertise, sorry.

noblem · October 22, 2021, 1:53pm

I believe the setting controls the maxium advertised size of a UDP DNS packet supported by your server, so will only have a potential impact on traffic between your clients and router. You should be able to safely increase it to the MTU size of your local network less overhead, although I think it's unlikely you will see any real difference

Normal DNS packets are limited to 512 bytes, which is more than enough in most circumstances. Without eDNS, a DNS response will use TCP if over this limit. eDNS is a mechanism to increase the amount of data that can fit in UDP packet, but mostly only used where the DNS response is dnssec signed...

KONG · October 22, 2021, 11:03pm

Well well. Now it gets interesting. My ISP obviously did some changes. Without any qos set on my router I get a straight line with 100Mbps and the latency is not too bad and guess what. Overhead settings work now like a charm.

This is without qos:

and now with sqm qos and piece of cake set to 100000/32000 overhead 34, check out the latency (in the link):

For the last couple of years, this did never work and I did hundreds of tests whenever I released a new build.

immi803 · October 26, 2021, 4:35am

@kong @moeller0
I want sqm to be last service active in a reboot
Should i use start service? What should be input ?
Or a cron job is preferable?

namidairo · October 26, 2021, 7:32am

You would just control that with the rc.d script?

immi803 · October 26, 2021, 11:47am

i don't know
that's why asked, what's the foolproof way to do that?

moeller0 · October 26, 2021, 5:04pm

Edit /etc/init.s/sqm and replace the 50 with a higher number like 99.

But why do you want to start SQM as late as possible, if I might ask?

immi803 · October 27, 2021, 11:26am

because often sqm applies before reloading of firewall ifup wan, so want to apply sqm after firewall

Secondly, i'm getting following error for sqm, and not sure what module it's looking for actually, what impact of this error?

Tue Oct 26 17:44:42 2021 daemon.err modprobe: failed to find a module named act_ipt

Please guide?

moeller0 · October 27, 2021, 12:00pm

What is the consequence of SQM starting before reloading the firewall?

This really just tells us that we could not load the act_ipt module. The bigger question is why is SQM trying to load that module by default.... as far as I can tell the standard scripts do not actually need that module.... You could edit /usr/lib/sqm/defaults.sh and remove act_ipt from the list of modules to load to squelch that error message.