1462

I have implemented the updates to now just pull the download and upload values directly from the SQM config file. These will be used for the base_ul_rate and base_dl_rate vars.

I also added a check to make sure the enabled option in the SQM config file is set to 1 (enabled) and bail out of the Lua script really early if it is 0.

1463

The README has been updated to reflect the simplified "get up and going" process: https://github.com/Fail-Safe/sqm-autorate/blob/experimental/README.md

One major(ish) change (thanks (!!!) to @CharlesJC's pointers around leveraging UCI) is some refactoring to just use the interface name from the SQM config file. This becomes the ul_if in the code.

From there, I lookup against tc to get the corresponding ifb device for that upload interface. Based on the tc output, the resulting download interface becomes dl_if in the code.

In this way, there should be no need to manually configure these interface values anymore. If one desires, however, they can be overridden directly in the Lua file.

2 Likes
1464

@dlakelan and @_FailSafe might it not be an idea to provide an outline of the latest routine you are working on to benefit from the feedback and wisdom available on this thread? I mean in terms of its operating principles.

I am really keen to keep up collective collaboration because that was the original motivation behind starting this thread. We want one solution to rule them all, set against the various hacks that have been put together without such collective collaboration.

In this connection I am keen to get the shell port working well so that we have a baseline. I'll try enabling that extra output line to see if that sheds light on why it is failing for me.

@CharlesJC, @moeller0, @Lochnair and others, any chance all of you could test the port version and report findings so we can hopefully get it working?

1465

So I am not sure whether tight coupling between sqm and autorate by scraping unadorned sqm configs is robust enough. For example, we have a number of users that do not use an ifb for ingress shaping (sqm does not offer the veth method, but one can simply instantiate the ingress shaper on the egress side of LAN-directed interface). And I could envision that e.g. the hard shaper settings in sqm's config might not be the values people would prefer to set in autorate. (E.g. @Lynx might set the minimum for a static sqm config, but might set something higher as baseline rate in autorate knowing that on bufferbloat autorate will adjust accordingly.
The very least config/sqm would need is a toggle whether to use autorate on a given section or not.
Why? Because a user setting a hard value might be surprised, if tc -s qdisc returns something quite different.
Yes, even if the same user installed autorate and sqm him-/herself....
What I like about storing autorate configuration parametets in /etc/config/sqm however is that it allows to decouple the autorate implementation from its configuration.
SQM itself uses defaults.sh to decouple UCI configuration from its own internal keeping of parameters....

1466

Yes. Perhaps the /etc/config/sqm-autorate file could have options for UL & DL interface names and for the UL and DL rates. If those values are set, the script would use them. If empty, the script could look to the /etc/config/sqm file.

Simple things are simple; complex things are possible.

1 Like
1467

First of all, I am hoping for the same. But the fact of it is that so far I believe only you have provided anything close to feedback on how the port-from-shell experience is going. There is such a deficit of feedback as to whether that branch is on-target or completely missed the mark that I have nothing, other than my own experience, to base the level of "success" as it compares to your shell version.

Secondly, the port-from-shell code is out there in plain sight for everyone to see. No trickery. No "hacks", as you like to say. Even if one is not familiar with the intricacies of Lua syntax, the basic logic around the bandwidth evaluation (conditionals) should be fairly straight-forward to follow. If it isn't working as expected, help me figure out why. Collective collaboration.

I suppose when you make statements like this, you must realize the verbiage you use has a sharp tone to it. Is it fair to assume that is intended?

Again, agreed. In the absence of any significant feedback around the operational equivalence of the port-from-shell branch against the original shell OWD version, the conversation stalled.

Lastly, there was a lot of talk about whether pursuing this effort was worthwhile given potentially infringing on others' work/patent(s). Not sure what the consensus was. Did anyone ever reach a conclusion on that topic?

1 Like
1468

I am still working this week, so will only be able/willing to join the testing-fun next week (I do not expect to be productive over this weekend). It is on my internal todo list, I hope I finally get the "round toit" this christmas :wink:

1469

Duly noted. I would call out that this case is not accounted for even in @Lynx's shell scripts, so we can definitely work toward modifications to allow a one-way sqm-autorate as we iterate.

Fair point and noted, as well. However, could we assume that improvements in the algorithms employed lead to a lessening of the importance of any hard-set values to begin with? In other words, let the SQM config upload/download values be a "guide" to start from, but the algorithms do their own testing and learning to set rates based on actual line data?

Just to be perfectly clear (and a code review will say the same), we are not adding anything new to the SQM config file. As it stands, the code is only consuming that which was already there in the SQM config. The values can still be fully overridden in the Lua file itself if one wants to go the "expert"/advanced route of manually tuning these things.

I was trying to strike a balance, based on outstanding feedback from @richb-hanover-priv and @CharlesJC, between functionality and UX (user experience). Trying to build for the wide range of potential users is not easy, obviously.

1 Like
1470

Yes, but things need to be robust... as well. Currently we assume that cake is running continuously, but due to hotplug, cake instances can and do transiently go away, at which point our script will try to change a non-existing qdisc resulting most likely in an yet unhandled error message. There are a few ways around this, like either starting autorate from inside SQM, or by duplicating sqm's hotplug machiney for autorate and make sure autorate only starts if cake instances exist on the configured interfaces....

My point is, linking autorate and SQM robustly and reliably might be complex in itself. Personally I would postpone tackling this until things run smoother in uncoupled mode. (But then, I have not been adding anything testing or coding wise, so those that do get to make these decisions*)

*) This is IMHO how it should be in open source, those you actually do, get to make decisions, while people on the peanut gallery (like myself) can comment, but should not expect that pure words trump actual deeds.

1 Like
1471

@Lynx @_FailSafe @all,

Thanks for this good work. I agree with the desire to compare the performance (latency-busting capabilities) of these two implementations. The goal as I understand it would be come up with a single reference implementation that we all agree "works pretty well". That gives us a basis for further experimentation and improvement.

Toward that end, I have had good luck with both the original shell script and the port-from-shell branch. I can't say much yet about the current experimental branch - I have sent @_FailSafe a couple private messages with reports of problems I've seen.

Like @moeller0 I will be busy with the holidays (family visits, etc) but will chime in from time to time. But finally...

Take a moment to reflect on how far we've come, and the immense realizations we've had in the last couple months for making CAKE even better... Pat yourselves on the back. Thanks again for all this good work.

3 Likes
1472

Back when I was a beginning programmer, my mentor told me this rule:

  • Amount of effort to make a tool that I can use: X
  • Amount of effort to make the tool work for close colleagues: 3X
  • Amount of effort to make the tool useful everywhere: 9X

So we're somewhere between between 1X and 1.5X in this progression :slight_smile:

Thanks again.

2 Likes
1473

I know, I argue we should ;), something like a string "autorate_function_to_start_in_background" (allowing to pass fancy parameters as arguments in that string, assuming one can pass arguments into lua code)...

I understand, it is just IMHO that that can result in unexpected behavior from an end-user's perspective. Anybody not having spend time recently in the innards of either autorate or SQM.

I am really impressed how you are making progress, and I do not want ot stand in the way. All I want is to enumerate the (potential) issues I see.

2 Likes
1474

That's where I am confused. Help me out here, please :slight_smile: Let's say the autorate functionality is bigger than just OpenWrt... wouldn't that tight coupling with config files that exist only on OpenWrt cause trouble for those on, say, Ubuntu or Fedora distros?

The way the experimental branch sits now, if we aren't on OpenWrt, the UCI module won't be loaded and therefore the Lua file will require hard-set values within the Lua file itself. So it's fairly "portable" at this point.

...
...

I think I've talked myself in a circle here because there are already assumptions for getting values from the SQM config file in place now.

In light of that, if those who own the luci-app-sqm (@tohojo / @dtaht / others?) aren't opposed to further manipulation of the SQM config file, then perhaps that is the best thing to consider.

1 Like
1475

Well, we could simply add that script string variable to sqm's default.sh...

Fair enough. Question, would it be difficult to add another lua file that allows to selectively override parameters fro their defaults? It might be less dangerous than having everybody and their spouse edit the actual lua code file....

Ownership of that is distributed :wink: but it moved from the sqm-repository into one of the OpenWrt repositories and I lost track there (I also think it changed from lua to javascript but might have that one wrong...

But, personally (in spite of having medled with luci-app-sqm from early on) I would make this dependant on @tohojo's buy-in. That said, it is going to be much easier to convince anybody if we whip up a working version that showcases what integration between the two can get us....

1 Like
1476

Indeed, my wife would do nothing good inside the existing Lua code file. :laughing: Let me look into this as you bring up a great point. Part of this might be complicated a bit by the Lua local/global variable scope (as @dlakelan correctly pointed out yesterday). But I'll give it a fair look for sure.

Right-on! I'm officially on Christmas break time-off from work through year-end. I will have a little more time and flexibility to dedicate to testing some things on my live OpenWrt box without the concern of affected video-conferencing. I'll do some tinkering and see what can work.

1 Like
1477

I am trying out the port from shell, so far I am getting better results i would say, i have started to save some of the shell output to txt files in the hope they might be usable. I made a session the other night to see if the port would take full advantage of my line speed at night, and how it would adapt if i was pulling down 2 torrents. It peaked my line at 250 Mbps after i set the base rate to 60,, so it definitely works in the broad strokes it is supposed too. My next plan was, in the hopes it is useful, to try and pick a time with high load on my tower, and then fully saturate my line with several devices active with video streaming and downloads, and then save that session shell output in a txt file . From my initial experiments i can say that i am pretty freaking amazed by SQM in general, i have settled on Piece of cake with: nat dual-dsthost ingress and nat dual-srchost, nothing more fancy. My Rpi4 build right now is also a vanilla Rooter version (snapshot 21.02), that's it, no software offload and no packet steering. I only installed IRQ balance. I would say as a test platform for any output it is very clean, it really doesn't have much funny business. And as it is, it all works like a charm, i have only seen the port stall twice i think... My only problem is that i am really busy and working all through Christmas :laughing: because it is really fun to play with, and it looks to be an amazing feature for any OpenWRT build IMHO. I will PM you Failsafe with some data along the way :slightly_smiling_face:

1478

Would be enough if we would store key value pairs in the config file, does not strictly need to be lua code. Although that would make things slightly more complicated.

1479

Quick clarification, the SQM config file or SQM-autorate config file?

Point of reference--SQM-autorate config file:
https://github.com/Fail-Safe/sqm-autorate/blob/experimental/sqm-autorate.config

1480

Excellent feedback--thank you! Were you previously using the sqm-autorate.sh file before switching to the Lua version? If so, do they seem on-par or is one performing better at this point?

I would say probably better to go ahead and drop any data you collect here in this main thread. That way @Lynx, @moeller0, and others can help analyze the output and make suggestions. I'm not the math brains behind all of this like they are. I just type code :wink:

1 Like
1481

The second.

This looks like UCI notation, no? Which is great for OpenWrt, but not necessarily for other OSs. That is the reason, why in SQM all UCI variables also exist in defaults.sh (and not all variables in defaults.sh are exposed in /etc/config/sqm).