Yes that's it
Ok all, this morning I'm playing with analyzing data from my own multi-hour run of this LUA data collector... Of course, I have a custom tuned QoS system so I'm finding not much, ping is uncorrelated with bandwidth used or CPU time or anything like that 
I guess this means my QoS system works well.
For those of you who are willing to collect data and contribute it, it would be very useful to see what happens on someone elses link who isn't running SQM or who is running on a much less powerful machine etc. I need this data to be able to design a statistical inference algorithm which will determine SQM speeds that result in good link quality from both a bandwidth and CPU usage perspective.
The discrete different levels of ping times seem to be due to some DNS roundrobin on fbcdn.net or similar. Unrelated to load or cpu but rather to the actual location of the far end host.
Here is a plot of log10(Rate/1Mbps) through time during my example run, just for fun
And Idle vs BW usage for BW over 20Mbps
Forgot to mention service is 600/20 on a docsis 3.1 cable modem.
Edit: no check your google drive
1 Like
was that comment supposed to go in a different thread?
Ah thanks, I see your donated data. Will put a note in the directory about the 600/20 docsis 3.1
@anomeome, it seems that during this data collection it basically never went above 750KB/s, or about 6 Mbps, is that what you'd expect from what you were doing, or is there something wrong with my measurement technique like some kind of clock calibration problem or something?
I would not be expecting that, I just added your lua program and requisite additional packages to my usual image, flashed and let it run. I just ran dslresports now and performance is as expected; unless my ISP had issues overnight. I still have that image on the other partition, I will check its performance.
edit: everything seems fine with image performance at this time. Let me know if you want me to check anything.
Well, it's only looking for 1 second out of every 10 to 20, so I guess if you weren't doing much you wouldn't see high transfer rates. The script itself doesn't induce any load, it doesn't run speed tests etc, it just watches passively to see what happens. But I will look more carefully at your data. Which interface is your WAN? I was assuming eth1.2
Yes wan is eth1.2. That was just an overnight run on an idle device. An artificial load is going to be required, at least on this device, as I really just throw dev work at it and move thnings to the real edge device for my shack when comfortable with the image.
aha, yes thanks, basically it looks idle and it is idle 
I was worrying that the occasional spike to 6Mbps was actually you doing 600Mbps of speedtest or something and my measurement procedure was off by a fixed factor of 100 or something.
So your data isn't too informative in terms of load, but it is informative in terms of being able to run successfully for several hours on a real-world device and extract accurate information. Thanks!
If after this test you feel comfortable running it on your actual edge router for an active time period in your day it could be great to get some real load stats. One of the things I"m trying to determine is whether this kind of passive load watching will be sufficient to discover enough information about people's real links to help them configure SQM and estimate their max bandwidth without saturating the CPU and similar things.