I will do some cleanup of that branch and get the README aligned to the Lua script particulars.
As for experiments, I would defer to @Lynx and @moeller0 as they had an overall vision for goals these scripts were driving toward. The Lua version was simply a port that was intended to match the latest shell version as closely as possible.
I'd go farther and say overall the purpose of the port is to provide a platform where more sophisticated things can be done thanks to a more sophisticated language. The port from shell branch is supposed to work like the shell version. On some other branches we are working out UDP reflectors that work in ipv6 and v4, and I have on my branch an algorithm that keeps a history of known good speeds and jumps to a randomly selected speed when delay is encountered. It tends to have almost no duration of delay as it usually jumps below the available speed threshold in one step.
I'm not clear if things are really ready for public testing but they are close. My branch keeps two csv data files and has a Julia script for plotting what happened.
I concur as I have been running @dlakelan's branch very successfully over the past ~2 days and see great overall results.
Another innovation we are considering is using UDP packets with small time to live and looking for time expired messages as a way to take load off reflectors. Most routers will have a different 3rd hop but at the moment we have only 4 UDP reflectors graciously provided by @_FailSafe and @Lochnair at their expense. These could be consulted less often if the TTL technique works.
I updated the README in the port-from-shell branch and added Lua specifics just now.
I keep repeating this is not a robust and reliable solution in general, intermediate hops are even more likely to employ ICMP rate limiting and deprioritisation... so we already know that there will likely be apparent RTT increases that are independent of the access link. I understand the attraction though, but I do not believe that is a viable route...
Side-note, if the idea is to use UDP packets for OWD measurements, it might be possible to just take one of NTP's frame formats inheriting their design and format debugging that way...
The UDP reflector stuff is working already. We just shove seconds and nanoseconds in a packet, send it off, and the other end appends seconds and nanonseconds.
Agreed that the TTL expiration messages are not something everyone can rely on. But with a proper programming language we can evaluate the reliability in individual cases, and have more sophisticated logic. anyone who has a reliable reflector can then reduce their OWD measurement rate, and then boost it up to a higher rate if the TTL reflector indicates a potential delay. If there is no delay in the TTL data there is no need to hit the OWD reflectors and this is a huge savings in terms of how many people the reflectors can handle.
I do not doubt it works, but I am not convinced it is really optimal... Point is, if you are sending timestamps they should either be 8n the same time coordinate system, or should report the timebase as well. The NTP guys have that solved already, hence the proposal to take inspiration from what the send.
Well, it's intended they're in "Unix epoch".
Mmmh, I still think that going for one of the NTP formats (which is close to unix epoch except with a different anchor and a roll-over in the near future 2036) we would gain a relative large set of potential reflectors (all NTP servers)... (but NTP servers apparently only allow down to 6 seconds sampling period per client, so each client would need to query 6 * 5 = 30 servers if we want new estimates on average every 200ms).
I think OpenWrt already be default enables NTP so we know that the infrastructure sees to be available that can already serve most OpenWrt's at 1/30 or our desired rate....
yes and no, we already know that occasionally they will mostly work well, after all this is what traceroute/mtr use, what we do not know is what happens if the load is increased. Our best estimate however is that we will run into ICMp rate-limiting and/or depriorisation (see from slide 31). these will be hard to detect and work around (especially second by second), especially if all reflectors are collected along the same path...
I am not sure that a latency detector that is supposed to react fast should require too much sophisticated logic 
but obviously it might well turn out that such sophistication is well merited, so I am all for testing different avenues.
I think if we want to do NTP we'll write separate code for that. But I do think it's worthwhile putting a "version number" in our custom packet so we can adjust what we send on the wire.
That is always a good idea, as is to store the originate time stamp and receive and transmit from the remote end; and finally to make sure that the packet size does not differ between the request and the response, so that there is no amplification issue where a small spoofed packet can elicit a considerably larger response sent to an unsuspecting end-point.
Lua Port (Shell Equivalent)
Code:
Instructions for setup:
Changelog:
Huge thanks to @dlakelan and @Lochnair for their Lua contributions and continued development.
Is this the format to which you're referring? https://www.meinbergglobal.com/english/info/ntp-packet.htm
Yes, that would be the idea, with deciding on which to copy size wise (I guess NTPv4 would do) so that the same timestamps sit at the same position such that an extractor script would not care whether a self build reflector has filled the timestamps or a genuine NTP server....
Also this might be helpful. Again the idea is less to do proper NTP but just make it easy to use NTP servers as reflectors. There are orders of magnitude more NTP servers around in the internet than reflectors for any format you might design, so IMHO a new format needs to be massively better that either ICMP timestamps or NTP timestamps to overcome the difference in remote end-point density.
Pretty much finished my normal work for Christmas, and so I am hoping to have more time again for testing very shortly.
Has the port included the same (or at least similar) output lines so that I can graph outputs and compare?
I'm also curious about the idea about historical record of known rates and trying to exploit that. And other clever approaches.
But I'm hopeful we can get the port working at least as well as the shell script as a good baseline (just like our base algorithm!) for the funny business. Then we can test the different ideas against baseline to see whether they actually improve things.
I am eager to get some more pretty graphs on this thread!
Feel free to look, give it a try, and let us know if the output isn't agreeable:
https://github.com/Fail-Safe/sqm-autorate/blob/port-from-shell/sqm-autorate.lua#L17
The current on-wire format looks like this:
-- Type - 1 byte -- Code - 1 byte: -- Checksum - 2 bytes -- Identifier - 2 bytes -- Sequence number - 2 bytes -- Original timestamp - 4 bytes -- Original timestamp (nanoseconds) - 4 bytes -- Received timestamp - 4 bytes -- Received timestamp (nanoseconds) - 4 bytes -- Transmit timestamp - 4 bytes -- Transmit timestamp (nanoseconds) - 4 bytes
I originally started with the ICMP timestamp header, so the Code field isn't really used for anything, we could repurpose that as a version field.
Beyond that it mostly works like ICMP timestamps, the exception being that we don't use time after midnight, it's simply clock_gettime's results converted to network byte-order and put on the wire. I don't remember the reasoning for this, I did that at @dlakelan's request.
The reflector currently doesn't check packet sizes, but we do indeed send the same packet sizes both ways, received and transmit timestamps are filled with nulls when the request is sent.
My point is that unless we expect to take over the world tomorrow with our reflectors, we probably should not try to invent something completely new here, but use something where there is at lest a decent chance of finding "compatible" reflectors, it is fine to use a flag in ICMP-ts or NTP packets to denote timestamps are in a novel format but the offsets and lengths of these fields should probably be identical to those for which a decent number of reflectors already exist.
I am not trying to knock your efforts down here (I am more than impressed), I am just trying to keep this relevant for the near future and I might be insufficiently creative but I do not really see our own reflectors pop up in quantity any time soon, so we will need with what we have available and then it would be nice if a better reflector could be introduced gently, by keeping an already known format.
I admit though, I could be completely wrong here and a new format might be perfectly justified and the right thing to do.
Yes, this is IMHO a good design, network folks somehow dislike protocols that lend themselves to easy amplification attacks, such sensitive souls (joking aside, not becoming part of the problem is the right design)