How would I get a list?
I am quite willing to upgrade the router if it's necessary but I would like to future proof myself. What would handle 1Gb with SQM?
Mmmh, maybe just run top -d 1for a while, and look at the %CPU column that should show which process consume considerable CPU time. (or use ps aux to just get an instantaneous list of processes)
That requires some beefy CPU, so either x86 or ARM cores. There are reports here in the forum, that the raspberry Pi4B paired with an TP-Link UE300 USB§ Gigabit ethernet dongle will allow traffic shaping up to a gigabit (if configured correctly), but to replace your current router you will also need a gigabit ethernet switch. So cost wise you are looking at ballpark 100EUR (depending on the Pi4B model you buy, just make sure that you have sufficient cooling).
Cheaper devices like the Brume or turris MOX will probably allow shaping up to around 500-600 Mbps (combined up and downlink) and might be an option at least for your current link.
Please note that gl-inet and turris will ship customised OSes based on OpenWrt, as neither of the three seems to be supported by current stable OpenWrt builds. That said, I expect the RPi4B to be added hopefully with the coming 20.X stable version of OpenWrt (albeit it seems more like a 21.X version by now).
In terms of x86, have you got any suggestions? What kind of hardware would I need on a minimum for 1Gb with cake?
Sorry, no real first hand experience I can offer. Personally I would probably go the raspberry pi4B route first, since it should be considerably cheaper.
But then I limited my 100/40 link down to 49/31 as that was the maximum my old router could achieve, and I value low latency under load more than pure throughput 
1 Like
@moeller0 I've swapped to a Zyxel NBG6617 in the meantime. I hope this will buy me a bit of breathing room.
Mmmh, for your current internet access speed that might do. Could you repeat the top test while torrenting with SQM active, please?
So we see whether that helped?
Had a ping monitor running via Think Broadband for a couple of days.
This to me isn't looking too good but the pings from the router, e.g. to BBC or Google are consistently low even when downloading. I have set the download and upload speeds to be low.
Why does Think Broadband show such a different picture?
I'm starting to wonder if there might be something else deeper wrong here.
I've setup another ping service. The network is completely empty, I have ensured nothing else is going on.
Pings:
Sat, 17 Oct 2020 14:30:41 GMT , 7ms
Sat, 17 Oct 2020 14:30:42 GMT , 7ms
Sat, 17 Oct 2020 14:30:47 GMT , 7ms
Sat, 17 Oct 2020 14:30:52 GMT , 7ms
Sat, 17 Oct 2020 14:30:57 GMT , 7ms
Sat, 17 Oct 2020 14:31:07 GMT , 7ms
Sat, 17 Oct 2020 14:31:11 GMT , 7ms
Sat, 17 Oct 2020 14:31:12 GMT , 7ms
Sat, 17 Oct 2020 14:31:13 GMT , 8ms
Sat, 17 Oct 2020 14:31:19 GMT , 7ms
Sat, 17 Oct 2020 14:31:20 GMT , 7ms
Sat, 17 Oct 2020 14:31:21 GMT , 7ms
Sat, 17 Oct 2020 14:31:25 GMT , 7ms
Sat, 17 Oct 2020 14:31:29 GMT , 7ms
Sat, 17 Oct 2020 14:31:41 GMT , 7ms
Sat, 17 Oct 2020 14:31:42 GMT , 7ms
Sat, 17 Oct 2020 14:31:43 GMT , 7ms
Sat, 17 Oct 2020 14:31:49 GMT , 7ms
Sat, 17 Oct 2020 14:31:50 GMT , 7ms
Sat, 17 Oct 2020 14:31:54 GMT , 7ms
Sat, 17 Oct 2020 14:31:57 GMT , 8ms
Sat, 17 Oct 2020 14:32:00 GMT , 7ms
Sat, 17 Oct 2020 14:32:05 GMT , 8ms
Sat, 17 Oct 2020 14:32:09 GMT , 7ms
Sat, 17 Oct 2020 14:32:10 GMT , 7ms
Sat, 17 Oct 2020 14:32:15 GMT , 7ms
Sat, 17 Oct 2020 14:32:19 GMT , 7ms
Sat, 17 Oct 2020 14:32:20 GMT , 7ms
Sat, 17 Oct 2020 14:32:21 GMT , 7ms
Sat, 17 Oct 2020 14:32:22 GMT , 52ms
Sat, 17 Oct 2020 14:32:29 GMT , 7ms
Sat, 17 Oct 2020 14:32:30 GMT , 7ms
Sat, 17 Oct 2020 14:32:31 GMT , 7ms
Sat, 17 Oct 2020 14:32:49 GMT , 8ms
Sat, 17 Oct 2020 14:32:50 GMT , 7ms
What is that random spike? I had a check, nothing bandwidth intensive was going on then.
The plot thickens...
Sat, 17 Oct 2020 17:00:03 GMT , 7ms
Sat, 17 Oct 2020 17:00:04 GMT , 7ms
Sat, 17 Oct 2020 17:00:05 GMT , 7ms
Sat, 17 Oct 2020 17:00:13 GMT , 13ms
Sat, 17 Oct 2020 17:00:14 GMT , 8ms
Sat, 17 Oct 2020 17:00:18 GMT , 8ms
Sat, 17 Oct 2020 17:00:19 GMT , 7ms
Sat, 17 Oct 2020 17:00:27 GMT , 7ms
Sat, 17 Oct 2020 17:00:28 GMT , 8ms
Sat, 17 Oct 2020 17:00:31 GMT , 8ms
Sat, 17 Oct 2020 17:00:32 GMT , 11ms
Sat, 17 Oct 2020 17:00:37 GMT , 8ms
Sat, 17 Oct 2020 17:00:38 GMT , 10ms
Sat, 17 Oct 2020 17:00:42 GMT , 9ms
Sat, 17 Oct 2020 17:00:43 GMT , 11ms
Sat, 17 Oct 2020 17:00:44 GMT , 8ms
Sat, 17 Oct 2020 17:00:47 GMT , 10ms
Sat, 17 Oct 2020 17:00:52 GMT , 9ms
Sat, 17 Oct 2020 17:01:01 GMT , 7ms
Sat, 17 Oct 2020 17:01:02 GMT , 7ms
Sat, 17 Oct 2020 17:01:03 GMT , 7ms
Sat, 17 Oct 2020 17:01:06 GMT , 7ms
Sat, 17 Oct 2020 17:01:07 GMT , 8ms
Sat, 17 Oct 2020 17:01:08 GMT , 8ms
Sat, 17 Oct 2020 17:01:11 GMT , 7ms
Sat, 17 Oct 2020 17:01:14 GMT , 7ms
Sat, 17 Oct 2020 17:01:17 GMT , 7ms
Sat, 17 Oct 2020 17:01:18 GMT , 7ms
Sat, 17 Oct 2020 17:01:21 GMT , 7ms
Sat, 17 Oct 2020 17:01:28 GMT , 7ms
Sat, 17 Oct 2020 17:01:29 GMT , 7ms
Sat, 17 Oct 2020 17:01:30 GMT , 8ms
Sat, 17 Oct 2020 17:01:38 GMT , 7ms
Sat, 17 Oct 2020 17:01:39 GMT , 7ms
Sat, 17 Oct 2020 17:01:40 GMT , 7ms
Sat, 17 Oct 2020 17:01:41 GMT , 7ms
Sat, 17 Oct 2020 17:01:44 GMT , 9ms
Sat, 17 Oct 2020 17:01:45 GMT , 7ms
Sat, 17 Oct 2020 17:01:46 GMT , 8ms
Sat, 17 Oct 2020 17:01:47 GMT , 7ms
Sat, 17 Oct 2020 17:01:51 GMT , 8ms
Sat, 17 Oct 2020 17:01:55 GMT , 8ms
Sat, 17 Oct 2020 17:01:56 GMT , 8ms
Sat, 17 Oct 2020 17:02:01 GMT , 7ms
Sat, 17 Oct 2020 17:02:11 GMT , 7ms
Sat, 17 Oct 2020 17:02:16 GMT , 7ms
Sat, 17 Oct 2020 17:02:32 GMT , 8ms
Sat, 17 Oct 2020 17:02:33 GMT , 7ms
Sat, 17 Oct 2020 17:02:37 GMT , 7ms
Sat, 17 Oct 2020 17:02:42 GMT , 7ms
Sat, 17 Oct 2020 17:02:43 GMT , 8ms
Sat, 17 Oct 2020 17:02:48 GMT , 8ms
Sat, 17 Oct 2020 17:02:57 GMT , 7ms
Sat, 17 Oct 2020 17:02:58 GMT , 9ms
Sat, 17 Oct 2020 17:02:59 GMT , 8ms
Sat, 17 Oct 2020 17:03:00 GMT , 9ms
Sat, 17 Oct 2020 17:03:01 GMT , 9ms
Sat, 17 Oct 2020 17:03:10 GMT , 7ms
Sat, 17 Oct 2020 17:03:24 GMT , 7ms
Sat, 17 Oct 2020 17:03:25 GMT , 7ms
Sat, 17 Oct 2020 17:03:28 GMT , 7ms
Sat, 17 Oct 2020 17:03:38 GMT , 7ms
Sat, 17 Oct 2020 17:03:39 GMT , 7ms
Sat, 17 Oct 2020 17:03:43 GMT , 7ms
Sat, 17 Oct 2020 17:03:46 GMT , 7ms
Sat, 17 Oct 2020 17:03:47 GMT , 8ms
Sat, 17 Oct 2020 17:03:51 GMT , 7ms
Sat, 17 Oct 2020 17:04:01 GMT , 7ms
Sat, 17 Oct 2020 17:04:06 GMT , 7ms
Sat, 17 Oct 2020 17:04:11 GMT , 8ms
Sat, 17 Oct 2020 17:04:14 GMT , 7ms
Sat, 17 Oct 2020 17:04:15 GMT , 7ms
Sat, 17 Oct 2020 17:04:16 GMT , 7ms
Sat, 17 Oct 2020 17:04:25 GMT , 7ms
Sat, 17 Oct 2020 17:04:26 GMT , 7ms
Sat, 17 Oct 2020 17:04:34 GMT , 7ms
Sat, 17 Oct 2020 17:04:35 GMT , 7ms
Sat, 17 Oct 2020 17:04:36 GMT , 7ms
Sat, 17 Oct 2020 17:04:48 GMT , 8ms
Sat, 17 Oct 2020 17:04:54 GMT , 7ms
Sat, 17 Oct 2020 17:04:55 GMT , 8ms
Sat, 17 Oct 2020 17:04:58 GMT , 7ms
Sat, 17 Oct 2020 17:04:59 GMT , 7ms
Sat, 17 Oct 2020 17:05:00 GMT , 7ms
Sat, 17 Oct 2020 17:05:03 GMT , 8ms
Sat, 17 Oct 2020 17:05:14 GMT , 8ms
Sat, 17 Oct 2020 17:05:24 GMT , 7ms
Sat, 17 Oct 2020 17:05:28 GMT , 7ms
Sat, 17 Oct 2020 17:05:29 GMT , 7ms
Sat, 17 Oct 2020 17:05:30 GMT , 7ms
Sat, 17 Oct 2020 17:05:31 GMT , 7ms
Sat, 17 Oct 2020 17:05:37 GMT , 7ms
Sat, 17 Oct 2020 17:05:42 GMT , 7ms
Sat, 17 Oct 2020 17:05:46 GMT , 8ms
Sat, 17 Oct 2020 17:05:55 GMT , 7ms
Sat, 17 Oct 2020 17:06:04 GMT , 7ms
Sat, 17 Oct 2020 17:06:14 GMT , 7ms
Sat, 17 Oct 2020 17:06:19 GMT , 8ms
Sat, 17 Oct 2020 17:06:20 GMT , 7ms
Sat, 17 Oct 2020 17:06:21 GMT , 7ms
Sat, 17 Oct 2020 17:06:24 GMT , 7ms
Sat, 17 Oct 2020 17:06:25 GMT , 7ms
Sat, 17 Oct 2020 17:06:39 GMT , 7ms
Sat, 17 Oct 2020 17:06:40 GMT , 7ms
Sat, 17 Oct 2020 17:06:44 GMT , 7ms
Sat, 17 Oct 2020 17:06:45 GMT , 7ms
Sat, 17 Oct 2020 17:06:51 GMT , 8ms
Sat, 17 Oct 2020 17:06:54 GMT , 7ms
Sat, 17 Oct 2020 17:06:55 GMT , 7ms
Sat, 17 Oct 2020 17:07:00 GMT , 7ms
Sat, 17 Oct 2020 17:07:05 GMT , 7ms
Sat, 17 Oct 2020 17:07:10 GMT , 7ms
Sat, 17 Oct 2020 17:07:11 GMT , 7ms
Sat, 17 Oct 2020 17:07:12 GMT , 7ms
Sat, 17 Oct 2020 17:07:17 GMT , 8ms
Sat, 17 Oct 2020 17:07:28 GMT , 7ms
Sat, 17 Oct 2020 17:07:32 GMT , 8ms
Sat, 17 Oct 2020 17:07:33 GMT , 8ms
Sat, 17 Oct 2020 17:07:34 GMT , 7ms
Sat, 17 Oct 2020 17:07:35 GMT , 7ms
Sat, 17 Oct 2020 17:07:36 GMT , 7ms
Sat, 17 Oct 2020 17:07:37 GMT , 8ms
Sat, 17 Oct 2020 17:07:41 GMT , 8ms
Sat, 17 Oct 2020 17:07:46 GMT , 26ms
Sat, 17 Oct 2020 17:07:47 GMT , 7ms
Sat, 17 Oct 2020 17:07:51 GMT , 7ms
Sat, 17 Oct 2020 17:07:57 GMT , 7ms
Sat, 17 Oct 2020 17:09:05 GMT , 7ms
Sat, 17 Oct 2020 17:09:08 GMT , 8ms
Sat, 17 Oct 2020 17:09:14 GMT , 7ms
Sat, 17 Oct 2020 17:09:15 GMT , 7ms
Sat, 17 Oct 2020 17:09:16 GMT , 7ms
Sat, 17 Oct 2020 17:09:21 GMT , 7ms
Sat, 17 Oct 2020 17:09:29 GMT , 7ms
Sat, 17 Oct 2020 17:09:30 GMT , 8ms
Sat, 17 Oct 2020 17:09:33 GMT , 8ms
Sat, 17 Oct 2020 17:09:34 GMT , 7ms
Sat, 17 Oct 2020 17:09:35 GMT , 7ms
Sat, 17 Oct 2020 17:09:38 GMT , 7ms
Sat, 17 Oct 2020 17:09:39 GMT , 7ms
Sat, 17 Oct 2020 17:09:43 GMT , 7ms
Sat, 17 Oct 2020 17:09:44 GMT , 7ms
Sat, 17 Oct 2020 17:09:51 GMT , 7ms
Sat, 17 Oct 2020 17:09:52 GMT , 8ms
Sat, 17 Oct 2020 17:09:59 GMT , 7ms
Sat, 17 Oct 2020 17:10:03 GMT , 7ms
Sat, 17 Oct 2020 17:10:04 GMT , 7ms
Sat, 17 Oct 2020 17:10:05 GMT , 8ms
Sat, 17 Oct 2020 17:10:08 GMT , 7ms
Sat, 17 Oct 2020 17:10:16 GMT , 8ms
Sat, 17 Oct 2020 17:10:27 GMT , 7ms
Sat, 17 Oct 2020 17:10:36 GMT , 7ms
Sat, 17 Oct 2020 17:10:40 GMT , 7ms
Sat, 17 Oct 2020 17:10:41 GMT , 7ms
Sat, 17 Oct 2020 17:10:49 GMT , 7ms
Sat, 17 Oct 2020 17:10:54 GMT , 7ms
Sat, 17 Oct 2020 17:10:55 GMT , 7ms
Sat, 17 Oct 2020 17:11:00 GMT , 7ms
Sat, 17 Oct 2020 17:11:01 GMT , 8ms
Sat, 17 Oct 2020 17:11:05 GMT , 7ms
Sat, 17 Oct 2020 17:11:10 GMT , 7ms
Sat, 17 Oct 2020 17:11:11 GMT , 7ms
Sat, 17 Oct 2020 17:11:20 GMT , 7ms
Sat, 17 Oct 2020 17:11:23 GMT , 8ms
Sat, 17 Oct 2020 17:11:24 GMT , 8ms
Sat, 17 Oct 2020 17:11:25 GMT , 7ms
Sat, 17 Oct 2020 17:11:31 GMT , 8ms
Sat, 17 Oct 2020 17:11:32 GMT , 7ms
Sat, 17 Oct 2020 17:11:55 GMT , 7ms
Sat, 17 Oct 2020 17:11:56 GMT , 7ms
Sat, 17 Oct 2020 17:11:57 GMT , 7ms
Sat, 17 Oct 2020 17:11:58 GMT , 7ms
Sat, 17 Oct 2020 17:12:09 GMT , 7ms
Sat, 17 Oct 2020 17:12:12 GMT , 7ms
Sat, 17 Oct 2020 17:12:15 GMT , 7ms
Sat, 17 Oct 2020 17:12:16 GMT , 7ms
Sat, 17 Oct 2020 17:12:17 GMT , 8ms
Sat, 17 Oct 2020 17:12:23 GMT , 8ms
Sat, 17 Oct 2020 17:12:24 GMT , 8ms
Sat, 17 Oct 2020 17:12:25 GMT , 7ms
Sat, 17 Oct 2020 17:12:26 GMT , 8ms
Sat, 17 Oct 2020 17:12:27 GMT , 8ms
Sat, 17 Oct 2020 17:12:28 GMT , 7ms
Sat, 17 Oct 2020 17:12:33 GMT , 7ms
Sat, 17 Oct 2020 17:12:34 GMT , 8ms
Sat, 17 Oct 2020 17:12:35 GMT , 8ms
Sat, 17 Oct 2020 17:12:40 GMT , 7ms
Sat, 17 Oct 2020 17:12:45 GMT , 7ms
Sat, 17 Oct 2020 17:12:46 GMT , 8ms
Sat, 17 Oct 2020 17:12:50 GMT , 7ms
Sat, 17 Oct 2020 17:12:58 GMT , 7ms
Sat, 17 Oct 2020 17:12:59 GMT , 7ms
Sat, 17 Oct 2020 17:13:00 GMT , 7ms
Sat, 17 Oct 2020 17:13:13 GMT , 8ms
Sat, 17 Oct 2020 17:13:14 GMT , 7ms
Sat, 17 Oct 2020 17:13:20 GMT , 7ms
Sat, 17 Oct 2020 17:13:33 GMT , 7ms
Sat, 17 Oct 2020 17:13:34 GMT , 8ms
Sat, 17 Oct 2020 17:13:35 GMT , 7ms
Sat, 17 Oct 2020 17:13:47 GMT , 8ms
Sat, 17 Oct 2020 17:13:50 GMT , 7ms
Sat, 17 Oct 2020 17:13:51 GMT , 8ms
Sat, 17 Oct 2020 17:13:55 GMT , 7ms
Sat, 17 Oct 2020 17:14:03 GMT , 7ms
Sat, 17 Oct 2020 17:14:06 GMT , 7ms
Sat, 17 Oct 2020 17:14:07 GMT , 7ms
Sat, 17 Oct 2020 17:14:15 GMT , 7ms
Sat, 17 Oct 2020 17:14:25 GMT , 7ms
Sat, 17 Oct 2020 17:14:34 GMT , 8ms
Sat, 17 Oct 2020 17:14:35 GMT , 7ms
Sat, 17 Oct 2020 17:14:36 GMT , 7ms
Sat, 17 Oct 2020 17:14:44 GMT , 7ms
Sat, 17 Oct 2020 17:14:49 GMT , 8ms
Sat, 17 Oct 2020 17:15:04 GMT , 7ms
Sat, 17 Oct 2020 17:15:05 GMT , 8ms
Sat, 17 Oct 2020 17:15:09 GMT , 7ms
Sat, 17 Oct 2020 17:15:10 GMT , 7ms
Sat, 17 Oct 2020 17:15:14 GMT , 7ms
Sat, 17 Oct 2020 17:15:15 GMT , 7ms
Sat, 17 Oct 2020 17:15:16 GMT , 7ms
Sat, 17 Oct 2020 17:15:17 GMT , 7ms
Sat, 17 Oct 2020 17:15:25 GMT , 7ms
Sat, 17 Oct 2020 17:15:28 GMT , 8ms
Sat, 17 Oct 2020 17:15:34 GMT , 7ms
Sat, 17 Oct 2020 17:15:35 GMT , 7ms
Sat, 17 Oct 2020 17:15:36 GMT , 7ms
Sat, 17 Oct 2020 17:15:39 GMT , 7ms
Sat, 17 Oct 2020 17:15:40 GMT , 8ms
Sat, 17 Oct 2020 17:15:41 GMT , 7ms
Sat, 17 Oct 2020 17:15:46 GMT , 8ms
Sat, 17 Oct 2020 17:15:53 GMT , 7ms
Sat, 17 Oct 2020 17:15:54 GMT , 7ms
Sat, 17 Oct 2020 17:15:55 GMT , 7ms
Sat, 17 Oct 2020 17:15:56 GMT , 7ms
Sat, 17 Oct 2020 17:15:57 GMT , 7ms
Sat, 17 Oct 2020 17:16:03 GMT , 7ms
Sat, 17 Oct 2020 17:16:04 GMT , 7ms
Sat, 17 Oct 2020 17:16:05 GMT , 8ms
Sat, 17 Oct 2020 17:16:06 GMT , 7ms
Sat, 17 Oct 2020 17:16:11 GMT , 7ms
Sat, 17 Oct 2020 17:16:14 GMT , 7ms
Sat, 17 Oct 2020 17:16:15 GMT , 7ms
Sat, 17 Oct 2020 17:16:16 GMT , 7ms
Sat, 17 Oct 2020 17:16:20 GMT , 7ms
Sat, 17 Oct 2020 17:16:21 GMT , 8ms
Sat, 17 Oct 2020 17:16:25 GMT , 7ms
Sat, 17 Oct 2020 17:16:26 GMT , 7ms
Sat, 17 Oct 2020 17:16:35 GMT , 7ms
Sat, 17 Oct 2020 17:16:36 GMT , 7ms
Sat, 17 Oct 2020 17:16:40 GMT , 7ms
Sat, 17 Oct 2020 17:16:45 GMT , 7ms
Sat, 17 Oct 2020 17:16:50 GMT , 7ms
Sat, 17 Oct 2020 17:16:51 GMT , 7ms
Sat, 17 Oct 2020 17:16:52 GMT , 7ms
Sat, 17 Oct 2020 17:16:56 GMT , 8ms
Sat, 17 Oct 2020 17:16:57 GMT , 7ms
Sat, 17 Oct 2020 17:17:15 GMT , 7ms
Sat, 17 Oct 2020 17:17:16 GMT , 7ms
Sat, 17 Oct 2020 17:17:17 GMT , 7ms
Sat, 17 Oct 2020 17:17:18 GMT , 7ms
Sat, 17 Oct 2020 17:17:25 GMT , 7ms
Sat, 17 Oct 2020 17:17:26 GMT , 7ms
Sat, 17 Oct 2020 17:17:27 GMT , 7ms
Sat, 17 Oct 2020 17:17:37 GMT , 8ms
Sat, 17 Oct 2020 17:17:41 GMT , 7ms
Sat, 17 Oct 2020 17:17:44 GMT , 7ms
Sat, 17 Oct 2020 17:17:48 GMT , 7ms
Sat, 17 Oct 2020 17:17:49 GMT , 7ms
Sat, 17 Oct 2020 17:17:59 GMT , 8ms
Sat, 17 Oct 2020 17:18:00 GMT , 7ms
Sat, 17 Oct 2020 17:18:07 GMT , 8ms
Sat, 17 Oct 2020 17:18:08 GMT , 8ms
Sat, 17 Oct 2020 17:18:13 GMT , 9ms
Sat, 17 Oct 2020 17:18:16 GMT , 10ms
Sat, 17 Oct 2020 17:18:20 GMT , 7ms
Sat, 17 Oct 2020 17:18:21 GMT , 9ms
Sat, 17 Oct 2020 17:18:36 GMT , 7ms
Sat, 17 Oct 2020 17:18:37 GMT , 7ms
Sat, 17 Oct 2020 17:18:43 GMT , 7ms
Sat, 17 Oct 2020 17:18:48 GMT , 7ms
Sat, 17 Oct 2020 17:18:51 GMT , 7ms
Sat, 17 Oct 2020 17:18:52 GMT , 8ms
Sat, 17 Oct 2020 17:19:01 GMT , 7ms
Sat, 17 Oct 2020 17:19:04 GMT , 8ms
Sat, 17 Oct 2020 17:19:05 GMT , 7ms
Sat, 17 Oct 2020 17:19:06 GMT , 7ms
Sat, 17 Oct 2020 17:19:16 GMT , 7ms
Sat, 17 Oct 2020 17:19:20 GMT , 8ms
Sat, 17 Oct 2020 17:19:21 GMT , 7ms
Sat, 17 Oct 2020 17:19:24 GMT , 7ms
Sat, 17 Oct 2020 17:19:25 GMT , 8ms
Sat, 17 Oct 2020 17:19:26 GMT , 8ms
Sat, 17 Oct 2020 17:19:37 GMT , 7ms
Sat, 17 Oct 2020 17:19:42 GMT , 7ms
Sat, 17 Oct 2020 17:19:43 GMT , 7ms
Sat, 17 Oct 2020 17:19:44 GMT , 8ms
Sat, 17 Oct 2020 17:19:47 GMT , 8ms
Sat, 17 Oct 2020 17:19:54 GMT , 7ms
Sat, 17 Oct 2020 17:19:57 GMT , 8ms
Sat, 17 Oct 2020 17:20:05 GMT , 7ms
Sat, 17 Oct 2020 17:20:06 GMT , 8ms
Sat, 17 Oct 2020 17:20:09 GMT , 7ms
Sat, 17 Oct 2020 17:20:21 GMT , 8ms
Sat, 17 Oct 2020 17:20:22 GMT , 7ms
Sat, 17 Oct 2020 17:20:27 GMT , 7ms
Sat, 17 Oct 2020 17:20:28 GMT , 7ms
Sat, 17 Oct 2020 17:20:35 GMT , 7ms
Sat, 17 Oct 2020 17:20:47 GMT , 7ms
Sat, 17 Oct 2020 17:20:54 GMT , 7ms
Sat, 17 Oct 2020 17:21:07 GMT , 7ms
Sat, 17 Oct 2020 17:21:08 GMT , 7ms
Sat, 17 Oct 2020 17:21:09 GMT , 7ms
Sat, 17 Oct 2020 17:21:10 GMT , 7ms
Sat, 17 Oct 2020 17:21:21 GMT , 7ms
Sat, 17 Oct 2020 17:21:22 GMT , 7ms
Sat, 17 Oct 2020 17:21:25 GMT , 7ms
Sat, 17 Oct 2020 17:21:28 GMT , 7ms
Sat, 17 Oct 2020 17:21:29 GMT , 8ms
Sat, 17 Oct 2020 17:21:30 GMT , 8ms
Sat, 17 Oct 2020 17:21:31 GMT , 7ms
Sat, 17 Oct 2020 17:21:36 GMT , 7ms
Sat, 17 Oct 2020 17:21:43 GMT , 7ms
Sat, 17 Oct 2020 17:21:44 GMT , 7ms
Sat, 17 Oct 2020 17:21:45 GMT , 7ms
Sat, 17 Oct 2020 17:21:50 GMT , 7ms
Sat, 17 Oct 2020 17:21:58 GMT , 7ms
Sat, 17 Oct 2020 17:21:59 GMT , 7ms
Sat, 17 Oct 2020 17:22:04 GMT , 7ms
Sat, 17 Oct 2020 17:22:05 GMT , 7ms
Sat, 17 Oct 2020 17:22:06 GMT , 7ms
Sat, 17 Oct 2020 17:22:07 GMT , 7ms
Sat, 17 Oct 2020 17:22:08 GMT , 7ms
Sat, 17 Oct 2020 17:22:12 GMT , 8ms
Sat, 17 Oct 2020 17:22:13 GMT , 7ms
Sat, 17 Oct 2020 17:22:18 GMT , 7ms
Sat, 17 Oct 2020 17:22:19 GMT , 7ms
Sat, 17 Oct 2020 17:22:20 GMT , 8ms
Sat, 17 Oct 2020 17:22:23 GMT , 7ms
Sat, 17 Oct 2020 17:22:24 GMT , 7ms
Sat, 17 Oct 2020 17:22:25 GMT , 7ms
Sat, 17 Oct 2020 17:22:26 GMT , 8ms
Sat, 17 Oct 2020 17:22:30 GMT , 7ms
Sat, 17 Oct 2020 17:22:35 GMT , 7ms
Sat, 17 Oct 2020 17:22:40 GMT , 7ms
Sat, 17 Oct 2020 17:22:41 GMT , 7ms
Sat, 17 Oct 2020 17:22:50 GMT , 7ms
Sat, 17 Oct 2020 17:22:51 GMT , 7ms
Sat, 17 Oct 2020 17:22:55 GMT , 7ms
Sat, 17 Oct 2020 17:22:59 GMT , 7ms
Sat, 17 Oct 2020 17:23:04 GMT , 7ms
Sat, 17 Oct 2020 17:23:05 GMT , 7ms
Sat, 17 Oct 2020 17:23:10 GMT , 8ms
Sat, 17 Oct 2020 17:23:11 GMT , 7ms
Sat, 17 Oct 2020 17:23:16 GMT , 8ms
Sat, 17 Oct 2020 17:23:21 GMT , 7ms
Sat, 17 Oct 2020 17:23:26 GMT , 7ms
Sat, 17 Oct 2020 17:23:29 GMT , 7ms
Sat, 17 Oct 2020 17:23:30 GMT , 7ms
Sat, 17 Oct 2020 17:23:31 GMT , 7ms
Sat, 17 Oct 2020 17:23:36 GMT , 7ms
Sat, 17 Oct 2020 17:23:45 GMT , 7ms
Sat, 17 Oct 2020 17:23:53 GMT , 7ms
Sat, 17 Oct 2020 17:24:02 GMT , 10ms
Sat, 17 Oct 2020 17:24:11 GMT , 8ms
Sat, 17 Oct 2020 17:24:12 GMT , 9ms
Sat, 17 Oct 2020 17:24:17 GMT , 7ms
Sat, 17 Oct 2020 17:24:22 GMT , 7ms
Sat, 17 Oct 2020 17:24:27 GMT , 8ms
Sat, 17 Oct 2020 17:24:34 GMT , 8ms
Sat, 17 Oct 2020 17:24:44 GMT , 7ms
Sat, 17 Oct 2020 17:24:51 GMT , 7ms
Sat, 17 Oct 2020 17:24:54 GMT , 7ms
Sat, 17 Oct 2020 17:24:55 GMT , 7ms
Sat, 17 Oct 2020 17:24:56 GMT , 8ms
Sat, 17 Oct 2020 17:24:59 GMT , 7ms
Sat, 17 Oct 2020 17:25:04 GMT , 7ms
Sat, 17 Oct 2020 17:25:20 GMT , 7ms
Sat, 17 Oct 2020 17:25:24 GMT , 7ms
Sat, 17 Oct 2020 17:25:29 GMT , 8ms
Sat, 17 Oct 2020 17:25:30 GMT , 7ms
Sat, 17 Oct 2020 17:25:31 GMT , 7ms
Sat, 17 Oct 2020 17:25:34 GMT , 8ms
Sat, 17 Oct 2020 17:25:35 GMT , 8ms
Sat, 17 Oct 2020 17:25:47 GMT , 7ms
Sat, 17 Oct 2020 17:25:48 GMT , 8ms
Sat, 17 Oct 2020 17:25:51 GMT , 7ms
Sat, 17 Oct 2020 17:25:54 GMT , 7ms
Sat, 17 Oct 2020 17:25:57 GMT , 7ms
Sat, 17 Oct 2020 17:26:04 GMT , 7ms
Sat, 17 Oct 2020 17:26:05 GMT , 7ms
Sat, 17 Oct 2020 17:26:09 GMT , 7ms
Sat, 17 Oct 2020 17:26:12 GMT , 7ms
Sat, 17 Oct 2020 17:26:13 GMT , 7ms
Sat, 17 Oct 2020 17:26:14 GMT , 7ms
Sat, 17 Oct 2020 17:26:15 GMT , 7ms
Sat, 17 Oct 2020 17:26:16 GMT , 7ms
Sat, 17 Oct 2020 17:26:25 GMT , 7ms
Sat, 17 Oct 2020 17:26:33 GMT , 7ms
Sat, 17 Oct 2020 17:26:41 GMT , 7ms
Sat, 17 Oct 2020 17:26:42 GMT , 7ms
Sat, 17 Oct 2020 17:26:43 GMT , 7ms
Sat, 17 Oct 2020 17:26:48 GMT , 7ms
Sat, 17 Oct 2020 17:26:49 GMT , 7ms
Sat, 17 Oct 2020 17:26:50 GMT , 7ms
Sat, 17 Oct 2020 17:26:54 GMT , 7ms
Sat, 17 Oct 2020 17:26:59 GMT , 7ms
Sat, 17 Oct 2020 17:27:03 GMT , 7ms
Sat, 17 Oct 2020 17:27:08 GMT , 7ms
Sat, 17 Oct 2020 17:27:16 GMT , 7ms
Sat, 17 Oct 2020 17:27:27 GMT , 7ms
Sat, 17 Oct 2020 17:27:33 GMT , 7ms
Sat, 17 Oct 2020 17:27:34 GMT , 8ms
Sat, 17 Oct 2020 17:27:41 GMT , 7ms
Sat, 17 Oct 2020 17:27:42 GMT , 8ms
Sat, 17 Oct 2020 17:27:43 GMT , 7ms
Sat, 17 Oct 2020 17:27:46 GMT , 7ms
Sat, 17 Oct 2020 17:27:47 GMT , 7ms
Sat, 17 Oct 2020 17:27:50 GMT , 7ms
Sat, 17 Oct 2020 17:27:55 GMT , 7ms
Sat, 17 Oct 2020 17:27:59 GMT , 7ms
Sat, 17 Oct 2020 17:28:02 GMT , 7ms
Sat, 17 Oct 2020 17:28:07 GMT , 7ms
Sat, 17 Oct 2020 17:28:11 GMT , 7ms
Sat, 17 Oct 2020 17:28:12 GMT , 8ms
Sat, 17 Oct 2020 17:28:16 GMT , 7ms
Sat, 17 Oct 2020 17:28:19 GMT , 7ms
Sat, 17 Oct 2020 17:28:20 GMT , 8ms
Sat, 17 Oct 2020 17:28:21 GMT , 7ms
Sat, 17 Oct 2020 17:28:30 GMT , 7ms
Sat, 17 Oct 2020 17:28:36 GMT , 7ms
Sat, 17 Oct 2020 17:28:39 GMT , 9ms
Sat, 17 Oct 2020 17:28:40 GMT , 8ms
Sat, 17 Oct 2020 17:28:41 GMT , 7ms
Sat, 17 Oct 2020 17:28:44 GMT , 7ms
Sat, 17 Oct 2020 17:28:45 GMT , 7ms
Sat, 17 Oct 2020 17:29:00 GMT , 19ms
Sat, 17 Oct 2020 17:29:01 GMT , 7ms
Sat, 17 Oct 2020 17:29:06 GMT , 7ms
Sat, 17 Oct 2020 17:29:15 GMT , 8ms
Sat, 17 Oct 2020 17:29:16 GMT , 7ms
Sat, 17 Oct 2020 17:29:26 GMT , 7ms
Sat, 17 Oct 2020 17:29:34 GMT , 7ms
Sat, 17 Oct 2020 17:29:37 GMT , 7ms
Sat, 17 Oct 2020 17:29:41 GMT , 7ms
Sat, 17 Oct 2020 17:29:42 GMT , 7ms
Sat, 17 Oct 2020 17:29:43 GMT , 7ms
Sat, 17 Oct 2020 17:29:44 GMT , 7ms
Sat, 17 Oct 2020 17:30:00 GMT , 8ms
Sat, 17 Oct 2020 17:30:10 GMT , 8ms
Sat, 17 Oct 2020 17:30:13 GMT , 7ms
Sat, 17 Oct 2020 17:30:18 GMT , 7ms
Sat, 17 Oct 2020 17:30:19 GMT , 7ms
Sat, 17 Oct 2020 17:30:20 GMT , 7ms
Sat, 17 Oct 2020 17:30:24 GMT , 7ms
Sat, 17 Oct 2020 17:30:25 GMT , 7ms
Sat, 17 Oct 2020 17:30:29 GMT , 7ms
Sat, 17 Oct 2020 17:30:30 GMT , 7ms
Sat, 17 Oct 2020 17:30:35 GMT , 8ms
Sat, 17 Oct 2020 17:30:40 GMT , 7ms
Sat, 17 Oct 2020 17:30:47 GMT , 7ms
Sat, 17 Oct 2020 17:30:51 GMT , 9ms
Sat, 17 Oct 2020 17:30:52 GMT , 7ms
Sat, 17 Oct 2020 17:30:53 GMT , 7ms
Sat, 17 Oct 2020 17:30:54 GMT , 7ms
Sat, 17 Oct 2020 17:31:01 GMT , 8ms
Sat, 17 Oct 2020 17:31:02 GMT , 8ms
Sat, 17 Oct 2020 17:31:03 GMT , 7ms
Sat, 17 Oct 2020 17:31:04 GMT , 7ms
Sat, 17 Oct 2020 17:31:05 GMT , 7ms
Sat, 17 Oct 2020 17:31:06 GMT , 7ms
Sat, 17 Oct 2020 17:31:10 GMT , 7ms
Sat, 17 Oct 2020 17:31:11 GMT , 8ms
Sat, 17 Oct 2020 17:31:36 GMT , 8ms
Sat, 17 Oct 2020 17:31:41 GMT , 7ms
Sat, 17 Oct 2020 17:31:46 GMT , 8ms
Sat, 17 Oct 2020 17:31:56 GMT , 7ms
Sat, 17 Oct 2020 17:31:57 GMT , 7ms
Sat, 17 Oct 2020 17:32:01 GMT , 8ms
Sat, 17 Oct 2020 17:32:05 GMT , 7ms
Sat, 17 Oct 2020 17:32:06 GMT , 7ms
Sat, 17 Oct 2020 17:32:12 GMT , 7ms
Sat, 17 Oct 2020 17:32:13 GMT , 7ms
Sat, 17 Oct 2020 17:32:14 GMT , 7ms
Sat, 17 Oct 2020 17:32:15 GMT , 8ms
Sat, 17 Oct 2020 17:32:16 GMT , 9ms
Sat, 17 Oct 2020 17:32:17 GMT , 7ms
Sat, 17 Oct 2020 17:32:27 GMT , 7ms
Sat, 17 Oct 2020 17:32:30 GMT , 7ms
Sat, 17 Oct 2020 17:32:33 GMT , 9ms
Sat, 17 Oct 2020 17:32:34 GMT , 7ms
Sat, 17 Oct 2020 17:32:37 GMT , 7ms
Sat, 17 Oct 2020 17:32:47 GMT , 7ms
Sat, 17 Oct 2020 17:32:52 GMT , 7ms
Sat, 17 Oct 2020 17:32:53 GMT , 7ms
Sat, 17 Oct 2020 17:32:54 GMT , 9ms
Sat, 17 Oct 2020 17:32:57 GMT , 11ms
Sat, 17 Oct 2020 17:33:00 GMT , 7ms
Sat, 17 Oct 2020 17:33:03 GMT , 7ms
Sat, 17 Oct 2020 17:33:12 GMT , 7ms
Sat, 17 Oct 2020 17:33:13 GMT , 7ms
Sat, 17 Oct 2020 17:33:14 GMT , 8ms
Sat, 17 Oct 2020 17:33:15 GMT , 10ms
Sat, 17 Oct 2020 17:33:24 GMT , 7ms
Sat, 17 Oct 2020 17:33:25 GMT , 8ms
Sat, 17 Oct 2020 17:33:28 GMT , 7ms
Sat, 17 Oct 2020 17:33:29 GMT , 8ms
Sat, 17 Oct 2020 17:33:34 GMT , 7ms
Sat, 17 Oct 2020 17:33:40 GMT , 8ms
Sat, 17 Oct 2020 17:33:41 GMT , 8ms
Sat, 17 Oct 2020 17:33:42 GMT , 7ms
Sat, 17 Oct 2020 17:33:45 GMT , 7ms
Sat, 17 Oct 2020 17:33:46 GMT , 7ms
Sat, 17 Oct 2020 17:33:49 GMT , 11ms
Sat, 17 Oct 2020 17:34:02 GMT , 7ms
Sat, 17 Oct 2020 17:34:12 GMT , 7ms
Sat, 17 Oct 2020 17:34:16 GMT , 7ms
Sat, 17 Oct 2020 17:34:17 GMT , 7ms
Sat, 17 Oct 2020 17:34:18 GMT , 7ms
Sat, 17 Oct 2020 17:34:27 GMT , 8ms
Sat, 17 Oct 2020 17:34:30 GMT , 7ms
Sat, 17 Oct 2020 17:34:31 GMT , 7ms
Sat, 17 Oct 2020 17:34:32 GMT , 7ms
Sat, 17 Oct 2020 17:34:42 GMT , 7ms
Sat, 17 Oct 2020 17:34:45 GMT , 7ms
Sat, 17 Oct 2020 17:35:06 GMT , 7ms
Sat, 17 Oct 2020 17:35:11 GMT , 8ms
Sat, 17 Oct 2020 17:35:14 GMT , 7ms
Sat, 17 Oct 2020 17:35:22 GMT , 7ms
Sat, 17 Oct 2020 17:35:23 GMT , 7ms
Sat, 17 Oct 2020 17:35:24 GMT , 7ms
Sat, 17 Oct 2020 17:35:25 GMT , 7ms
Sat, 17 Oct 2020 17:35:32 GMT , 7ms
Sat, 17 Oct 2020 17:35:33 GMT , 7ms
Sat, 17 Oct 2020 17:35:34 GMT , 7ms
Sat, 17 Oct 2020 17:35:37 GMT , 7ms
Sat, 17 Oct 2020 17:35:38 GMT , 7ms
Sat, 17 Oct 2020 17:35:39 GMT , 7ms
Sat, 17 Oct 2020 17:35:42 GMT , 7ms
Sat, 17 Oct 2020 17:35:47 GMT , 7ms
Sat, 17 Oct 2020 17:35:52 GMT , 7ms
Sat, 17 Oct 2020 17:35:53 GMT , 8ms
Sat, 17 Oct 2020 17:35:54 GMT , 7ms
Sat, 17 Oct 2020 17:35:58 GMT , 7ms
Sat, 17 Oct 2020 17:35:59 GMT , 7ms
Sat, 17 Oct 2020 17:36:04 GMT , 7ms
Sat, 17 Oct 2020 17:36:09 GMT , 7ms
Sat, 17 Oct 2020 17:36:10 GMT , 7ms
Sat, 17 Oct 2020 17:36:11 GMT , 7ms
Sat, 17 Oct 2020 17:36:22 GMT , 7ms
Sat, 17 Oct 2020 17:36:27 GMT , 7ms
Sat, 17 Oct 2020 17:36:30 GMT , 7ms
Sat, 17 Oct 2020 17:36:31 GMT , 7ms
Sat, 17 Oct 2020 17:36:32 GMT , 7ms
Sat, 17 Oct 2020 17:36:36 GMT , 7ms
Sat, 17 Oct 2020 17:36:37 GMT , 7ms
Sat, 17 Oct 2020 17:36:42 GMT , 7ms
Sat, 17 Oct 2020 17:36:48 GMT , 8ms
Sat, 17 Oct 2020 17:37:05 GMT , 7ms
Sat, 17 Oct 2020 17:37:06 GMT , 7ms
Sat, 17 Oct 2020 17:37:07 GMT , 7ms
Sat, 17 Oct 2020 17:37:18 GMT , 7ms
Sat, 17 Oct 2020 17:37:19 GMT , 8ms
Sat, 17 Oct 2020 17:37:42 GMT , 7ms
Sat, 17 Oct 2020 17:37:47 GMT , 7ms
Sat, 17 Oct 2020 17:37:51 GMT , 7ms
Sat, 17 Oct 2020 17:37:55 GMT , 7ms
Sat, 17 Oct 2020 17:37:56 GMT , 7ms
Sat, 17 Oct 2020 17:37:59 GMT , 7ms
Sat, 17 Oct 2020 17:38:00 GMT , 7ms
Sat, 17 Oct 2020 17:38:01 GMT , 7ms
Sat, 17 Oct 2020 17:38:02 GMT , 8ms
Sat, 17 Oct 2020 17:38:03 GMT , 7ms
Sat, 17 Oct 2020 17:38:11 GMT , 7ms
Sat, 17 Oct 2020 17:38:12 GMT , 7ms
Sat, 17 Oct 2020 17:38:15 GMT , 8ms
Sat, 17 Oct 2020 17:38:25 GMT , 7ms
Sat, 17 Oct 2020 17:38:29 GMT , 7ms
Sat, 17 Oct 2020 17:38:30 GMT , 7ms
Sat, 17 Oct 2020 17:38:34 GMT , 7ms
Sat, 17 Oct 2020 17:38:41 GMT , 8ms
Sat, 17 Oct 2020 17:38:51 GMT , 7ms
Sat, 17 Oct 2020 17:38:52 GMT , 7ms
Sat, 17 Oct 2020 17:38:56 GMT , 7ms
Sat, 17 Oct 2020 17:39:01 GMT , 7ms
Sat, 17 Oct 2020 17:39:02 GMT , 7ms
Sat, 17 Oct 2020 17:39:06 GMT , 7ms
Sat, 17 Oct 2020 17:39:07 GMT , 7ms
Sat, 17 Oct 2020 17:39:08 GMT , 7ms
Sat, 17 Oct 2020 17:39:18 GMT , 7ms
Sat, 17 Oct 2020 17:39:19 GMT , 7ms
Sat, 17 Oct 2020 17:39:23 GMT , 7ms
Sat, 17 Oct 2020 17:39:24 GMT , 8ms
Sat, 17 Oct 2020 17:39:33 GMT , 8ms
Sat, 17 Oct 2020 17:39:38 GMT , 7ms
Sat, 17 Oct 2020 17:39:55 GMT , 7ms
Sat, 17 Oct 2020 17:40:08 GMT , 7ms
Sat, 17 Oct 2020 17:40:09 GMT , 7ms
Sat, 17 Oct 2020 17:40:22 GMT , 8ms
Sat, 17 Oct 2020 17:40:23 GMT , 7ms
Sat, 17 Oct 2020 17:40:27 GMT , 7ms
Sat, 17 Oct 2020 17:40:28 GMT , 7ms
Sat, 17 Oct 2020 17:40:29 GMT , 7ms
Sat, 17 Oct 2020 17:40:32 GMT , 8ms
Sat, 17 Oct 2020 17:40:42 GMT , 7ms
Sat, 17 Oct 2020 17:40:43 GMT , 7ms
Sat, 17 Oct 2020 17:40:44 GMT , 7ms
Sat, 17 Oct 2020 17:40:45 GMT , 8ms
Sat, 17 Oct 2020 17:40:49 GMT , 8ms
Sat, 17 Oct 2020 17:40:54 GMT , 8ms
Sat, 17 Oct 2020 17:40:57 GMT , 7ms
Sat, 17 Oct 2020 17:40:58 GMT , 7ms
Sat, 17 Oct 2020 17:41:02 GMT , 7ms
Sat, 17 Oct 2020 17:41:06 GMT , 7ms
Sat, 17 Oct 2020 17:41:07 GMT , 7ms
Sat, 17 Oct 2020 17:41:08 GMT , 7ms
Sat, 17 Oct 2020 17:41:17 GMT , 8ms
Sat, 17 Oct 2020 17:41:20 GMT , 7ms
Sat, 17 Oct 2020 17:41:26 GMT , 7ms
Sat, 17 Oct 2020 17:41:27 GMT , 7ms
Sat, 17 Oct 2020 17:41:31 GMT , 8ms
Sat, 17 Oct 2020 17:41:32 GMT , 8ms
Sat, 17 Oct 2020 17:41:35 GMT , 7ms
Sat, 17 Oct 2020 17:41:46 GMT , 7ms
Sat, 17 Oct 2020 17:41:47 GMT , 7ms
Sat, 17 Oct 2020 17:41:48 GMT , 7ms
Sat, 17 Oct 2020 17:41:53 GMT , 7ms
Sat, 17 Oct 2020 17:41:54 GMT , 7ms
Sat, 17 Oct 2020 17:41:58 GMT , 7ms
Sat, 17 Oct 2020 17:42:01 GMT , 7ms
Sat, 17 Oct 2020 17:42:04 GMT , 7ms
Sat, 17 Oct 2020 17:42:05 GMT , 7ms
Sat, 17 Oct 2020 17:42:08 GMT , 9ms
Sat, 17 Oct 2020 17:42:09 GMT , 7ms
Sat, 17 Oct 2020 17:42:10 GMT , 7ms
Sat, 17 Oct 2020 17:42:14 GMT , 7ms
Sat, 17 Oct 2020 17:42:15 GMT , 7ms
Sat, 17 Oct 2020 17:42:32 GMT , 7ms
Sat, 17 Oct 2020 17:42:36 GMT , 8ms
Sat, 17 Oct 2020 17:42:37 GMT , 7ms
Sat, 17 Oct 2020 17:42:40 GMT , 7ms
Sat, 17 Oct 2020 17:42:41 GMT , 7ms
Sat, 17 Oct 2020 17:42:49 GMT , 7ms
Sat, 17 Oct 2020 17:42:50 GMT , 7ms
Sat, 17 Oct 2020 17:42:55 GMT , 7ms
Sat, 17 Oct 2020 17:42:56 GMT , 8ms
Sat, 17 Oct 2020 17:42:57 GMT , 8ms
Sat, 17 Oct 2020 17:43:18 GMT , 8ms
Sat, 17 Oct 2020 17:43:19 GMT , 8ms
Sat, 17 Oct 2020 17:43:22 GMT , 7ms
Sat, 17 Oct 2020 17:43:25 GMT , 7ms
Sat, 17 Oct 2020 17:43:26 GMT , 7ms
Sat, 17 Oct 2020 17:43:35 GMT , 7ms
Sat, 17 Oct 2020 17:43:36 GMT , 7ms
Sat, 17 Oct 2020 17:43:40 GMT , 7ms
Sat, 17 Oct 2020 17:43:41 GMT , 7ms
Sat, 17 Oct 2020 17:43:51 GMT , 8ms
Sat, 17 Oct 2020 17:43:52 GMT , 7ms
Sat, 17 Oct 2020 17:43:53 GMT , 7ms
Sat, 17 Oct 2020 17:43:56 GMT , 7ms
Sat, 17 Oct 2020 17:44:01 GMT , 7ms
Sat, 17 Oct 2020 17:44:06 GMT , 7ms
Sat, 17 Oct 2020 17:44:10 GMT , 7ms
Sat, 17 Oct 2020 17:44:14 GMT , 7ms
Sat, 17 Oct 2020 17:44:15 GMT , 7ms
Sat, 17 Oct 2020 17:44:16 GMT , 7ms
Sat, 17 Oct 2020 17:44:26 GMT , 7ms
Sat, 17 Oct 2020 17:44:27 GMT , 7ms
Sat, 17 Oct 2020 17:44:28 GMT , 7ms
Sat, 17 Oct 2020 17:44:33 GMT , 7ms
Sat, 17 Oct 2020 17:44:40 GMT , 8ms
Sat, 17 Oct 2020 17:44:43 GMT , 7ms
Sat, 17 Oct 2020 17:44:44 GMT , 7ms
Sat, 17 Oct 2020 17:44:45 GMT , 7ms
Sat, 17 Oct 2020 17:44:50 GMT , 7ms
Sat, 17 Oct 2020 17:44:55 GMT , 7ms
Sat, 17 Oct 2020 17:44:58 GMT , 7ms
Sat, 17 Oct 2020 17:45:08 GMT , 7ms
Sat, 17 Oct 2020 17:45:12 GMT , 7ms
Sat, 17 Oct 2020 17:45:15 GMT , 7ms
Sat, 17 Oct 2020 17:45:16 GMT , 7ms
Sat, 17 Oct 2020 17:45:17 GMT , 7ms
Sat, 17 Oct 2020 17:45:22 GMT , 7ms
Sat, 17 Oct 2020 17:45:25 GMT , 7ms
Sat, 17 Oct 2020 17:45:26 GMT , 7ms
Sat, 17 Oct 2020 17:45:30 GMT , 7ms
Sat, 17 Oct 2020 17:45:35 GMT , 7ms
Sat, 17 Oct 2020 17:45:36 GMT , 7ms
Sat, 17 Oct 2020 17:45:42 GMT , 7ms
Sat, 17 Oct 2020 17:45:56 GMT , 7ms
Sat, 17 Oct 2020 17:45:57 GMT , 7ms
Sat, 17 Oct 2020 17:46:02 GMT , 7ms
Sat, 17 Oct 2020 17:46:07 GMT , 7ms
Sat, 17 Oct 2020 17:46:08 GMT , 7ms
Sat, 17 Oct 2020 17:46:13 GMT , 7ms
Sat, 17 Oct 2020 17:46:18 GMT , 7ms
Sat, 17 Oct 2020 17:46:28 GMT , 7ms
Sat, 17 Oct 2020 17:46:34 GMT , 7ms
Sat, 17 Oct 2020 17:46:35 GMT , 8ms
Sat, 17 Oct 2020 17:46:39 GMT , 8ms
Sat, 17 Oct 2020 17:46:40 GMT , 8ms
Sat, 17 Oct 2020 17:46:48 GMT , 7ms
Sat, 17 Oct 2020 17:46:49 GMT , 7ms
Sat, 17 Oct 2020 17:46:53 GMT , 7ms
Sat, 17 Oct 2020 17:46:54 GMT , 7ms
Sat, 17 Oct 2020 17:46:58 GMT , 7ms
Sat, 17 Oct 2020 17:46:59 GMT , 7ms
Sat, 17 Oct 2020 17:47:03 GMT , 7ms
Sat, 17 Oct 2020 17:47:04 GMT , 7ms
Sat, 17 Oct 2020 17:47:13 GMT , 7ms
Sat, 17 Oct 2020 17:47:22 GMT , 7ms
Sat, 17 Oct 2020 17:47:28 GMT , 8ms
Sat, 17 Oct 2020 17:47:36 GMT , 7ms
Sat, 17 Oct 2020 17:47:37 GMT , 7ms
Sat, 17 Oct 2020 17:47:42 GMT , 7ms
Sat, 17 Oct 2020 17:47:43 GMT , 7ms
Sat, 17 Oct 2020 17:47:49 GMT , 7ms
Sat, 17 Oct 2020 17:47:52 GMT , 7ms
Sat, 17 Oct 2020 17:47:53 GMT , 7ms
Sat, 17 Oct 2020 17:48:07 GMT , 7ms
Sat, 17 Oct 2020 17:48:16 GMT , 7ms
Sat, 17 Oct 2020 17:48:20 GMT , 7ms
Sat, 17 Oct 2020 17:48:21 GMT , 7ms
Sat, 17 Oct 2020 17:48:25 GMT , 8ms
Sat, 17 Oct 2020 17:48:26 GMT , 8ms
Sat, 17 Oct 2020 17:48:32 GMT , 7ms
Sat, 17 Oct 2020 17:48:35 GMT , 7ms
Sat, 17 Oct 2020 17:48:38 GMT , 7ms
Sat, 17 Oct 2020 17:48:43 GMT , 7ms
Sat, 17 Oct 2020 17:48:44 GMT , 8ms
Sat, 17 Oct 2020 17:48:47 GMT , 7ms
Sat, 17 Oct 2020 17:48:52 GMT , 7ms
Sat, 17 Oct 2020 17:48:55 GMT , 8ms
Sat, 17 Oct 2020 17:48:56 GMT , 8ms
Sat, 17 Oct 2020 17:48:57 GMT , 7ms
Sat, 17 Oct 2020 17:48:58 GMT , 8ms
Sat, 17 Oct 2020 17:49:09 GMT , 7ms
Sat, 17 Oct 2020 17:49:15 GMT , 7ms
Sat, 17 Oct 2020 17:49:23 GMT , 8ms
Sat, 17 Oct 2020 17:49:24 GMT , 7ms
Sat, 17 Oct 2020 17:49:27 GMT , 7ms
Sat, 17 Oct 2020 17:49:28 GMT , 8ms
Sat, 17 Oct 2020 17:49:29 GMT , 7ms
Sat, 17 Oct 2020 17:49:35 GMT , 8ms
Sat, 17 Oct 2020 17:49:36 GMT , 7ms
Sat, 17 Oct 2020 17:49:42 GMT , 7ms
Sat, 17 Oct 2020 17:49:56 GMT , 7ms
Sat, 17 Oct 2020 17:50:01 GMT , 8ms
Sat, 17 Oct 2020 17:50:11 GMT , 8ms
Sat, 17 Oct 2020 17:50:12 GMT , 7ms
Sat, 17 Oct 2020 17:50:13 GMT , 8ms
Sat, 17 Oct 2020 17:50:18 GMT , 7ms
Sat, 17 Oct 2020 17:50:19 GMT , 7ms
Sat, 17 Oct 2020 17:50:20 GMT , 8ms
Sat, 17 Oct 2020 17:50:21 GMT , 7ms
Sat, 17 Oct 2020 17:50:31 GMT , 7ms
Sat, 17 Oct 2020 17:50:34 GMT , 7ms
Sat, 17 Oct 2020 17:50:35 GMT , 7ms
Sat, 17 Oct 2020 17:50:36 GMT , 7ms
Sat, 17 Oct 2020 17:50:41 GMT , 7ms
Sat, 17 Oct 2020 17:50:47 GMT , 8ms
Sat, 17 Oct 2020 17:50:54 GMT , 7ms
Sat, 17 Oct 2020 17:51:01 GMT , 7ms
Sat, 17 Oct 2020 17:51:08 GMT , 7ms
Sat, 17 Oct 2020 17:51:13 GMT , 7ms
Sat, 17 Oct 2020 17:51:18 GMT , 14ms
Sat, 17 Oct 2020 17:51:28 GMT , 8ms
Sat, 17 Oct 2020 17:51:34 GMT , 7ms
Sat, 17 Oct 2020 17:51:35 GMT , 8ms
Sat, 17 Oct 2020 17:51:38 GMT , 7ms
Sat, 17 Oct 2020 17:51:39 GMT , 7ms
Sat, 17 Oct 2020 17:51:42 GMT , 7ms
Sat, 17 Oct 2020 17:51:43 GMT , 7ms
Sat, 17 Oct 2020 17:51:44 GMT , 7ms
Sat, 17 Oct 2020 17:51:57 GMT , 7ms
Sat, 17 Oct 2020 17:52:03 GMT , 7ms
Sat, 17 Oct 2020 17:52:04 GMT , 7ms
Sat, 17 Oct 2020 17:52:05 GMT , 7ms
Sat, 17 Oct 2020 17:52:06 GMT , 7ms
Sat, 17 Oct 2020 17:52:09 GMT , 8ms
Sat, 17 Oct 2020 17:52:10 GMT , 7ms
Sat, 17 Oct 2020 17:52:13 GMT , 7ms
Sat, 17 Oct 2020 17:52:14 GMT , 7ms
Sat, 17 Oct 2020 17:52:15 GMT , 7ms
Sat, 17 Oct 2020 17:52:16 GMT , 7ms
Sat, 17 Oct 2020 17:52:17 GMT , 7ms
Sat, 17 Oct 2020 17:52:18 GMT , 8ms
Sat, 17 Oct 2020 17:52:19 GMT , 7ms
Sat, 17 Oct 2020 17:52:20 GMT , 7ms
Sat, 17 Oct 2020 17:52:21 GMT , 7ms
Sat, 17 Oct 2020 17:52:22 GMT , 7ms
The ping spikes occur every 22 minutes.
Maybe try to correlate these spikes with whatchappens/shows in dmesg and logread output. If you install the full ping binary, I believevthere is an option to get system timestamps for each icmp probe in the output. I have a hunch that -d is that option, but am unable to check right now.... Also intetestingvto correlate the modem's error counters with these spike events.
That said, what ever happens cyclically every 22 minutes does not necessarily need to happen on your router or link.... it can be something on the ISP side as well.
Is there any possibility this is just a bug in cake? Every so often it just fails to work and the ping goes up?
I can't see any reason for the spikes apart from that or an ISP issue but the ISP issue seems strange as it doesn't happen at all when I'm not using the connection.
Personally I think cake occasionally misses a packet and you get a latency spike like you would get if cake wasn't running. Perhaps I'm talking nonsense.
Sure, bugs are possible, since software rarely is perfect....
I really would like to see the tc -s qdisc output for such spikes (especially whether the peak delay pk_delay shows a value that corresponds to the observed ping spike).
I would also like to see mtr -ezb4 $YOUR.TEST.PINGHOST.IP-ADDRESS to get a feel for how RTT behaves/develops along the path. If you have another machine under your control in another network, it would be most excellent to run one mtr instance on each side targeting the respective other to get some visibility on the return path as well.
I am very happy to help but you'll have to walk me through how I do this.
Ah you need to log into the router via ssh, and then simply issue the following command:
tc -s qdisc
then copy and paste the output into the forum (properly formatted):
Please use the "Preformatted text </>" button for logs, scripts, configs and general console output.
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 30062941237 bytes 24190136 pkt (dropped 0, overlimits 0 requeues 1678)
backlog 0b 0p requeues 1678
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 20496646515 bytes 15909796 pkt (dropped 0, overlimits 0 requeues 971)
backlog 0b 0p requeues 971
maxpacket 10598 drop_overlimit 0 new_flow_count 9811 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 9566294722 bytes 8280340 pkt (dropped 0, overlimits 0 requeues 707)
backlog 0b 0p requeues 707
maxpacket 6486 drop_overlimit 0 new_flow_count 210 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 800d: dev eth1 root refcnt 9 bandwidth 5144Kbit besteffort dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead 34 mpu 72
Sent 1432660914 bytes 17038298 pkt (dropped 8192, overlimits 14740647 requeues 8)
backlog 0b 0p requeues 8
memory used: 677376b of 4Mb
capacity estimate: 5144Kbit
min/max network layer size: 16 / 1500
min/max overhead-adjusted size: 74 / 1558
average network hdr offset: 14
Tin 0
thresh 5144Kbit
target 5.0ms
interval 100.0ms
pk_delay 140us
av_delay 41us
sp_delay 3us
backlog 0b
pkts 17046490
bytes 1444643448
way_inds 2242460
way_miss 94066
way_cols 0
drops 8192
marks 0
ack_drop 0
sp_flows 22
bk_flows 1
un_flows 0
max_len 1514
quantum 300
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 28115859038 bytes 23915550 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc cake 800e: dev ifb4eth1 root refcnt 2 bandwidth 41120Kbit besteffort dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms ptm overhead 34 mpu 72
Sent 26896620409 bytes 22632853 pkt (dropped 1282695, overlimits 42197033 requeues 0)
backlog 3028b 2p requeues 0
memory used: 1398528b of 4Mb
capacity estimate: 41120Kbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 82 / 1558
average network hdr offset: 14
Tin 0
thresh 41120Kbit
target 5.0ms
interval 100.0ms
pk_delay 34.9ms
av_delay 14.6ms
sp_delay 163us
backlog 3028b
pkts 23915550
bytes 28450745700
way_inds 1859643
way_miss 92986
way_cols 0
drops 1282695
marks 0
ack_drop 0
sp_flows 30
bk_flows 1
un_flows 0
max_len 4542
quantum 1254
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 1065914843 bytes 16962997 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 17104 drop_overlimit 0 new_flow_count 105656 ecn_mark 0
new_flows_len 1 old_flows_len 3
1 Like
I would like to note I have capped my torrent client to 25Mb and the issue is still occurring.
So this indicates that cake itself saw a maximum of 35ms delay in its own queue which might explain some of your spikes. If your 22 minutes theory is correct, wait until say 2 minutes before the expected problem, run /etc/init.d/sqm stop ; /etc/init.d/sqm start to clear cake's statistics counters, wait for the ping spike to happen and call tc -s qdisc ASAP. Repeat the same exercise for a similar duration at a time when you expect no ping spikes and post the results here.
The next steps would be to see what keeps the router busy at around the ping spikes... but first let's try to confirm the issue.
BTW, a ping spike every X minutes seems rather survivable and non-catastrophic to me. we are talking about the best-effort internet here and not a well-managed 'deterministic' local network after all.
I'm not sure my 22 minutes theory turned out to be correct.
I will run the test you described above and see if I am right to discount the 22 minute theory.
During my torrent download - capped at 25Mb/s - I was seeing ping spikes every couple of seconds.
@moeller0, I paused my torrent download, capped the maximum speed in the torrent client to 10Mbit/s, stopped and started sqm and quickly started my download. I immediately saw a ping spike to 21ms.
qdisc noqueue 0: dev lo root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 35005281024 bytes 28548563 pkt (dropped 0, overlimits 0 requeues 1817)
backlog 0b 0p requeues 1817
qdisc fq_codel 0: dev eth0 parent :2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 23414769931 bytes 18540486 pkt (dropped 0, overlimits 0 requeues 1060)
backlog 0b 0p requeues 1060
maxpacket 10598 drop_overlimit 0 new_flow_count 12588 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc fq_codel 0: dev eth0 parent :1 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 11590511093 bytes 10008077 pkt (dropped 0, overlimits 0 requeues 757)
backlog 0b 0p requeues 757
maxpacket 6486 drop_overlimit 0 new_flow_count 295 ecn_mark 0
new_flows_len 0 old_flows_len 0
qdisc cake 8015: dev eth1 root refcnt 9 bandwidth 5144Kbit besteffort dual-srchost nat nowash ack-filter split-gso rtt 100.0ms ptm overhead 34 mpu 72
Sent 1349295 bytes 17497 pkt (dropped 1, overlimits 16973 requeues 0)
backlog 0b 0p requeues 0
memory used: 117Kb of 4Mb
capacity estimate: 5144Kbit
min/max network layer size: 16 / 1275
min/max overhead-adjusted size: 74 / 1330
average network hdr offset: 14
Tin 0
thresh 5144Kbit
target 5.0ms
interval 100.0ms
pk_delay 1.1ms
av_delay 80us
sp_delay 3us
backlog 0b
pkts 17498
bytes 1349369
way_inds 1279
way_miss 292
way_cols 0
drops 1
marks 0
ack_drop 0
sp_flows 20
bk_flows 1
un_flows 0
max_len 1289
quantum 300
qdisc ingress ffff: dev eth1 parent ffff:fff1 ----------------
Sent 23160832 bytes 24747 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc noqueue 0: dev br-lan root refcnt 2
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
qdisc fq_codel 0: dev pppoe-wan root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms memory_limit 32Mb ecn
Sent 30253892 bytes 221379 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
maxpacket 5968 drop_overlimit 0 new_flow_count 2939 ecn_mark 0
new_flows_len 0 old_flows_len 2
qdisc cake 8016: dev ifb4eth1 root refcnt 2 bandwidth 41120Kbit besteffort dual-dsthost nat nowash ingress no-ack-filter split-gso rtt 100.0ms ptm overhead 34 mpu 72
Sent 23421128 bytes 24680 pkt (dropped 49, overlimits 37566 requeues 0)
backlog 27252b 18p requeues 0
memory used: 799488b of 4Mb
capacity estimate: 41120Kbit
min/max network layer size: 46 / 1500
min/max overhead-adjusted size: 82 / 1558
average network hdr offset: 14
Tin 0
thresh 41120Kbit
target 5.0ms
interval 100.0ms
pk_delay 3.4ms
av_delay 584us
sp_delay 119us
backlog 27252b
pkts 24747
bytes 23507290
way_inds 1622
way_miss 279
way_cols 0
drops 49
marks 0
ack_drop 0
sp_flows 11
bk_flows 2
un_flows 0
max_len 1514
quantum 1254