Router Performance Monitor / SQM estimator

Installing and Using OpenWrt
1

Following on some ideas from How to make getting started with configuring easier and from some work we did 9 months ago that resulted in my github repo here: https://github.com/dlakelan/routerperf

I've just pushed a lua script to that repo that does a basic monitoring loop and dumps the results into a sqlite3 file. It uses lua because I can assume lua is already on OpenWrt routers... but it will require installing the GNU ping and luaposix and lua-sqlite3 (let me know if it needs anything else).

The basic structure is this: generate a random time between 10 and 20 seconds uniformly, choose randomly from some major enterprise hosts to ping, choose a random packet size, and then measure CPU, bandwidth, and latency for about a second, plop all that data into two tables in the sqlite3 database, and repeat.

Ideally this will build up a database of router performance data which can then be used to configure SQM, as well as to estimate some real-world performance abilities for various hardware.

I'm pretty sure at least @moeller0 will be interested in this, and he might be willing to test it :wink: and testing from others is also welcome. First thing I'd like to make sure is that this can run on real routers and doesn't just bork (I tested it mainly on a Linux desktop and it doesn't bork there). After the non-borking phase, I'd like to have some people contribute sqlite3 databases to a google drive directory where I can then download them and develop some analysis methods that will result in some kind of step towards an auto-configuration of SQM and also generation of statistics on hardware performance.... Then maybe we can see about translating that analysis method into a lua script that could become a LUCI-app. It'd be great to click a button and then tell a user to run 3 dslreports.com speedtests in a row, and then 2 minutes later they have suggested SQM config and a description of what fine tuning they might want to do.

2

I pushed a new version that does a better job of error handling and therefore doesn't die when there's a problem pinging the host... also doesn't print the debugging message at the beginning of each iteration of the loop.

3

tests ok > -4 archer-c5-v2

wanted

coreutils-sleep
coreutils-date
iputils-ping
iputils-ping6

Found this;

WANinterface(`

')
LANinterface(`
br-lan
')


pingformat(1)
nstimestart(1552063115,722987260)
pingdata(`
PING 8.8.8.8 (8.8.8.8) 16(44) bytes of data.
[1552063115.740943] 24 bytes from 8.8.8.8: icmp_req=1 ttl=59 time=9.85 ms
[1552063115.946499] 24 bytes from 8.8.8.8: icmp_req=2 ttl=59 time=8.80 ms
[1552063116.155851] 24 bytes from 8.8.8.8: icmp_req=3 ttl=59 time=8.15 ms
[1552063116.367743] 24 bytes from 8.8.8.8: icmp_req=4 ttl=59 time=9.94 ms
[1552063116.575948] 24 bytes from 8.8.8.8: icmp_req=5 ttl=59 time=8.16 ms
[1552063116.787389] 24 bytes from 8.8.8.8: icmp_req=6 ttl=59 time=9.69 ms
[1552063117.003010] 24 bytes from 8.8.8.8: icmp_req=7 ttl=59 time=15.3 ms
4

Hi there, instead of the shell script, try the new Lua version. Run the ./luamonitor.lua script it basically replaces the earlier shell function

1 Like
5 
opkg install luasql-sqlite3 && opkg install luaposix
./luamonitor.lua

**some minimal status? = good**
perpetual?

cpu

mostly 0% -> flicks up to 2% = good
( small 10% peaks > calcs ) = not too shabby
  • Ctrl-c after 10 mins.....
  • Found datafile from strace ( /tmp/sql..... ) = 36K

escape / end summary? = good

Contents.... ( snipped bottom 70% due to posting limits )

PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
CREATE TABLE stats (secs integer primary key, nsecs int, addr varchar(255), packsize int, cpuuser float, cpunice float, cpusys float, cpuidle float, cpuio float, cpuirq float, cpusoftirq float, cpusteal float, cpuguest float, gnice float, ping1 float, pingmed float, ping5 float, cpuuser2 float, cpunice2 float, cpusys2 float, cpuidle2 float, cpuio2 float, cpuirq2 float, cpusoftirq2 float, cpusteal2 float, cpuguest2 float, gnice2 float, endsecs integer, endnsecs integer);
INSERT INTO "stats" VALUES(1552064256,787884755,'cloudflare.com',272,1324470.0,5599.0,1063325.0,356462593.0,754.0,0.0,1531813.0,0.0,0.0,0.0,10.3,13.7,9.21,1324472.0,5599.0,1063330.0,356465095.0,756.0,0.0,1531815.0,0.0,0.0,0.0,1552064281,922115115);
INSERT INTO "stats" VALUES(1552064292,937878177,'cloudflare.com',1384,1324472.0,5599.0,1063331.0,356466195.0,756.0,0.0,1531816.0,0.0,0.0,0.0,10.4,11.1,9.93,1324474.0,5599.0,1063335.0,356468700.0,756.0,0.0,1531817.0,0.0,0.0,0.0,1552064318,62118272);
INSERT INTO "stats" VALUES(1552064344,957895482,'fbcdn.net',1095,1324475.0,5599.0,1063339.0,356471384.0,756.0,0.0,1531818.0,0.0,0.0,0.0,10.2,14.9,9.55,1324475.0,5599.0,1063342.0,356471485.0,756.0,0.0,1531818.0,0.0,0.0,0.0,1552064346,7618965);
INSERT INTO "stats" VALUES(1552064378,917902487,'one.one.one.one',1439,1324479.0,5600.0,1063342.0,356474771.0,756.0,0.0,1531819.0,0.0,0.0,0.0,10.0,10.9,9.17,1324480.0,5600.0,1063343.0,356474855.0,756.0,0.0,1531820.0,0.0,0.0,0.0,1552064379,791941785);
INSERT INTO "stats" VALUES(1552064418,777910912,'fbcdn.net',1272,1324482.0,5601.0,1063349.0,356478745.0,756.0,0.0,1531820.0,0.0,0.0,0.0,11.5,13.9,9.83,1324482.0,5601.0,1063350.0,356478828.0,756.0,0.0,1531820.0,0.0,0.0,0.0,1552064419,622416240);
INSERT INTO "stats" VALUES(1552064430,637912417,'akamai.com',1079,1324482.0,5602.0,1063351.0,356479928.0,756.0,0.0,1531820.0,0.0,0.0,0.0,10.0,11.8,9.01,1324483.0,5602.0,1063351.0,356480014.0,756.0,0.0,1531820.0,0.0,0.0,0.0,1552064431,512638987);
INSERT INTO "stats" VALUES(1552064448,527935907,'akamai.com',1074,1324483.0,5602.0,1063352.0,356481715.0,756.0,0.0,1531820.0,0.0,0.0,0.0,10.2,11.4,13.6,1324483.0,5602.0,1063352.0,356481800.0,756.0,0.0,1531820.0,0.0,0.0,0.0,1552064449,382861155);
INSERT INTO "stats" VALUES(1552064466,397923362,'fbcdn.net',685,1324483.0,5603.0,1063353.0,356483499.0,756.0,0.0,1531821.0,0.0,0.0,0.0,10.0,12.4,9.02,1324483.0,5603.0,1063354.0,356483583.0,756.0,0.0,1531821.0,0.0,0.0,0.0,1552064467,255177667);
INSERT INTO "stats" VALUES(1552064493,147931107,'one.one.one.one',161,1324485.0,5604.0,1063355.0,356486168.0,756.0,0.0,1531822.0,0.0,0.0,0.0,11.1,15.7,9.29,1324485.0,5604.0,1063356.0,356486251.0,756.0,0.0,1531823.0,0.0,0.0,0.0,1552064494,3765930);
INSERT INTO "stats" VALUES(1552064594,308295042,'one.one.one.one',516,1324518.0,5607.0,1063397.0,356496196.0,756.0,0.0,1531832.0,0.0,0.0,0.0,10.1,13.9,9.44,1324547.0,5607.0,1063491.0,356496279.0,756.0,0.0,1531851.0,0.0,0.0,0.0,1552064598,42877582);
INSERT INTO "stats" VALUES(1552064608,458606632,'akamai.com',751,1324589.0,5608.0,1063621.0,356497279.0,756.0,0.0,1531868.0,0.0,0.0,0.0,10.2,14.1,16.4,1324607.0,5608.0,1063726.0,356497365.0,756.0,0.0,1531883.0,0.0,0.0,0.0,1552064612,77305560);
INSERT INTO "stats" VALUES(1552064627,488300100,'akamai.com',745,1324639.0,5608.0,1063850.0,356498866.0,756.0,0.0,1531905.0,0.0,0.0,0.0,12.1,9.28,9.74,1324665.0,5608.0,1063955.0,356498947.0,756.0,0.0,1531916.0,0.0,0.0,0.0,1552064631,88458927);
INSERT INTO "stats" VALUES(1552064644,498634290,'akamai.com',483,1324698.0,5609.0,1064079.0,356500246.0,756.0,0.0,1531937.0,0.0,0.0,0.0,109.0,110.0,111.0,1324718.0,5609.0,1064182.0,356500342.0,756.0,0.0,1531953.0,0.0,0.0,0.0,1552064648,324932770);
INSERT INTO "stats" VALUES(1552064658,738306870,'cloudflare.com',699,1324740.0,5609.0,1064322.0,356501343.0,756.0,0.0,1531979.0,0.0,0.0,0.0,10.0,10.8,9.92,1324757.0,5610.0,1064438.0,356503849.0,756.0,0.0,1531998.0,0.0,0.0,0.0,1552064686,765452767);
INSERT INTO "stats" VALUES(1552064715,158316325,'fbcdn.net',651,1324844.0,5611.0,1064794.0,356506334.0,756.0,0.0,1532052.0,0.0,0.0,0.0,11.1,12.2,13.4,1324869.0,5611.0,1064894.0,356506418.0,756.0,0.0,1532069.0,0.0,0.0,0.0,1552064718,795266405);
INSERT INTO "stats" VALUES(1552064751,268633300,'one.one.one.one',1136,1324986.0,5612.0,1065250.0,356509272.0,756.0,0.0,1532126.0,0.0,0.0,0.0,10.3,13.7,9.6,1325013.0,5612.0,1065351.0,356509355.0,756.0,0.0,1532139.0,0.0,0.0,0.0,1552064754,937761082);
INSERT INTO "stats" VALUES(1552064771,348648955,'cloudflare.com',995,1325054.0,5612.0,1065480.0,356510953.0,756.0,0.0,1532155.0,0.0,0.0,0.0,10.1,9.63,9.7,1325089.0,5612.0,1065578.0,356513451.0,756.0,0.0,1532173.0,0.0,0.0,0.0,1552064799,241594750);
INSERT INTO "stats" VALUES(1552064813,648745087,'cloudflare.com',1173,1325126.0,5612.0,1065711.0,356514849.0,756.0,0.0,1532186.0,0.0,0.0,0.0,10.8,11.4,9.75,1325154.0,5613.0,1065818.0,356517349.0,756.0,0.0,1532201.0,0.0,0.0,0.0,1552064841,606527690);
INSERT INTO "stats" VALUES(1552064861,18651530,'fbcdn.net',812,1325181.0,5613.0,1065973.0,356519245.0,756.0,0.0,1532209.0,0.0,0.0,0.0,10.1,10.7,19.1,1325208.0,5613.0,1066073.0,356519330.0,756.0,0.0,1532221.0,0.0,0.0,0.0,1552064864,645586627);
INSERT INTO "stats" VALUES(1552064883,48800855,'one.one.one.one',939,1325238.0,5613.0,1066204.0,356521127.0,756.0,0.0,1532242.0,0.0,0.0,0.0,10.1,8.79,9.85,1325265.0,5613.0,1066314.0,356521209.0,756.0,0.0,1532254.0,0.0,0.0,0.0,1552064886,732857612);
INSERT INTO "stats" VALUES(1552064899,138768985,'cloudflare.com',563,1325304.0,5613.0,1066441.0,356522407.0,756.0,0.0,1532268.0,0.0,0.0,0.0,10.0,13.3,9.9,1325331.0,5614.0,1066546.0,356524905.0,756.0,0.0,1532285.0,0.0,0.0,0.0,1552064927,77281807);
INSERT INTO "stats" VALUES(1552064943,488799302,'fbcdn.net',1312,1325370.0,5615.0,1066677.0,356526502.0,756.0,0.0,1532304.0,0.0,0.0,0.0,10.3,10.5,13.8,1325401.0,5615.0,1066775.0,356526587.0,756.0,0.0,1532321.0,0.0,0.0,0.0,1552064947,178644377);
INSERT INTO "stats" VALUES(1552064961,468049482,'one.one.one.one',927,1325426.0,5615.0,1066907.0,356527973.0,756.0,0.0,1532345.0,0.0,0.0,0.0,10.3,10.7,9.77,1325426.0,5615.0,1066908.0,356528059.0,756.0,0.0,1532345.0,0.0,0.0,0.0,1552064962,343575680);
INSERT INTO "stats" VALUES(1552064990,218065720,'one.one.one.one',735,1325428.0,5616.0,1066911.0,356530841.0,756.0,0.0,1532345.0,0.0,0.0,0.0,10.4,11.2,9.8,1325429.0,5616.0,1066912.0,356530925.0,756.0,0.0,1532345.0,0.0,0.0,0.0,1552064991,83230237);
CREATE TABLE ifstats (secs int, nsecs int, iface varchar(255), rbytes int, rpack int, rerr int, rdrop int, rfifo int, rframe int, rcompr int, rmulti int, tbytes int, tpack int, terr int, tdrop int, tfifo int, tcolls int, tcarrier int, tcompr int, rbytes2 int, rpack2 int, rerr2 int, rdrop2 int, rfifo2 int, rframe2 int, rcompr2 int, rmulti2 int, tbytes2 int, tpack2 int, terr2 int, tdrop2 int, tfifo2 int, tcolls2 int, tcarrier2 int, tcompr2 int);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'eth0.1',10145791212,125501750,0,0,0,0,0,5249065,337004610646,226617686,0,0,0,0,0,0,10145801608,125501774,0,0,0,0,0,5249065,337004616768,226617704,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'br-lan',10145750748,125501165,0,0,0,0,0,0,337004610646,226617686,0,0,0,0,0,0,10145761144,125501189,0,0,0,0,0,0,337004616768,226617704,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'ifb1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'eth0.2',333781907010,226126149,0,0,0,0,0,0,10739161880,119133391,0,0,0,0,0,0,333781923481,226126297,0,0,0,0,0,0,10739185133,119133546,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'lo',67247115,904752,0,0,0,0,0,0,67247115,904752,0,0,0,0,0,0,67251647,904814,0,0,0,0,0,0,67251647,904814,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'eth2',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'ifb0',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'teql0',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'eth0',2810560822,358314838,0,0,0,0,0,0,1674621815,352356315,0,0,0,0,0,0,2810590785,358315010,0,0,0,0,0,0,1674651882,352356488,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064256,787884755,'eth1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'eth0.1',10145815679,125501802,0,0,0,0,0,5249069,337004625202,226617723,0,0,0,0,0,0,10145833273,125501839,0,0,0,0,0,5249072,337004636240,226617749,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'br-lan',10145775215,125501217,0,0,0,0,0,0,337004625202,226617723,0,0,0,0,0,0,10145792809,125501254,0,0,0,0,0,0,337004636240,226617749,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'ifb1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'eth0.2',333781931741,226126318,0,0,0,0,0,0,10739198842,119133571,0,0,0,0,0,0,333781958198,226126468,0,0,0,0,0,0,10739233759,119133728,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'lo',67251647,904814,0,0,0,0,0,0,67251647,904814,0,0,0,0,0,0,67256179,904876,0,0,0,0,0,0,67256179,904876,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'eth2',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'ifb0',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'teql0',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'eth0',2810613998,358315059,0,0,0,0,0,0,1674674201,352356532,0,0,0,0,0,0,2810661415,358315246,0,0,0,0,0,0,1674720888,352356715,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064292,937878177,'eth1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'eth0.1',10145842927,125501857,0,0,0,0,0,5249075,337004642563,226617758,0,0,0,0,0,0,10145844438,125501859,0,0,0,0,0,5249075,337004643251,226617760,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'br-lan',10145802463,125501272,0,0,0,0,0,0,337004642563,226617758,0,0,0,0,0,0,10145803974,125501274,0,0,0,0,0,0,337004643251,226617760,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'ifb1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'eth0.2',333781964499,226126479,0,0,0,0,0,0,10739243149,119133745,0,0,0,0,0,0,333781966005,226126490,0,0,0,0,0,0,10739249571,119133758,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'lo',67256179,904876,0,0,0,0,0,0,67256179,904876,0,0,0,0,0,0,67257227,904888,0,0,0,0,0,0,67257227,904888,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'eth2',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'ifb0',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'teql0',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'eth0',2810677892,358315275,0,0,0,0,0,0,1674736705,352356741,0,0,0,0,0,0,2810681143,358315288,0,0,0,0,0,0,1674743875,352356756,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064331,77891512,'eth1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064344,957895482,'eth0.1',10145853022,125501879,0,0,0,0,0,5249079,337004646960,226617773,0,0,0,0,0,0,10145853074,125501880,0,0,0,0,0,5249079,337004647754,226617775,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064344,957895482,'br-lan',10145812558,125501294,0,0,0,0,0,0,337004646960,226617773,0,0,0,0,0,0,10145812610,125501295,0,0,0,0,0,0,337004647754,226617775,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064344,957895482,'ifb1',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064344,957895482,'eth0.2',333781969504,226126502,0,0,0,0,0,0,10739257567,119133773,0,0,0,0,0,0,333781976080,226126511,0,0,0,0,0,0,10739263472,119133781,0,0,0,0,0,0);
INSERT INTO "ifstats" VALUES(1552064344,957895482,'lo',67257227,904888,0,0,0,0,0,0,67257227,904888,0,0,0,0,0,0,67258328,904900,0,0,0,0,0,0,67258328,904900,0,0,0,0,0,0);
COMMIT;

maybe possible to limit these?

stat64("/tmp/sqmmonitor.db-journal", 0xbea33188) = -1 ENOENT (No such file or directory)
_llseek(3, 24, [24], SEEK_SET)          = 0
read(3, "\0\0\4[\0\0\0\31\0\0\0\0\0\0\0\0", 16) = 16
stat64("/tmp/sqmmonitor.db-wal", 0xbea33188) = -1 ENOENT (No such file or directory)
6

awesome thanks for the feedback. yes I originally had it printing out "getting data" each time through its cycle, but I just commented that line out. it is in fact a perpetual monitor ultimately would run like a daemon maybe.

I think those messages about the -journal and -wal file are just sqlite checking to see if it needs to do crash recovery when it initially opens. I don't think those are really problems.

Would you be willing to run this thing like

./luamonitor.lua & (so it goes into the background) and let it run for a day and contribute the resulting sqlite file to an analysis?

warning, it's collecting information that would let me know how much bandwidth you're using at different times of day, but it doesn't collect any information about traffic other than total bandwidth used on each interface.

7

done in 24 hrs :wink:

conn:execute("PRAGMA locking_mode=EXCLUSIVE;")
8

What symptoms are you seeing that make you suggest the locking mode line?

9

No syptoms at all.... just there for brevvity...... As you mentioned those failed reads are just sqlite3 going through it's locking checks as to be expected....

Maybe someone in the future will find a use for it....

p.s. the script seems to gen around 4.0K per 30secs.... ( or whatever the update interval is ) as a worst guesstimate....

10

Update interval is random but averages 15 seconds. 4k seems largish... But sqlite isn't super space efficient. If you're right it will be close to a megabyte of data per hour. Sqlite might allocate chunks so it might look more expensive than it is if you measure over short times.

11

ok, looking more like 1MB per 3hr

on this device..... average seemed alot higher.... maybe noteworthy... ( say closer to 40secs )

12

Interesting, it definitely generates a random number between 10 and 20 secs for the sleep time, so average should be 15, the data collection should take a second, and then a little time to slam it in the DB. I wonder if the nanosleep function has issues on some routers. Are you over clocking or anything like that?

13

Nope. shes a pretty vanilla c5-v2 ( which is not a common image )

Buuuuuuuttt........ I was messing around with a sqm script.......

So..... that has HUGE potential to interfere...... i guess it don't work too well icmp should be faster....... In strace earlier today..... it was just static..... so I doubt it's external.... Anyways..... that's what the testings for right? A few more samples and the cause if any will surface soon enough is spose....

Anyways..... we just found another lil, catch to be mindful of.... :slight_smile:

EDIT: Disk potentially....

root@c5v2-2019:~# cat /proc/cpuinfo 
processor	: 0
model name	: ARMv7 Processor rev 0 (v7l)
BogoMIPS	: 800.00
Features	: half thumb fastmult edsp tls 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x3
CPU part	: 0xc09
CPU revision	: 0

Hardware	: BCM5301X




           CPU0       
 16:          0     GIC-0  27 Edge      gt
 17:  363205716     GIC-0  29 Edge      twd
 18:        178     GIC-0 117 Level     gpio, ttyS0
 23:      14450     GIC-0 104 Level     spi_lr_fullness_reached
 24:          0     GIC-0 105 Level     spi_lr_session_aborted
 25:          0     GIC-0 106 Level     spi_lr_impatient
 26:       2783     GIC-0 107 Level     spi_lr_session_done
 28:        753     GIC-0 109 Level     mspi_done
 29:          0     GIC-0 110 Level     mspi_halted
 30:  459314258     GIC-0 179 Level     bgmac
 37:      11922     GIC-0 111 Level     ehci_hcd:usb1, ohci_hcd:usb2
14

I guess if your roundtrip time is measured in seconds the 5 pings will take a while :joy:

Anyway thanks for the testing, my router is x86 so it's unlikely to show up real world issues that embedded devices will have.

1 Like
15

Just for clarification. Locking mode exclusive locks the db-file once and doesn't unlock after every access like normal mode does. So it saves some IO-calls. You could also try journal mode off, to get rid of the journal and reduce writes by that. In case something crashes or router reboots db-file might get corrupted without journal though.

16

right now the db is in the /tmp which is in RAM on most machines anyway :wink:

exclusive probably makes good sense in this usage.

17

I've run my script all night, so maybe around 12 hours and got about 6.5MB of data. So it looks like it's around about 1/2 MB/hr

MOST of the time the router is probably idle, certainly at night. So it might make sense to build up a day's worth of data, curate it to find some summaries of what idle looks like, drop the idle data, and then have it only keep further data if it represents "activity".

1 Like
18

This already makes it dangerous to run on a low-end 16/64 router, unless the db is stored on an external drive. one potential use-case would be the ATM-overhead detector script (to avoid the pesky collection step); for that use case it might be sufficient to store say the minimum RTT per packet size and month (to allow for the ISP to change the encapsulation), so ~150012/1024 = 17.578125 KB per year, heck even storing daily minimums would only eat 1500365/1024 = 534.66 KB per year.

19

Right it certainly shouldn't store all this detail forever. (Or even for more than an hour maybe). But for now I need some detailed data so I can figure out what method should be used for summarization and inference. Pure averages etc are not relevant because when it comes to SQM the behavior under extreme events is the relevant information.

Right now I'm hoping to establish two things, that the CPU requirements are low enough to run even on low end hardware, and that it doesn't crash or collect erroneous data. Also to get detailed stats from a few volunteers so I can develop an analysis scheme.

Here's a Google drive Link where people can donate sqlite dbs, please also provide a text file describing your hardware and the general info on the link, speed up and down, sqm settings, and anything you think is relevant.

https://drive.google.com/folderview?id=1v_S3oFhLEIq49ShKMxjZkgvBQK8IP9ko

2 Likes
20

Should it be luasql-sqlite3 ?

1 Like