There is another way to do this, which moves the processing off the router. I do some similar tracking but use softflowd. softflowd creates NetFlow records for flows (traffic between pairs of addresses and ports) and sends them to another system for storage and processing.
This (mostly) moves the storage, backup and analysis problem onto another system, which typically has databases on real disks. maybe with RAID, and lots of processor power to analyse the data in various ways. However, flow data is normally only sent when a flow finishes so if the router crashes you can lose the information about the flows which were in progress. And the flow data is sent using UDP and can be lost.
It works well if your goal is to understand and monitor traffic. But maybe not well enough for charging.
Anyway, not suggesting you should switch to softflowd but just wanted to point out that it is available on LEDE and is an alternative to building lots of tools on the router itself, depending on your requirements.
Both do essentially the same. My version packs up all databases into one tar.gz archive and brings its own restore function, and being an init script it runs on startup and -- more importantly -- on shutdown. @mkresin's copies them file by file, uses the restore function that is built into the vnstat init script, and, being a standalone script, doesn't backup on shutdown. I think the latter is important, you might disagree.
I'm currently working on refactoring the (essentially quite simple, but convoluted) vnstat init script, bringing a backup function (that does both of the above, configurable) and procd support.
@takimata,
thank you for your clarification. OKay, so I've gone with your code.
I did
chmod +x /etc/init.d/vnstat_backup
# /etc/init.d/vnstat_backup enable
# /etc/init.d/vnstat_backup backup
What do I do now when I want to see the "Wired" usage for the billing cycle? And where do I go to see the "Wireless" usage for the billing cycle? And where do I go if I want to look at older months (older billing cycles"?
In /etc/config/vnstat you list the interfaces you want counted. Depending on your device that might include a variation of eth0, eth1, eth0.1, eth0.2, or similar for wired connections, wlan0 and/or wlan1 for your wireless connections, maybe even pppoe-wan for the incoming WAN connection. You really need to look that up for yourself.
Once you set up those interfaces (each on its own line) you start/restart vnstat (/etc/init.d/vnstat start or /etc/init.d/vnstat restart). From then on, if you simply type vnstat into your console, you will get statistics (for the last update interval). Try vnstat --help to select daily, weekly, monthly, even hourly statistics for all or just one specific interface.
They are separated automatically, depending on your view option. Again, see --help for display options.
(Hint: If you are watching multiple interfaces, see the "default interface" option at the very top of /etc/vnstat.conf ... By default it is set to eth0, which you very likely don't have on your device. Setting this to an interface you watch will get rid of the "eth0 not found" error message if you run vnstat without any parameters.)
The below commands will add a cronjob entry that triggers a backup every 6 hours every day, if necessary adjust to an interval you feel comfortable with:
Thanks, I've installed luci-app-vnstat, as you've kindly suggested. What happens when, say month #2 comes around? Will I see the historical data for month #1? If not, where would I go to see older months (i.e. older billing months)?
I do something similar using collectd to stream all the stats off the router - and into a collector. I use influxdb with the collectd listener (there are other options), and then grafana to chart and display stats - as they are logged at the granularity of interface, its fairly simple to pull up a chart of whatever i want.
No, I don't know. Nobody knows. If your clients are not transferring within the network it should really be just a tiny fraction of actual traffic. But noone can tell what exactly your clients do or give you exact numbers just from looking at the raw in&out traffic of an interface.
Although you could of course make an educated guess by simply looking at the numbers. Look at the WAN interface and your LAN/WLAN interfaces. Naturally, the difference is the traffic all your local clients are generating within the network. You'd at least get some idea of how much traffic stays within your local network.
Not possible with vnstat. It doesn't look where the traffic is going, just how much is coming in and how much is going out on any interface.
I believe darkstat can be configured to only look at traffic going to certain netmasks, or ignore certain netmasks. But I didn't use darkstat yet, so I couldn't tell you for certain if it's possible and if so, how to do that. Sorry.
I run ubuntu, so used the latest apt sources for influxdb and grafana, set up influxdb as a network collectd listener, and then use the network plugin on connectd on the openwrt box, to send all stats to influxdb database.
grafana can understand the syntax of influxdb for the most part - so the creation of charts generally only needs the drop down menus in the chart creation screen. The one exception is the interface statistics, as you want to turn a counter into a rate, so you manually need to enter the query value as SELECT non_negative_derivative(mean("value"),10s) from "interface_rx" ... etc, do the same for "interface_tx" too.
I have a good bit of experience with Docker so I have set up InfluxDB and Graphana on my Mac. I just need to set up CollectD now on LEDE to collect and send data to Influx.
Would you mind sharing your CollectD config? Thanks again.
To follow up to myself: Yes, it's possible in darkstat. You'd have to uncomment the network_filter option in /etc/config/darkstat and adjust the two netmasks in the statement to your configuration. Effectively it says to disregard anything that comes from and goes to the local network. I'd also recommend to use the network_netmask (again, adjusted for the local configuration) and local_only options (they have to be used in conjunction) so the hosts list in the statistics is not cluttered with the destination hosts.
darkstat also has an option to import/export database on start/stop (the import and export options), but they are not supported by the UCI config file. I'll have to look into that, but for now it doesn't backup or restore the database.
[edit] In line with the original question, I didn't find any option to adjust the "billing cycle", i.e. start-of-month, for darkstat though.
I've had your solution running for about a month. And yesterday, the router (and most of the house) briefly lost power as I was trying to fix an electrical problem (not related to the router).
I checked sftp://192.168.1.1/etc/vnstat_backup.tar.gz and it looks like it's empty.
The below commands will add a cronjob entry that triggers a backup every 6 hours every day, if necessary adjust to an interval you feel comfortable with:
In terminal, I first did ssh root@192.168.1.1
then echo "* */6 * * * /etc/init.d/vnstat_backup backup" >> /etc/crontabs/root
and /etc/init.d/cron restart
How can I verify that I correctly have set up automatic backups?
Thank you very much for your reply and adding to the advice on this thread.
Could you kindly tell me how nlbwmon and luci-app-nlmwmon compare with vnstat and the luci version of vnstat?
By the way, you said that nlbwmon is for measuring per-host traffic. Is that what I'm seeking to do -- measure per-host traffic? I'm not familiar with all these terms (e.g. host).
I had to do some terminal coding (copy-pasting) to set up cron for auto backups. With nlbwmon, is it simpler to set everything up.