I am setting up a router with 3G/4G WAN and would like to understand where my data is consumed. Typically I would like to know how much iOS, Windows Upgrade, Netflix, Spotify, etc. is consuming in order to block/educate the users so that the bandwidth will last expected length.
Most bandwidth packages I have seen only monitors total bandwidth, not to which server.
Can you point me in the right direction of where to look?
This has become more challenging that it was, as many/most of these services use content distribution networks (CDN). The bulk of the data transferred might not come from, for example, a host that resolves to a netflix.com name, but one that resolves to akamai.com or amazon.com (AWS), or ...
If knowing what the requested domain name is, at least for HTTP and some HTTP-S connections then a web proxy is a somewhat complex way to start to track that. Past that, especially with an encrypted service, you're pretty much out of luck in knowing the intent of the connection if the reverse DNS isn't helpful. You'll have to decide how valuable that information is against the complexity, CPU resources, and other costs in collecting it.
For many people, the pattern of usage is about the same over time, so reverse DNS and port number is "enough".
As you say "educate people" it sounds like tracking the source of the connection would be valuable to you as well.
Yes, Netflow (or sFlow, or IPFIX) can give a great amount of info about the sources, destinations and volumes of traffic flowing through your network. The softflowd package will send that info to a separate program that collects the stats and displays them. NfSen is one.
Update: I wrote my initial note quickly, and wasn't quite fair to YAMon. Here's a more considered response.
I like YAMon a lot. The traffic stats are easy to use with a good web-based display. you really can tell "who's hogging the bandwidth", both right now (nice real-time display) and over time ("who used the most last month?") It also lets you enter monthly bandwidth caps so you can see what percentage you've used, and how far into the month you are. The example charts at, say, http://usage-monitoring.com/about.php show the kinds of helpful info that YAMon collects.
You then ssh in and run a script that goes through a number of steps to download and configure the software. It would be great if there were a facility to say, "Just use the defaults..." since they're usually the right thing.
richb-hanover thanks for the mention but ouch!
To be clear, YAMon tracks on a per device basis rather than per destination... Howevr, that said the Live Usage tab should give you some indication of what sites your devices are connecting to...