Many providers are now initiating monthly bandwidth caps. When you go over they start charging by the gigabyte. I am wondering what things people do or are planning to do to mitigate this?
Some of my ideas:
As for monitoring I plan to gather stats. I think iptables might be the best place to gather bandwidth metrics.
With the proliferation of https traffic, the classic proxy use case (unless a large part of your traffic is unencrypted, e.g. linux package updates) doesn't save you any bandwidth anymore.
Good point! Just throwing out ideas at this point. I may also look for new providers.
Another idea....
Run my own DNS cache so that I can change TTL and also make it easy to block wide swaths of abusive ad sites.
I think internet access needs to be considered an infrastructure just like highways/roads/sidewalks and airspace and then it becomes ubiquitous and very cheap and probably fiber everywhere. I'm speaking of USA and I wouldn't be surprised if other countries are already doing this.
luci-app-adblock is a good idea either way, regardless of the bandwidth savings.
I would suggest Pihole DNS also DHCP with a click in a check box, It can run on many versions of Linux either on bare metal or VM.
Not a big fan but: ban any/most video and movie portal using pihole or luci adblock. With limited internet, you sadly have to rely on dvd/bluray.
And run a local nas with nextcloud and hard drive compression (or at least a big hdd on your router and zip everything). Save any file you had to download once as a backup to save traffic. Backup your downloaded games too (steam backups, in other clients like origin just copy pasta the game folder itself).
Or shorter: backup every crap for the entire network there, and made it a clear rule: check first the nas, if not available check how much traffic left. If yes, download.
And well try to download bigger files elsewhere.
Hey there.
I wouldn't do all of that on a router unless the number of devices on your network that actually benefit from this has at least two digits.
Personally I find it increasingly annoying to maintain a remote servers ban list (even if it's my own remote server sitting as far away from me as in my basement) that only holds a common ban list for two people with a total of six computers, smart phones and tablets. You end up adjusting that remote ban list every second day because your brother complains about "why the hell can't I watch that video on this cooking recipes site" (well, it's because their JavaScript crashes because they somehow rely on their ad servers jQuery) or "why can't I read that news site" (well, that's because their pay wall detected you blocking ads so they won't let you unless you either pay or deactivate ad blocking).
I completely dropped every commonly maintained, server side ad handling in favor of individually maintained browser plugins. Works nearly equally nice from a users perspective. Of course it's not as nerd-fancy-ish as we all hope it can be, but realistically that's way less hassle.
Could you please name the region of the world it is that the internet providers start capping internet and some details about the price? My first idea was to start spending a little more and move to an ISP that doesn't cap. But of course I can only tell from my personal situation where both, availability and pricing allows for that.
To me, something like 250GB per user per month is sufficient. For my internet connection -- that I share with my brother -- that would mean I could accept a cap of 750GB per month (so there's a little buffer) if the price is really really worth it. In clear numbers: Today I pay 40€/month for 100mbit down and 15mbit up. I could accept a cap to 750GB per month if the price was around 20€.
Regards,
Stephan.
And with this you ignore every other piece of software and devices not capable of decent ad and telemetry blocking.
They exist a lot of host files in the web, even some regarding blocking video and movie websites. Sure it all depends on how much traffic you have. I have a friend with only 50gb per month for 4 persons, after that he is completely offline (and his isp is still expensiv as fuck). A no go if you need internet for school or work and live in the middle of nowhere.
How far you want to go depend surely on you. Recently i decided myself to go hardcore against telemetry and ads and blocked around 950k domains network wide. Im lucky to have decent internet for myself and my sis (100/40 mbit). Plus im running a network of 5 -untouched- hotspots with around 10-20 clients depending on the time of the day. All for the less fortunate in my neighborhood.
My ISP say the average usage is about 75gb in my country, im killing easily 20x more 
I have setup a DNS on the LAN already. It is quite easy and I can now use TLS for DNS. So I get to block all those ads and the ISP can't collect DNS queries since they're all via TLS to other upstream DNS servers.
This way all hosts on the LAN are protected from all the ads and other forced content. The other good thing is that when friends come over and use wifi they don't have to configure anything it will all just work.
I am a little worried about Steam but not a lot. Sometimes ESO updates are really large.
There are lists of hosts that I include in my DNS config that are maintained by others and I can add my own.
I decided to get one of these to run the DNS on:
http://www.friendlyarm.com/index.php?route=product/product&path=69&product_id=196
Almost looks to good to be true, but I'll find out soon.
I live in the state of Massachusetts which many consider relatively well off. The state is about 150 miles wide. The eastern half of the state has relatively modern infrastructure. The western half of the state still has a number of locations where high speed internet is not available. The state sponsored the installation of a large fiber backbone to connect hospitals and schools a few years back, but (not clear on the specifics) telcos were not interested in the business due to infrastructure costs. The municipality could set up their own infrastructures, but could not get the economies to make it viable, but were constricted to their boundaries.
As I see it business lobbied for favorable regulation, but sparsely populated areas became unserved with effectively no recourse.
You can set quotas on your router to stop or throttle download speed after X downloaded GB are reached.
https://combo.cc/articles/openwrt-upload-quota-and-bandwidth-limiting.html
As for monitoring I plan to gather stats. I think iptables might be the best place to gather bandwidth metrics.
THere are applications to do that, https://wiki.openwrt.org/doc/howto/bwmon but I don't know how well they work.
I used https://github.com/vortex-5/ddwrt-bwmon when I was using a dd-wrt router, I think it can be used on LEDE too.
YAMon (www.usage-monitoring.com) uses iptables to collect statistics and displays them through a web-based GUI that runs on the LEDE router itself. It is well-developed and supported software.
The installation instructions for version 3.3.4 work fine on LEDE after you make two additional tweaks as described in YAMon - per user & per device usage monitoring - #5 by al_c
Try Arch arm, and if im not wrong the ARMv8 generic image should do the job.
@RangerZ yeah good example of why internet access should be treated like a road, everyone should have equal access. It's going to happen. Ironically the ISPs are forcing the issue. As they charge more people will just block more ads, etc. But the content providers hate that so the content providers will lobby for equal access and fiber everywhere. It's always about money. But this time the ISPs lose because they have no content they can only sell their poor service. It's been a long battle of content providers versus the ISPs. But the ISPs will lose this one.
@bobafetthotmail I will look at those. My router has small resources so I need lightweight solutions. Right now I am just looking at these two stat counters:
/sys/devices/virtual/net/br-lan/statistics/rx_bytes
/sys/devices/virtual/net/br-lan/statistics/tx_bytes
Which should be good enough for now and then I use gnuplot to plot a time range, i.e. 1 month.
One concern is that those numbers will roll over but shouldn't be a problem.
@richb-hanover yeah I had a look at YAmon and it looks good but maybe too much for me and my small router. For now I just care about total usage and not per device stats.
@BIGFAT I will look at Arch, I will eventually go with whatever's the smallest.
Update: Apline linux looks small and based on Musl.
I have been using a simple bandwidth monitor that is low in resources and space needed to install, If you are comfortable using SSH it's really easy to install, There are two packages you need to download from GitHub to make it work.
I wrote a quick tutorial for myself for my router but it should work for any router.
http://1289707005.blogspot.ca/2017/09/install-luci-wrtbwmon-and-wrtbwmon-on.html
@sml156, Have not looked at that solution closely. I found the simplest approach was to use what's already there, as I already said.
/sys/devices/virtual/net/br-lan/statistics/rx_bytes
/sys/devices/virtual/net/br-lan/statistics/tx_bytes
That data plus a cron job using cat and logger is working fine for me. The data gets picked up by rsyslogd and then I chart that on the rsyslogd host. So no extra packages needed. Just two cron jobs for rx/tx.