I have a hosts file (19M) that I'd like to use with dnsmasq to block advertising and porn sites. I uploaded this file to my router (/etc) before I realized how large it was. For some reason that worked, even though my router has only 8MB of flash.

So my question is, first of all, why am I able to upload a file to my router that's larger than the available flash memory? Is /etc stored in RAM as opposed to flash? (I have 32MB RAM.)

Second, since this file is so large, is there any way I can use it to block sites with dnsmasq without uploading it to the router? I know one solution would be to have a separate firewall/dns server, but I'd like to do this on the router if possible.

Edit: Okay I'm pretty sure /etc is not in RAM but in flash. Here's my disk usage before upload:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    4.6M    364.0K      4.3M   8% /
/dev/root                 2.3M      2.3M         0 100% /rom
tmpfs                    14.1M     80.0K     14.0M   1% /tmp
/dev/mtdblock3            4.6M    364.0K      4.3M   8% /overlay
overlayfs:/overlay        4.6M    364.0K      4.3M   8% /
tmpfs                   512.0K         0    512.0K   0% /dev

Since /etc isn't listed, I think that means it falls under rootfs, which is mounted on /. And since there's only 4.3M available, there's no way I can store the hosts file there. Which is weird, because after I upload the file my disk usage is:

root@OpenWrt:~# df -h
Filesystem                Size      Used Available Use% Mounted on
rootfs                    4.6M      4.5M    108.0K  98% /
/dev/root                 2.3M      2.3M         0 100% /rom
tmpfs                    14.1M     80.0K     14.0M   1% /tmp
/dev/mtdblock3            4.6M      4.5M    108.0K  98% /overlay
overlayfs:/overlay        4.6M      4.5M    108.0K  98% /
tmpfs                   512.0K         0    512.0K   0% /dev

Now rootfs is almost entirely full, but why isn't it 100% full? And, assuming this is flash, why does it say size is 4.6M as opposed to 8MB?

I guess tmpfs, which is mounted on /tmp, is my RAM. But, in that case, why does it say size is 14.1M instead of 32MB? Can I store the hosts file there or will it get wiped out on reboot?

(Last edited by davidkennedy85 on 23 May 2015, 20:29)

this is how i do it
https://gist.github.com/teffalump/7227752

Yeah that's good stuff. The hosts file you're downloading is only like 500k though.

The flash filesystem uses compression. So an easy compressible file like a hosts file takes far less space than it's raw size.

BTW, if you have the possibility to store the files elsewhere (a NAS, some private webspace, ...) you don't need to store it in flash. It can be dowloaded in ram.

19MB hostfile with 32MB RAM? Sounds a bit tight to me. I would recommend using the most effective lists and shrink the file a bit. No use wasting RAM on ad hosts that might never actually be used. Did you sort it and remove duplicates?

Is that why I can upload a file that's larger than the available space? And does that explain why it says there's only 4.6M flash as opposed to 8MB?

How would that work? I don't think I can I point dnsmasq to an external hosts file.

I'm getting the lists from shallalist.de, so each category is already unique sorted. There's also urlblacklist.com, but their lists are even larger. I'm not sure where else to get a list of porn sites from.

I could use http://winhelp2002.mvps.org/hosts.txt for the list of ad sites, which would cut the file size nearly in half.

Edit: I found another list at http://rlwpx.free.fr/WPFF/hosts.htm but that one is 21MB just for porn!

(Last edited by davidkennedy85 on 24 May 2015, 20:33)

If your goal is a porn filter, why not use a service like OpenDNS?

The other 3.6 MB contains (readonly) bootloader, kernel and squashfs.

No, you can't. But you can write a script to download the hostfile to ram, and send a signal to dnsmasq to read it. (Or restart dnsmasq).

On the other hand, maybe this is impossible. The database has to exist 2 times, once as hostfile, and once as in-memory database in dnsmasq. If the database is indeed 19MB, it won't fit twice in ram.

This is a better idea if you want to offline the processing of DNS Filtering to External Service.
If the whitelist is too big, performance might be affected.
You can also set and increase dns cache size to improve lookup

Because I didn't think of that. Must be getting old or something.

Actually, what I ended up doing is using the blacklist and PAC file at securemecca.com. I think I will use OpenDNS as a failsafe though.

Edit: in case anyone is interested... https://github.com/Pajamaman/blacklist

(Last edited by davidkennedy85 on 28 May 2015, 07:34)