Can squid cache to a file instead of a device on OpenWrt?

Hello

I've installed on my raspberry pi running openwrt according to the guide: https://openwrt.org/docs/guide-user/services/proxy/proxy.squid?s[]=squid

At the section entitled storage configuration it recommends editing a config file to do with where Squid caches to. However in my set up I have an external disk drive already mounted to OpenWrt and have created a directory called Squid and point Squid to that via Luci.

Even thought I've followed the rest of the directions, squid does not appear to be caching. Can squid cache to a file instead of a device?

Also, I've updated the firewall per the instructions and see errors:

root@OpenWrt:/mnt/sda/squid# /etc/init.d/firewall reload
Warning: Unable to locate ipset utility, disabling ipset support
Warning: Section @zone[1] (wan) cannot resolve device of network 'wan'
Warning: Section @zone[1] (wan) cannot resolve device of network 'wan6'

I'm not sure if these errors are to do with Squid not caching or not. Can anyone shed a bit of light on the situation?

The firewall errors are not connected to the squid.
What is the output of:
mount; df -h; uci export fstab; cat /etc/squid/squid.conf

Default for squid is to cache to disk and needs a directory path in its config.
There might be the possibility, that /etc/config/squid overwrites /etc/squid/squid.conf
To be on safe side, I always start squid explicitly, i.e. from /etc/rc.local, to make shure, that only /etc/squid/squid.conf taken into consideration.

Thanks @reinerotto

I did some research and learnt something about how linux sees the physical device as sd, the partitions are then sd, the partitions are formatted with a file system and then finally mounted to a mount point, such as /tmp/squid.

I learnt a few other quirks of the system along the way, such as how the letter assigned to the designation ‘sd’ can change depending upon the order in which said devices are added to the system. This has cleared up a great deal confusion for me in particular it shed light on why the use of UUID is preferable to the human-friendly block/partition name when creating configs.

So what I was able to do was simply my set up, partitioning a micro-SD card into sda1 and sda2 with the latter getting 2/3rds of the space and have mounted those partitions to the appropriate places.

Still not caching however. I’m not sure why. But I’m confident that my config for this part of it is correct.

Hello @trendy and thanks for replying, sorry it took me a while to respond, I was auditing my work.

Output of mount:


root@router:~# mount
/dev/root on / type ext4 (rw,noatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,noatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,noatime)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
/dev/mmcblk0p1 on /boot type vfat (rw,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /dev type tmpfs (rw,nosuid,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,mode=600,ptmxmode=000)
/dev/sda2 on /tmp/squid type ext4 (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,noatime,mode=700)
/dev/sda1 on /mnt/graphs type ext4 (rw,relatime)
root@router:~# 

Output of df -h:


root@router:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               102.4M     32.3M     68.0M  32% /
tmpfs                     3.8G      1.3M      3.8G   0% /tmp
/dev/mmcblk0p1           63.9M     15.7M     48.2M  25% /boot
tmpfs                   512.0K         0    512.0K   0% /dev
/dev/sda2                21.8G     44.0M     20.6G   0% /tmp/squid
/dev/sda1                 7.3G     35.9M      6.8G   1% /mnt/graphs
root@router:~#

Output of uci export fstab:


root@router:~# uci export fstab
package fstab

config global
        option anon_swap '0'
        option anon_mount '0'
        option auto_swap '1'
        option auto_mount '1'
        option delay_root '5'
        option check_fs '1'

config mount
        option target '/boot'
        option uuid '43B6-75E3'
        option enabled '0'

config mount
        option target '/'
        option uuid 'ff313567-e9f1-5a5d-9895-3ba130b4a864'
        option enabled '0'

config mount
        option target '/mnt/graphs'
        option uuid '8598f689-77ad-42fe-9dbb-bd199cd9a5bf'
        option enabled '1'

config mount
        option target '/tmp/squid'
        option uuid '06664da2-d72c-497d-be27-cc24b771cafb'
        option enabled '1'
        option fstype 'ext4'
        option enabled_fsck '1'

root@router:~#

Output of cat /etc/squid/squid.conf:


root@router:~# cat /etc/squid/squid.conf
#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 0.0.0.1-0.255.255.255  # RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8             # RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10          # RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16         # RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12          # RFC 1918 local private network (LAN)
#acl localnet src 192.168.0.0/16                # RFC 1918 local private network (LAN)
acl localnet src 192.168.1.0/24         # home network
acl localnet src fc00::/7               # RFC 4193 local private network range
acl localnet src fe80::/10              # RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256
cache_dir aufs /tmp/squid/cache 900 16 512

# If you have 64 MB device RAM you can use 16 MB cache_mem, default is 8 MB
cache_mem 8 MB             
maximum_object_size_in_memory 100 KB
maximum_object_size 32 MB

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern .               0       20%     4320

# Squid user
cache_effective_user squid

#
# Logs, best to use only for debugging as they can become very large
#

access_log none  # daemon:/tmp/squid_access.log
cache_log /dev/null  # /tmp/squid_cache.log
root@router:~# 

I also ran the commands from the section entitled Creating the cache directory under ‘Configuration in https://openwrt.org/docs/guide-user/services/proxy/proxy.squid

The commands/config threw no errors except I was unsure what to make of the part that reads:


### Create swap directories under the cache directory and then exit...
  /opt/usr/sbin/squid -z

When I ran mkdir /opt/use/s in/squid -z the system complains


root@router:/opt# /opt/usr/sbin/squid -z
-ash: /opt/usr/sbin/squid: not found

I tried creating it with a series of mkdir commands and then ran:


du -m /opt/var/cache/squid/ | sort -nr | head -n17 | tail -n16

But seemingly the cache never populates.


root@router:/opt# du -m /opt/var/cache/squid/ | sort -nr | head -n17 | tail -n16
0       /opt/var/cache/squid/
root@router:/opt#

Path is wrong. squid executable on openwrt is not located there. Following should be sufficient:
root@router:/opt# squid -z

However, this most likely will error out, too, because of
cache_effective_user squid
which will not allow creation of the directories. Unless, you modify the
access rights. Something like
mkdir -p /tmp/squid/cache
chmod 0777 /tmp/squid/cache #Not for the paranoid openwrt fan ...

It might be a good idea to use
cache_log /tmp/squid_cache.log #To catch most important (error)logs

I.g. because of complexity of correct squid configuration, I always recommend first to set up a working squid on a standard linux, i.e. ubuntu (VM ?). And then to port to openwrt, to take care of openwrts specialities.

1 Like

squid is not installed there.
opkg files squid

Thanks again @reinerotto

I ran the command as you suggested and did see errors. After creating those directory and changing the perms on them I re-ran the command squid -z and saw this output which hung at the last line.

oot@router:/opt# squid -z
2022/06/19 11:11:38| Created PID file (/var/run/squid.pid)
root@router:/opt# 2022/06/19 11:11:38 kid1| Current Directory is /opt
2022/06/19 11:11:39 kid1| Creating missing swap directories
2022/06/19 11:11:39 kid1| /tmp/squid/cache exists
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/00
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/01
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/02
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/03
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/04
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/05
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/06
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/07
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/08
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/09
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/0A
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/0B
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/0C
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/0D
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/0E
2022/06/19 11:11:39 kid1| Making directories in /tmp/squid/cache/0F
2022/06/19 11:11:39| Removing PID file (/var/run/squid.pid)

Is it the norm for it to hang like that?

after letting the router run for a day I re-ran df and saw no changes to the cache size.

I was mistaken to think that installing and administering Squid would be non-trivial. Perhaps I will un-install squid and do as you suggest and get another device to practice setting it up on. If I go down that road, is there anything I should know to ensure the uninstall goes smoothly?

The messages confirm, that squid successfully created the dirs, required for caching. Which usually is a one-time activity, unless you delete /tmp/squid/cache/* . Or the cache-dir is changed in squid.conf .
If I remember correctly, squid exists after "squid -z", so you have to do a "standard" start of squid (now). Probably, simply typing "squid" should be suffcient now, assuming, /etc/squid/squid.conf exists. BUT there might be some more error messages in /tmp/squid.log, i.g. in case access rights are not properly set.