Friday, February 4, 2011

Disk full with Squid3

Besides working in an excellent company, I have some networks I help to administer. And recently, one of these friends called me complaining about the internet dropping. Immediately I got a SSH session to his firewall, and started my investigation job.

In fact, SQUID was not working anymore. Starting it was unuseful. Reading it's log gave me the following details:
$ sudo less /var/log/squid3/cache.log


CPU Usage: 0.044 seconds = 0.016 user + 0.028 sys
Maximum Resident Size: 23936 KB
Page faults with physical i/o: 0
2011/02/03 15:16:42| Starting Squid Cache version 3.0.STABLE19 for i486-pc-linux-gnu...
2011/02/03 15:16:42| Process ID 4628
2011/02/03 15:16:42| With 1024 file descriptors available
2011/02/03 15:16:42| Performing DNS Tests...
2011/02/03 15:16:42| Successful DNS name lookup tests...
2011/02/03 15:16:42| DNS Socket created at 0.0.0.0, port 35799, FD 7
2011/02/03 15:16:42| Adding nameserver 192.168.1.9 from squid.conf
2011/02/03 15:16:42| Unlinkd pipe opened on FD 12
2011/02/03 15:16:42| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2011/02/03 15:16:42| Swap maxSize 10240000 + 8192 KB, estimated 788322 objects
2011/02/03 15:16:42| Target number of buckets: 39416
2011/02/03 15:16:42| Using 65536 Store buckets
2011/02/03 15:16:42| Max Mem  size: 8192 KB
2011/02/03 15:16:42| Max Swap size: 10240000 KB
2011/02/03 15:16:42| /var/spool/squid3/swap.state.new: (28) No space left on device
FATAL: storeDirOpenTmpSwapLog: Failed to open swap log.
I figured the problem was solved, but surprisingly, space was not the problem, as I could notice:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9,2G  2,0G  6,8G  23% /
none                  434M  208K  433M   1% /dev
none                  438M     0  438M   0% /dev/shm
none                  438M   68K  438M   1% /var/run
none                  438M     0  438M   0% /var/lock
none                  438M     0  438M   0% /lib/init/rw
/dev/sda3              63G   42G   18G  70% /var
After some minutes mulling over this problem, I remembered that not only disk space is defined when a partition is formatted, but also the quantity of free inodes. To inspect this, I used df again:
$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   70254  540546   12% /
none                  110874     692  110182    1% /dev
none                  111925       1  111924    1% /dev/shm
none                  111925      36  111889    1% /var/run
none                  111925       2  111923    1% /var/lock
none                  111925       1  111924    1% /lib/init/rw
/dev/sda3            4153344 4153344       0  100% /var
The line above means I have no spare inodes, although there is a considerable free space. I have to look for which directory is consuming the inodes, or in other words, crowded with small files. Since I know it is in /var, there is where my search starts:
$ sudo du -s /var/* | sort -n
That will show the "heaviest" directory. In my case, it is /var/lib. So let's start again:
$ sudo du -s /var/lib/* | sort -n
The directory that was released was /var/lib/sarg, with the following files:
$ ls -l /var/lib/sarg/
total 52
drwxr-xr-x   7 root root  4096 2010-04-12 13:40 2010Apr12-2010Apr12
drwxr-xr-x  48 root root  4096 2010-04-15 04:00 2010Apr14-2010Apr14
drwxr-xr-x  45 root root  4096 2010-04-16 04:00 2010Apr15-2010Apr15
drwxr-xr-x  45 root root  4096 2010-04-16 16:45 2010Apr16-2010Apr16
drwxr-xr-x  41 root root  4096 2010-04-19 12:55 2010Apr19-2010Apr19
drwxr-xr-x  98 root root  4096 2010-09-06 14:22 2010Sep01-2010Sep06
drwxr-xr-x 168 root root 12288 2010-09-30 20:01 Daily
drwxr-xr-x   2 root root  4096 2010-04-12 13:40 images
-rw-r--r--   1 root root   780 2010-09-30 20:00 index.html
drwxr-xr-x   8 root root  4096 2010-09-01 02:38 Montly
drwxr-xr-x  49 root root  4096 2010-09-27 01:03 Weekly
Precisely, these are files done in a previous (unsuccessful installation of sarg). I may also notice that it happened almost one year ago. It was my fault to forget removing old useless files. After removing this directory, the inodes were restaured:
$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   70254  540546   12% /
none                  110874     692  110182    1% /dev
none                  111925       1  111924    1% /dev/shm
none                  111925      37  111888    1% /var/run
none                  111925       2  111923    1% /var/lock
none                  111925       1  111924    1% /lib/init/rw
/dev/sda3            4153344 1922631 2230713   47% /var
After that, squid could be started.