Creative Commons

Creative Commons License
Linux Web Blog by Gilberto Martins is licensed under a Creative Commons Atribuição 2.5 Brasil License. In case of reuse, the name of the author (Gilberto Martins) and the full URL must be mentioned.

Search in LinuxDrops

Loading...

Friday, February 4, 2011

Disk full with Squid3

Besides working in an excellent company, I have some networks I help to administer. And recently, one of these friends called me complaining about the internet dropping. Immediately I got a SSH session to his firewall, and started my investigation job.

In fact, SQUID was not working anymore. Starting it was unuseful. Reading it's log gave me the following details:
$ sudo less /var/log/squid3/cache.log


CPU Usage: 0.044 seconds = 0.016 user + 0.028 sys
Maximum Resident Size: 23936 KB
Page faults with physical i/o: 0
2011/02/03 15:16:42| Starting Squid Cache version 3.0.STABLE19 for i486-pc-linux-gnu...
2011/02/03 15:16:42| Process ID 4628
2011/02/03 15:16:42| With 1024 file descriptors available
2011/02/03 15:16:42| Performing DNS Tests...
2011/02/03 15:16:42| Successful DNS name lookup tests...
2011/02/03 15:16:42| DNS Socket created at 0.0.0.0, port 35799, FD 7
2011/02/03 15:16:42| Adding nameserver 192.168.1.9 from squid.conf
2011/02/03 15:16:42| Unlinkd pipe opened on FD 12
2011/02/03 15:16:42| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2011/02/03 15:16:42| Swap maxSize 10240000 + 8192 KB, estimated 788322 objects
2011/02/03 15:16:42| Target number of buckets: 39416
2011/02/03 15:16:42| Using 65536 Store buckets
2011/02/03 15:16:42| Max Mem  size: 8192 KB
2011/02/03 15:16:42| Max Swap size: 10240000 KB
2011/02/03 15:16:42| /var/spool/squid3/swap.state.new: (28) No space left on device
FATAL: storeDirOpenTmpSwapLog: Failed to open swap log.
I figured the problem was solved, but surprisingly, space was not the problem, as I could notice:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9,2G  2,0G  6,8G  23% /
none                  434M  208K  433M   1% /dev
none                  438M     0  438M   0% /dev/shm
none                  438M   68K  438M   1% /var/run
none                  438M     0  438M   0% /var/lock
none                  438M     0  438M   0% /lib/init/rw
/dev/sda3              63G   42G   18G  70% /var
After some minutes mulling over this problem, I remembered that not only disk space is defined when a partition is formatted, but also the quantity of free inodes. To inspect this, I used df again:
$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   70254  540546   12% /
none                  110874     692  110182    1% /dev
none                  111925       1  111924    1% /dev/shm
none                  111925      36  111889    1% /var/run
none                  111925       2  111923    1% /var/lock
none                  111925       1  111924    1% /lib/init/rw
/dev/sda3            4153344 4153344       0  100% /var
The line above means I have no spare inodes, although there is a considerable free space. I have to look for which directory is consuming the inodes, or in other words, crowded with small files. Since I know it is in /var, there is where my search starts:
$ sudo du -s /var/* | sort -n
That will show the "heaviest" directory. In my case, it is /var/lib. So let's start again:
$ sudo du -s /var/lib/* | sort -n
The directory that was released was /var/lib/sarg, with the following files:
$ ls -l /var/lib/sarg/
total 52
drwxr-xr-x   7 root root  4096 2010-04-12 13:40 2010Apr12-2010Apr12
drwxr-xr-x  48 root root  4096 2010-04-15 04:00 2010Apr14-2010Apr14
drwxr-xr-x  45 root root  4096 2010-04-16 04:00 2010Apr15-2010Apr15
drwxr-xr-x  45 root root  4096 2010-04-16 16:45 2010Apr16-2010Apr16
drwxr-xr-x  41 root root  4096 2010-04-19 12:55 2010Apr19-2010Apr19
drwxr-xr-x  98 root root  4096 2010-09-06 14:22 2010Sep01-2010Sep06
drwxr-xr-x 168 root root 12288 2010-09-30 20:01 Daily
drwxr-xr-x   2 root root  4096 2010-04-12 13:40 images
-rw-r--r--   1 root root   780 2010-09-30 20:00 index.html
drwxr-xr-x   8 root root  4096 2010-09-01 02:38 Montly
drwxr-xr-x  49 root root  4096 2010-09-27 01:03 Weekly
Precisely, these are files done in a previous (unsuccessful installation of sarg). I may also notice that it happened almost one year ago. It was my fault to forget removing old useless files. After removing this directory, the inodes were restaured:
$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   70254  540546   12% /
none                  110874     692  110182    1% /dev
none                  111925       1  111924    1% /dev/shm
none                  111925      37  111888    1% /var/run
none                  111925       2  111923    1% /var/lock
none                  111925       1  111924    1% /lib/init/rw
/dev/sda3            4153344 1922631 2230713   47% /var
After that, squid could be started.

0 comentários: