Saturday, August 27, 2011

Shell Expansion

This friday, my friend was developing a shell script in KSH for AIX. In this script, several files with a special naming convention should be selected, but only part of the name should be considered. The format is:
word1: any length
word2: any length 
extension: any length 
serial1: 6 chars 
serial2: 6 chars 
name: any length
So, these name would be ok, for instance:
From those names, everything before the serials should be extracted. That is:
At first, the nightmare are the 3 variable length parts and the NAME , which is also variable. But SHELL EXPANSION can beat it. And the best, not only BASH, but it does apply to KSH (I really didn't know that !!!).

So, the solution:
$ var='CONTAB1-PALAVRA2-NMT-123456-123456-gilberto.cmt' 
$ echo ${var%%-??????-??????-*.cmt} 
The second example:
$ var='CTB1-PLV2-XYZ-654321-000001-joaozinho.cmt' 
$ echo ${var%%-??????-??????-*.cmt} 
Once again, the apparently cryptographic hieroglyphic SHELL did it!!!


Friday, February 4, 2011

Disk full with Squid3

Besides working in an excellent company, I have some networks I help to administer. And recently, one of these friends called me complaining about the internet dropping. Immediately I got a SSH session to his firewall, and started my investigation job.

In fact, SQUID was not working anymore. Starting it was unuseful. Reading it's log gave me the following details:
$ sudo less /var/log/squid3/cache.log

CPU Usage: 0.044 seconds = 0.016 user + 0.028 sys
Maximum Resident Size: 23936 KB
Page faults with physical i/o: 0
2011/02/03 15:16:42| Starting Squid Cache version 3.0.STABLE19 for i486-pc-linux-gnu...
2011/02/03 15:16:42| Process ID 4628
2011/02/03 15:16:42| With 1024 file descriptors available
2011/02/03 15:16:42| Performing DNS Tests...
2011/02/03 15:16:42| Successful DNS name lookup tests...
2011/02/03 15:16:42| DNS Socket created at, port 35799, FD 7
2011/02/03 15:16:42| Adding nameserver from squid.conf
2011/02/03 15:16:42| Unlinkd pipe opened on FD 12
2011/02/03 15:16:42| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec
2011/02/03 15:16:42| Swap maxSize 10240000 + 8192 KB, estimated 788322 objects
2011/02/03 15:16:42| Target number of buckets: 39416
2011/02/03 15:16:42| Using 65536 Store buckets
2011/02/03 15:16:42| Max Mem  size: 8192 KB
2011/02/03 15:16:42| Max Swap size: 10240000 KB
2011/02/03 15:16:42| /var/spool/squid3/ (28) No space left on device
FATAL: storeDirOpenTmpSwapLog: Failed to open swap log.
I figured the problem was solved, but surprisingly, space was not the problem, as I could notice:
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9,2G  2,0G  6,8G  23% /
none                  434M  208K  433M   1% /dev
none                  438M     0  438M   0% /dev/shm
none                  438M   68K  438M   1% /var/run
none                  438M     0  438M   0% /var/lock
none                  438M     0  438M   0% /lib/init/rw
/dev/sda3              63G   42G   18G  70% /var
After some minutes mulling over this problem, I remembered that not only disk space is defined when a partition is formatted, but also the quantity of free inodes. To inspect this, I used df again:
$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   70254  540546   12% /
none                  110874     692  110182    1% /dev
none                  111925       1  111924    1% /dev/shm
none                  111925      36  111889    1% /var/run
none                  111925       2  111923    1% /var/lock
none                  111925       1  111924    1% /lib/init/rw
/dev/sda3            4153344 4153344       0  100% /var
The line above means I have no spare inodes, although there is a considerable free space. I have to look for which directory is consuming the inodes, or in other words, crowded with small files. Since I know it is in /var, there is where my search starts:
$ sudo du -s /var/* | sort -n
That will show the "heaviest" directory. In my case, it is /var/lib. So let's start again:
$ sudo du -s /var/lib/* | sort -n
The directory that was released was /var/lib/sarg, with the following files:
$ ls -l /var/lib/sarg/
total 52
drwxr-xr-x   7 root root  4096 2010-04-12 13:40 2010Apr12-2010Apr12
drwxr-xr-x  48 root root  4096 2010-04-15 04:00 2010Apr14-2010Apr14
drwxr-xr-x  45 root root  4096 2010-04-16 04:00 2010Apr15-2010Apr15
drwxr-xr-x  45 root root  4096 2010-04-16 16:45 2010Apr16-2010Apr16
drwxr-xr-x  41 root root  4096 2010-04-19 12:55 2010Apr19-2010Apr19
drwxr-xr-x  98 root root  4096 2010-09-06 14:22 2010Sep01-2010Sep06
drwxr-xr-x 168 root root 12288 2010-09-30 20:01 Daily
drwxr-xr-x   2 root root  4096 2010-04-12 13:40 images
-rw-r--r--   1 root root   780 2010-09-30 20:00 index.html
drwxr-xr-x   8 root root  4096 2010-09-01 02:38 Montly
drwxr-xr-x  49 root root  4096 2010-09-27 01:03 Weekly
Precisely, these are files done in a previous (unsuccessful installation of sarg). I may also notice that it happened almost one year ago. It was my fault to forget removing old useless files. After removing this directory, the inodes were restaured:
$ df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1             610800   70254  540546   12% /
none                  110874     692  110182    1% /dev
none                  111925       1  111924    1% /dev/shm
none                  111925      37  111888    1% /var/run
none                  111925       2  111923    1% /var/lock
none                  111925       1  111924    1% /lib/init/rw
/dev/sda3            4153344 1922631 2230713   47% /var
After that, squid could be started.

Tuesday, January 25, 2011

Installing a brand new Ubuntu box

I am using a Dell Optiplex 780 as my desktop at my job now, and so I do need my environment back, that means, I have to reinstall Ubuntu.

Got a usb memory prepared to install it, asked Windows 7 to release some space in HD (almost 80% of space, in fact), and started the procedure.

Ubuntu 10.10 up and ready, let's start to make it the best I can, and that's the reason why I wrote this post.

Cleaning old lists
First of all, I need to rebuild the apt lists:
sudo rm -v /var/lib/apt/lists/*
sudo apt-get update
This is only needed because for an unknown reason the original lists has some errors. I am really not aware about these mistakes.

Basic tools

Before starting, I need to prepare my surgery tools:
sudo apt-get install vim aptitude htop ssh k3b ssh ipcalc \
vlc gnome-mplayer compizconfig-settings-manager
Compiz Configs
There are some adjustments I always do in Compiz interface. Of course, they are optional:
as_active_plugins = core;ccp;move;resize;place;decoration;gnomecompat;workarounds;neg;vpswitch;text;svg;ring;regex;commands;dbus;session;imgjpeg;mousepoll;shift;png;reflex;resizeinfo;animation;wobbly;fade;cube;rotate;3d;cubeaddon;scale;scaleaddon;expo;ezoom;
as_next_key = Tab
as_prev_key = Tab
as_initiate_key = Disabled
These configurations may be inserted through compizconfig-settings-manager, instead of dealing with configuration files.

Installing Java on Maverick Meercat

I don't know why, but Java is not available from Canonical repositories anymore. I just came over that because I needed to have my job's desktop reinstalled.

So, These are easy and single steps to provide Java-6 and Jetty application service in my Ubuntu server:

$ sudo aptitude install python-software-properties
$ add-apt-repository "deb lucid partner"
$ sudo aptitude install sun-java6-jdk sun-java6-jre
$ sudo aptitude install libjetty-extra-java libjetty-java jetty

Working fine now!!!

This was edited after reading Thanks, Alexandre.