Shell Expansion

This friday, my friend was developing a shell script in KSH for AIX. In this script, several files with a special naming convention should be selected, but only part of the name should be considered. The format is:

word1-word2-extension-serial1-serial2-name.cmt

Where:

word1: any length

word2: any length 

extension: any length 

serial1: 6 chars 

serial2: 6 chars 

name: any length

So, these name would be ok, for instance:

CONTAB1-PALAVRA2-NMT-123456-123456-gilberto.cmt 

CTB1-PLV2-XYZ-654321-000001-joaozinho.cmt

From those names, everything before the serials should be extracted. That is:

CONTAB1-PALAVRA2-NMT 

CTB1-PLV2-XYZ

At first, the nightmare are the 3 variable length parts and the NAME , which is also variable. But SHELL EXPANSION can beat it. And the best, not only BASH, but it does apply to KSH (I really didn't know that !!!).

So, the solution:

$ var='CONTAB1-PALAVRA2-NMT-123456-123456-gilberto.cmt' 

$ echo ${var%%-??????-??????-*.cmt} 

CONTAB1-PALAVRA2-NMT

The second example:

$ var='CTB1-PLV2-XYZ-654321-000001-joaozinho.cmt' 

$ echo ${var%%-??????-??????-*.cmt} 

CTB1-PLV2-XYZ

Once again, the apparently cryptographic hieroglyphic SHELL did it!!!

:D

Disk full with Squid3

Besides working in an excellent company, I have some networks I help to administer. And recently, one of these friends called me complaining about the internet dropping. Immediately I got a SSH session to his firewall, and started my investigation job.

In fact, SQUID was not working anymore. Starting it was unuseful. Reading it's log gave me the following details:

$ sudo less /var/log/squid3/cache.log

CPU Usage: 0.044 seconds = 0.016 user + 0.028 sys

Maximum Resident Size: 23936 KB

Page faults with physical i/o: 0

2011/02/03 15:16:42| Starting Squid Cache version 3.0.STABLE19 for i486-pc-linux-gnu...

2011/02/03 15:16:42| Process ID 4628

2011/02/03 15:16:42| With 1024 file descriptors available

2011/02/03 15:16:42| Performing DNS Tests...

2011/02/03 15:16:42| Successful DNS name lookup tests...

2011/02/03 15:16:42| DNS Socket created at 0.0.0.0, port 35799, FD 7

2011/02/03 15:16:42| Adding nameserver 192.168.1.9 from squid.conf

2011/02/03 15:16:42| Unlinkd pipe opened on FD 12

2011/02/03 15:16:42| Local cache digest enabled; rebuild/rewrite every 3600/3600 sec

2011/02/03 15:16:42| Swap maxSize 10240000 + 8192 KB, estimated 788322 objects

2011/02/03 15:16:42| Target number of buckets: 39416

2011/02/03 15:16:42| Using 65536 Store buckets

2011/02/03 15:16:42| Max Mem  size: 8192 KB

2011/02/03 15:16:42| Max Swap size: 10240000 KB

2011/02/03 15:16:42| /var/spool/squid3/swap.state.new: (28) No space left on device

FATAL: storeDirOpenTmpSwapLog: Failed to open swap log.

I figured the problem was solved, but surprisingly, space was not the problem, as I could notice:

$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda1             9,2G  2,0G  6,8G  23% /

none                  434M  208K  433M   1% /dev

none                  438M     0  438M   0% /dev/shm

none                  438M   68K  438M   1% /var/run

none                  438M     0  438M   0% /var/lock

none                  438M     0  438M   0% /lib/init/rw

/dev/sda3              63G   42G   18G  70% /var

After some minutes mulling over this problem, I remembered that not only disk space is defined when a partition is formatted, but also the quantity of free inodes. To inspect this, I used df again:

$ df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/sda1             610800   70254  540546   12% /

none                  110874     692  110182    1% /dev

none                  111925       1  111924    1% /dev/shm

none                  111925      36  111889    1% /var/run

none                  111925       2  111923    1% /var/lock

none                  111925       1  111924    1% /lib/init/rw

/dev/sda3            4153344 4153344       0  100% /var

The line above means I have no spare inodes, although there is a considerable free space. I have to look for which directory is consuming the inodes, or in other words, crowded with small files. Since I know it is in /var, there is where my search starts:

$ sudo du -s /var/* | sort -n

That will show the "heaviest" directory. In my case, it is /var/lib. So let's start again:

$ sudo du -s /var/lib/* | sort -n

The directory that was released was /var/lib/sarg, with the following files:

$ ls -l /var/lib/sarg/

total 52

drwxr-xr-x   7 root root  4096 2010-04-12 13:40 2010Apr12-2010Apr12

drwxr-xr-x  48 root root  4096 2010-04-15 04:00 2010Apr14-2010Apr14

drwxr-xr-x  45 root root  4096 2010-04-16 04:00 2010Apr15-2010Apr15

drwxr-xr-x  45 root root  4096 2010-04-16 16:45 2010Apr16-2010Apr16

drwxr-xr-x  41 root root  4096 2010-04-19 12:55 2010Apr19-2010Apr19

drwxr-xr-x  98 root root  4096 2010-09-06 14:22 2010Sep01-2010Sep06

drwxr-xr-x 168 root root 12288 2010-09-30 20:01 Daily

drwxr-xr-x   2 root root  4096 2010-04-12 13:40 images

-rw-r--r--   1 root root   780 2010-09-30 20:00 index.html

drwxr-xr-x   8 root root  4096 2010-09-01 02:38 Montly

drwxr-xr-x  49 root root  4096 2010-09-27 01:03 Weekly

Precisely, these are files done in a previous (unsuccessful installation of sarg). I may also notice that it happened almost one year ago. It was my fault to forget removing old useless files. After removing this directory, the inodes were restaured:

$ df -i

Filesystem            Inodes   IUsed   IFree IUse% Mounted on

/dev/sda1             610800   70254  540546   12% /

none                  110874     692  110182    1% /dev

none                  111925       1  111924    1% /dev/shm

none                  111925      37  111888    1% /var/run

none                  111925       2  111923    1% /var/lock

none                  111925       1  111924    1% /lib/init/rw

/dev/sda3            4153344 1922631 2230713   47% /var

After that, squid could be started.

Installing a brand new Ubuntu box

I am using a Dell Optiplex 780 as my desktop at my job now, and so I do need my environment back, that means, I have to reinstall Ubuntu.

Got a usb memory prepared to install it, asked Windows 7 to release some space in HD (almost 80% of space, in fact), and started the procedure.

Ubuntu 10.10 up and ready, let's start to make it the best I can, and that's the reason why I wrote this post.

Cleaning old lists
First of all, I need to rebuild the apt lists:

sudo rm -v /var/lib/apt/lists/*
sudo apt-get update

This is only needed because for an unknown reason the original lists has some errors. I am really not aware about these mistakes.

Basic tools
Before starting, I need to prepare my surgery tools:

sudo apt-get install vim aptitude htop ssh k3b ssh ipcalc \
vlc gnome-mplayer compizconfig-settings-manager \
compiz-fusion-plugins-extra

Compiz Configs
There are some adjustments I always do in Compiz interface. Of course, they are optional:

as_active_plugins = core;ccp;move;resize;place;decoration;gnomecompat;workarounds;neg;vpswitch;text;svg;ring;regex;commands;dbus;session;imgjpeg;mousepoll;shift;png;reflex;resizeinfo;animation;wobbly;fade;cube;rotate;3d;cubeaddon;scale;scaleaddon;expo;ezoom;
as_next_key = Tab
as_prev_key = Tab
as_initiate_key = Disabled

These configurations may be inserted through compizconfig-settings-manager, instead of dealing with configuration files.

Installing Java on Maverick Meercat

I don't know why, but Java is not available from Canonical repositories anymore. I just came over that because I needed to have my job's desktop reinstalled.

So, These are easy and single steps to provide Java-6 and Jetty application service in my Ubuntu server:

$ sudo aptitude install python-software-properties
$ add-apt-repository "deb http://archive.canonical.com/ lucid partner"
$ sudo aptitude install sun-java6-jdk sun-java6-jre
$ sudo aptitude install libjetty-extra-java libjetty-java jetty

Working fine now!!!

This was edited after reading http://x4nd3m4c.blogspot.com/2011/02/instalando-o-jetty-no-ubuntu-1004.html. Thanks, Alexandre.