intuitive engineering

I love good tools.

 I recently wrote a log zipping script specific to the environment I currently work in.  We don't remove logs from the servers (yet – that will change, but has not so far).  The partitions on which the logs live are quite large.  The servers' lifespan is short enough before rebuild or replace to make leaving logs in place possible. In many cases no compressing of logs is needed.  On some servers the traffic and the application do create files that must be compressed to keep free space on the partition acceptable.  

We had an ad hoc script to do this in detail.  The complexity in it was in trying to work around not touching live logs that a process was writing to, tweaking and tuning the regular expressions to miss those active files  and still catch the ones wanted.   Lots of hand work. 

#! /bin/bash

###
### version: 20100414.01
### author:  dsm
###
### Note - "rm -f $i >> $LOG 2>&1" for deleting testing
###        log files is commented out in 2 places
###        to prevent mistake - uncomment if you really
###        want to delete files.  Until
###        uncommented, files that would have been deleted
###        will by logged but not actually
###        deleted from the filesystem...
###

### VARIABLES

# file to alter retention for individual servers
# create the file via something like
#   export MYHOST=`hostname -s`
#   echo "3" > /etc/$MYHOST.AGE.CTCT_gzip_and_cleanup_logs
# This would change the AGE variable to "3"
# this resolves having different files to
# propagate for individual servers or losing the settings
# when a new version is copied out...

MYSELF=`hostname -s`
MYSELF_AGE_FILE="/etc/gzip/$MYSELF.AGE.gzip_and_cleanup_logs"

DATE=`date +%y%m%d%H%M%S`
LOG="/var/logs/gzip_and_cleanup.${DATE}"
date > $LOG # initialize log
if [[ -f $MYSELF_AGE_FILE ]]; then
    AGE=`cat $MYSELF_AGE_FILE`
else
    AGE=7 # files must be older than $AGE days to be gzipped
fi
# debug
#echo "age set to: $AGE"

LSOF="/usr/sbin/lsof"  #location of lsof binary...
if [[ -f $LSOF ]]; then
    echo "found $LSOF, continuing..." >> $LOG
else
    echo "FAILED TO FIND LSOF AT:  $LSOF - EXITING" >> $LOG
    echo "this will have to be corrected before this script will run" >> $LOG
    exit
fi

GZIP="/usr/bin/gzip"
if [[ -f $GZIP ]]; then
    echo "found $GZIP, continuing..." >> $LOG
else
    echo "FAILED TO FIND gzip AT:  $GZIP - EXITING" >> $LOG
    echo "this will have to be corrected before this script will run" >> $LOG
    exit
fi

#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
TESTING="false" # this enables the delele function
              # NOT TO BE ENABLED ON PRODUCTION MACHINES!
KEEP=30 # days to keep, older will be deleted if TESTING set to "true"
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

### MAIN

#
# If TESTING:  Delete logfiles older than $KEEP
#
if [[ $TESTING == "true" ]]; then
    export LOGDIR=/var/logs
    cd ${LOGDIR}
    date >> $LOG
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR DELETE:  TESTING is $TESTING" >> $LOG
    echo "-----------------------------------" >> $LOG
    for i in `ls --color=no | grep -i log | egrep -v .gz$`
    do
        if [[ `find . -name ${i} -mtime +${KEEP}` ]]; then
    	    if [[ `$LSOF $i` ]]; then
	    	echo "found $i older than $KEEP - but found to be opened by a process - NOT deleting" >> $LOG
	    else
                echo "found $i older than $KEEP:  deleting" >> $LOG
                # rm -f $i >> $LOG 2>&1
	    fi
        else
            echo "$i found to be newer than $KEEP:  keeping this file" >> $LOG
        fi
    done
    export LOGDIR=/var/logs/http
    cd ${LOGDIR}
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR DELETE:  TESTING is $TESTING" >> $LOG
    echo "-----------------------------------" >> $LOG
    for i in `ls --color=no | grep -i log | egrep -v .gz$`
    do
        if [[ `find . -name ${i} -mtime +${KEEP}` ]]; then
 	    if [[ `$LSOF $i` ]]; then
                echo "found $i older than $KEEP - but found to be opened by a process - NOT deleting" >> $LOG
            else
                echo "found $i older than $KEEP:  deleting" >> $LOG
                # rm -f $i >> $LOG 2>&1
	    fi
        else
            echo "$i found to be newer than $KEEP:  keeping this file" >> $LOG
        fi
    done
else
    echo "TESTING set to false:  $TESTING:  NOT deleting any log files..." >> $LOG
fi

#
# Begin cleanup with application server logs
#

export LOGDIR=/var/logs
cd ${LOGDIR} >> $LOG 2>&1
date >> $LOG
echo "" >> $LOG
echo "-----------------------------------">> $LOG
echo "start $LOGDIR cleanup" >> $LOG
echo "-----------------------------------" >> $LOG
echo "" >> $LOG
for i in `ls --color=no | grep -i log | egrep -v .gz$`
do
    if [[ `$LSOF ${i}` ]]; then
        echo "" >> $LOG
	echo "found $i OPEN - not handling" >> $LOG
	echo "" >> $LOG
    else
        if [[ `find . -name ${i} -mtime +${AGE}` ]]; then
	    if [[ -f ${i}.gz ]]; then
	        echo "found gzipped file of the same name:  ${i}.gz - not gzipping ${i}..." >> $LOG
	    else
	        echo "gzipping $i as it is older than $AGE..." >> $LOG
	        $GZIP $i >> $LOG 2>&1
	    fi
	else
	    echo "$i is not open but is newer than $AGE, so NOT being gzipped" >> $LOG
        fi
    fi
done
echo "completed cleanup of application logs..." >> $LOG

#
# Continue cleanup with web server logs
#

export LOGDIR=/var/logs/http
if [[ -d $LOGDIR ]]; then # these may not exist on some apps...
    cd ${LOGDIR} >> $LOG 2>&1
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR cleanup" >> $LOG
    echo "-----------------------------------" >> $LOG
    echo "" >> $LOG
    for i in `ls --color=no | grep -i log | egrep -v .gz$ | egrep -v error_log$ | egrep -v access_log$`
    do
        if [[ `$LSOF ${i}` ]]; then
	    echo "" >> $LOG
            echo "found $i OPEN - not handling" >> $LOG
	    echo "" >> $LOG
        else
	    if [[ `find . -name ${i} -mtime +${AGE}` ]]; then
		if [[ -f ${i}.gz ]]; then
	            echo "found gzipped file of the same name:  ${i}.gz - not gzipping ${i}..." >> $LOG
	        else
                    echo "gzipping $i as it is older than $AGE..." >> $LOG
                    $GZIP $i >> $LOG 2>&1
		fi
	    else
                echo "$i is not open but is newer than $AGE, so NOT being gzipped" >> $LOG
            fi
        fi
    done
    echo "completed cleanup of apache logs..." >> $LOG
else
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR cleanup" >> $LOG
    echo "-----------------------------------" >> $LOG
    echo "" >> $LOG
    echo "??? failed to find $LOGDIR - PROBLEM???" >> $LOG
    echo "" >> $LOG
fi

#
# Cleanup logs once processed...
#

# these logs are post-processing - no need to check for lsof and no need to retain unzipped...

export LOGDIR=/var/logs/app
if [[ -d $LOGDIR ]]; then
    cd ${LOGDIR} >> $LOG 2>&1
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start app $LOGDIR cleanup" >> $LOG
    echo "-----------------------------------" >> $LOG
    echo "" >> $LOG
    for i in `ls --color=no | grep open0`
    do
	echo "looking through $LOGDIR/${i}/processed..." >> $LOG
        echo "" >> $LOG
        cd $LOGDIR/${i}/processed
        for j in `ls --color=no | egrep -v .gz$`
	do
	    if [[ -f ${j}.gz ]]; then
                echo "found gzipped file of the same name:  ${j}.gz - not gzipping ${j}..." >> $LOG
            else
                echo "gzipping $j app log..." >> $LOG
                    $GZIP $j >> $LOG 2>&1
            fi
	done
        echo "COMPLETED $LOGDIR/${i}/processed..." >> $LOG
	echo "" >> $LOG
    done
    echo "" >> $LOG
    echo "Processing ${LOGDIR}/processed app logs..." >> $LOG
    echo "" >> $LOG
    cd ${LOGDIR}/processed
    for k in `ls --color=no | egrep -v .gz$`
    do
        if [[ -f ${k}.gz ]]; then
            echo "found gzipped file of the same name:  ${k}.gz - not gzipping ${k}..." >> $LOG
        else
            echo "gzipping $k app log..." >> $LOG
            $GZIP $k >> $LOG 2>&1
        fi
    done
    echo "completed app logs cleanup..." >> $LOG
else
    echo "" >> $LOG
    echo "failed to find $LOGDIR - skipping cleanup of application logs..." >> $LOG
    echo "" >> $LOG
fi 

#### Changelog ####
dsm - edited for broad issue

 

.

My brother Dennis, my youngest brother, the baby of the family for so long, died Saturday 24th April 2010, very suddenly. 

He had a huge loving heart, and leaves an overwhelming hole behind him.  He was there for me without any question or reservation and I hope he begins to know how much he is loved and how much he is missed.  How proud I am of him.  He was a very good man.

Tags: Dennis.

My daughter is in remission from AML type M4.   She is coming off a ventilator she's been on for 38 days now, in an ICU in California. 

I'm leaving tomorrow after six weeks out with her.  That's good – I miss being home, I miss my family in MA.  I am so glad my daughter is better enough that I can see leaving.

But leaving isn't easy, either. 

–doug

Tags: daughter.

Not something you hope to have to hope for…

My daughter has visually more healthy bone marrow than she had 36 days ago when she came into the hospital in crisis.   She has at least managed to knock the cancer back and rebuild a bone marrow that is not diseased.  More detailed tests will look for any trace at all of cancerous marrow.  I'm hoping there will be none to find at all.  That would be remission, the first step toward a life outside of a hospital room.

–doug

Tags: daughter.

I've been willing my daughter to breathe for the last 24 days. 

She came down with leukemia.   Acute, dangerous and sudden leukemia that tried very hard to kill her.  Right now she is on an oscillating ventilator, with two chest tubes, daily dialysis, and a wall of IVs you would have to see to believe.  And she is still breathing. 

I have a log rotation script I put together that works for my new employer, Constant Contact.  I'll put that up soon enough. 

As soon as she is out the other side of all of this.

–doug

Tags: doughter, script.

I found a piece of code in a nagios alerting script that returns “success” matter what is happening with the jboss application it is checking.

This script had been perpetuated as a service alerting script for years in the environs I work in, edited and passed on as working. It reads:

if [failure code here]; then
# return failure to nagios nrpe daemon
else
# return success to nrpe
fi

The test for a failure failed to match, even when the test was looking for a jboss instance not present on the server. Because the code falls through to success, everything looks just fine, all of the time.

The test for failure being incorrect struck me first when I looked through the script. The more basic flaw in logic hit me after. I think it would be true in an alerting script you would NEVER drop through a loop to a final success. The test would be for success, the fall-through to failure. This would have prevented a false sense of security, and the failure to detect success would have been dealt with immediately.

Something like:

if [success code here]; then
# return success to nagios nrpe daemon
else
# return failure nrpe
fi

Obvious. But an epiphany anyway.

–doug

Tags: bash shell, code, nagios.

One of the features of subversion + apache2 is the ability to list repositories – natively. 

SVNParentPath /apps/repos
SVNListParentPath on

Unfortunately, once you restrict the httpd.conf to individual repositories and start handling permissions separately you lose that.  Both of these permissions, set at the top level, at the parent path to the repositories, have to be commented out to have the individual permissions on the directories below take effect.

And listing just the repositories wasn’t enough for what I had in mind – I wanted a read-only table showing an individual developer or a team lead for a project who has what permissions for the subversion repositories. All the users, all the repositories.

Just your basic cgi.

 and the script:

#! /bin/perl -w

sub DisplayTable {
    my $ref_repos = shift;  #@repos
    my $ref_tabledata = shift; # %tabledata
    my $ref_users = shift; # @users
    # dereference
    my @repos = @$ref_repos;
    my %tabledata = %$ref_tabledata;
    my @users = @$ref_users;
    # repos across the top, users down, R or RW for permissions
    # $tabledata{$rep}{$user} = "readwrite";
    # start table
    print ("<table>\n<tbody>\n");
    # table header
    my $cols = ($#repos + 1);
    print ("<tr><td>Users</td><td colspan=\"$cols\">Repositories</td></tr>\n");
    print ("<div id=\"repotitles\"><tr>\n<td>&nbsp;</td>");
    foreach my $rep (@repos) {
        print ("<td>$rep</td>");
    }
    print ("\n</tr>\n</div>\n");
    foreach my $user (@users) {
        print ("<tr>\n<td>$user</td>");
        foreach my $repo (@repos) {
            if ($tabledata{$repo}{$user}) {
                if ($tabledata{$repo}{$user} eq "read") {
                    print ("<td>R</td>");
                } elsif ($tabledata{$repo}{$user} eq "readwrite") {
                    print ("<td>RW</td>");
                }
            } else {
                print ("<td>&nbsp; - &nbsp;</td>");
            }
        }
        print ("\n</tr>\n");
    }
    # finish table
    print ("</tbody>\n</table>\n");
}

Tags: CGI.pm, script, subversion, svn.

One of the tedious tasks in repository administration is managing users over repositories. Who has access to what repository and to what degree (read-only, or write). Subversion over apache2 allows a tremendous amount of control, down to individual directories within the repository. (see "Per Directory Access Control" in the subversion book).

So far I haven’t placed that but I have set up a default deny and then separate htpasswd files for read, or write access permission. This does at times cause TWO passwords to need to be used to first read and then again write to a repository. However, these are cached, so we’ll see how much developers find to complain about in that.

The script svnadmin.pl assumes that a naming convention for the relationship bewtween htpasswd access files and subversion repositories is set so that the htpasswd file is named <repository_name>_read and <repository_name>_write. You need to setthe path to htpasswd, the path to the htpasswd files, and a location for the top level directory of subversion – the parent directory for all repositories. Within the script you’ll also need to set the paths for the css file ($css) and for the header image ($headerimg). This image should be roughly 420 px wide x 200 px high. I used my corporate logo.

The script also uses a username-as-corporate-ID assumption (begins with "a" or "d" or "x", contains up to eight characters), and a reasonable password assumption (at least 8 charcaters, nor more than 12, must contain at least one capital letter and one digit). This is for internal use, not to be exposed, so if you are going to do something like this on ethe internet, you would want to revisit that and lock it down further.

svnadmin.pl

main.css

To run the script – save-file-as and then change the first line to "#! /usr/bin/perl" (or the appropriate path for the perl you want to use). This is basically removing the extra "#" mark. Rename the script to "svnadmin.pl", put in your cgi-bin location, and put the main.css file in an appropriate location. Edit the variables as above.

Screenshots:

Really fairly simple. This started with a script called "htpasswd.pl" which I downloaded and reworked, then adapted to use CGI.pm for multi-screen form presentation. That framework alone was work rediscovering. I used a similar framework with perl DBI and mysql to do fluid reporting on large system installations 9 years ago. It recurs.

— doug

Tags: apache, perl, script, subversion, svn.