intuitive engineering

Leaf at St. Joseph’s Hospital by Ashley Lyn Munsinger

When Ashley was born, on a bright day in July,
her nickname was immediately “Bug,”
as in “cute as a bug.”

I went in to watch her sleep in her crib almost every night after we brought her home.
I watched her very small chest rise and fall.
That was a miracle.

Having this child meant being completely vulnerable to the whims of the universe.

I cast hope out into the future.
I get to watch over her and love her, forever.

When she was about three Ashley got quieter and quieter over several days.
Her doctor said she had pneumonia
and was hours from needing admittance to a hospital.

She got better.
But she scared me.

She rode her bike without training wheels in Lake View Terrace, CA.
She helped me build a coffee table in Boulder CO.
She sledded down the hill with our dog Samantha at the house in Milford MA.

She was golden blonde,
sunshine
and warmth.

She faded the winter she spent in Massachusetts,
so I sent her back to California.

Ashley traveled through Europe when she was fifteen.
She determined that Corfu was her favorite place on the planet.
She talked about that trip for years…

“When I was in France…”

Ashley studied International Business in New York City
She found she hated it.
She discovered space and color and form and balance and function.
She studied Interior Design, and she found she loved it.

She got quiet once again, this time over several months.
She was admitted to the hospital and we got word that she had leukemia.

By the time I got across the country from Boston to Los Angeles,
Ashley was on a ventilator and could no longer speak or communicate directly.

All of us,
all of you,
family and friends,
came together
and somehow we got her through the worst two months ever.

At the lowest point in that February I was holding Ash’s hand,
feeling the warmth of her still presence,
accepting each moment as a gift.

That diagnosis
and then that reality,
cancer patient,
Ash just accepted.

She looked ahead with true courage at each point in her journey
and asked,
“What next?”

Ashley knew she was dying that last week.
She found
the rare gift
in that
rather than the despair of it,
that ending.

I tried to find everything I needed to say to her, as her dad, as someone responsible for her and to her.
I came close enough.

She managed a death
at peace
and without despair or anxiety.
Her passing on was with eyes wide open.
She said “It’s over…”
and she meant this life, this struggle.

And then she asked
“What’s next?”
and she went on.

An expansion of being, rather than in any way diminished.

This breath holds the spirit to the body.

And so it was.
Ash let go
and the breath stopped.

July 2nd 1986 – 3rd January 2011

I will always miss her presence here.

Koi Pond at St. Joseph’s Hospital by Ashley Lyn Munsinger

This is from an email I wrote to steve@apple.com:

…she died January 3rd. 

 

My iphone HAD been a reliable alarm.  I missed the daylight savings update debacle, at least for anything vital. 

I set a 3:30 AM and a 4 AM alarm for getting up to catch a 6:30 flight from Boston to Los Angeles on Sunday 2 January, which would have had me in LA at 10:30. 

I woke at 5:19 AM, looked at my iphone (which showed both alarms set) in horror, raced to the airport and made it to the gate at 6:35.  The plane was gone. 

 

The woman behind the counter at United at first said there were no more seats available on any plane to get to L.A. on January 2nd.  

 

After I explained that my daughter was terminal from Leukemia she found me a flight to Washington DC and then to LA. 

I got to L.A. at 7:30 PM and spent from 8  to 11:30 with my daughter in the cancer ward on the fourth floor at Cedars-Sinai. 

She died at 11:02 AM on January  3rd.   She was 24 and had battled leukemia for one week short of a year.   We had been told by doctors that she had perhaps another couple of weeks, but it turned out that wasn't so. 

 

If I had listened to the woman behind the counter I would not have been there at my daughter's side.  If your alarm had actually been fixed and functioned January 2nd, I would have had an additional 9 to 10 hours with my daughter on the last two days of her life. 

 

There's no way I can begin to count what the iphone bug managed to cost me.  

But I thought you should know that it was not trivial, not a simple bug that has no impact.  

–doug

I love good tools.

 I recently wrote a log zipping script specific to the environment I currently work in.  We don't remove logs from the servers (yet – that will change, but has not so far).  The partitions on which the logs live are quite large.  The servers' lifespan is short enough before rebuild or replace to make leaving logs in place possible. In many cases no compressing of logs is needed.  On some servers the traffic and the application do create files that must be compressed to keep free space on the partition acceptable.  

We had an ad hoc script to do this in detail.  The complexity in it was in trying to work around not touching live logs that a process was writing to, tweaking and tuning the regular expressions to miss those active files  and still catch the ones wanted.   Lots of hand work. 

#! /bin/bash

###
### version: 20100414.01
### author:  dsm
###
### Note - "rm -f $i >> $LOG 2>&1" for deleting testing
###        log files is commented out in 2 places
###        to prevent mistake - uncomment if you really
###        want to delete files.  Until
###        uncommented, files that would have been deleted
###        will by logged but not actually
###        deleted from the filesystem...
###

### VARIABLES

# file to alter retention for individual servers
# create the file via something like
#   export MYHOST=`hostname -s`
#   echo "3" > /etc/$MYHOST.AGE.CTCT_gzip_and_cleanup_logs
# This would change the AGE variable to "3"
# this resolves having different files to
# propagate for individual servers or losing the settings
# when a new version is copied out...

MYSELF=`hostname -s`
MYSELF_AGE_FILE="/etc/gzip/$MYSELF.AGE.gzip_and_cleanup_logs"

DATE=`date +%y%m%d%H%M%S`
LOG="/var/logs/gzip_and_cleanup.${DATE}"
date > $LOG # initialize log
if [[ -f $MYSELF_AGE_FILE ]]; then
    AGE=`cat $MYSELF_AGE_FILE`
else
    AGE=7 # files must be older than $AGE days to be gzipped
fi
# debug
#echo "age set to: $AGE"

LSOF="/usr/sbin/lsof"  #location of lsof binary...
if [[ -f $LSOF ]]; then
    echo "found $LSOF, continuing..." >> $LOG
else
    echo "FAILED TO FIND LSOF AT:  $LSOF - EXITING" >> $LOG
    echo "this will have to be corrected before this script will run" >> $LOG
    exit
fi

GZIP="/usr/bin/gzip"
if [[ -f $GZIP ]]; then
    echo "found $GZIP, continuing..." >> $LOG
else
    echo "FAILED TO FIND gzip AT:  $GZIP - EXITING" >> $LOG
    echo "this will have to be corrected before this script will run" >> $LOG
    exit
fi

#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
TESTING="false" # this enables the delele function
              # NOT TO BE ENABLED ON PRODUCTION MACHINES!
KEEP=30 # days to keep, older will be deleted if TESTING set to "true"
#!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

### MAIN

#
# If TESTING:  Delete logfiles older than $KEEP
#
if [[ $TESTING == "true" ]]; then
    export LOGDIR=/var/logs
    cd ${LOGDIR}
    date >> $LOG
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR DELETE:  TESTING is $TESTING" >> $LOG
    echo "-----------------------------------" >> $LOG
    for i in `ls --color=no | grep -i log | egrep -v .gz$`
    do
        if [[ `find . -name ${i} -mtime +${KEEP}` ]]; then
    	    if [[ `$LSOF $i` ]]; then
	    	echo "found $i older than $KEEP - but found to be opened by a process - NOT deleting" >> $LOG
	    else
                echo "found $i older than $KEEP:  deleting" >> $LOG
                # rm -f $i >> $LOG 2>&1
	    fi
        else
            echo "$i found to be newer than $KEEP:  keeping this file" >> $LOG
        fi
    done
    export LOGDIR=/var/logs/http
    cd ${LOGDIR}
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR DELETE:  TESTING is $TESTING" >> $LOG
    echo "-----------------------------------" >> $LOG
    for i in `ls --color=no | grep -i log | egrep -v .gz$`
    do
        if [[ `find . -name ${i} -mtime +${KEEP}` ]]; then
 	    if [[ `$LSOF $i` ]]; then
                echo "found $i older than $KEEP - but found to be opened by a process - NOT deleting" >> $LOG
            else
                echo "found $i older than $KEEP:  deleting" >> $LOG
                # rm -f $i >> $LOG 2>&1
	    fi
        else
            echo "$i found to be newer than $KEEP:  keeping this file" >> $LOG
        fi
    done
else
    echo "TESTING set to false:  $TESTING:  NOT deleting any log files..." >> $LOG
fi

#
# Begin cleanup with application server logs
#

export LOGDIR=/var/logs
cd ${LOGDIR} >> $LOG 2>&1
date >> $LOG
echo "" >> $LOG
echo "-----------------------------------">> $LOG
echo "start $LOGDIR cleanup" >> $LOG
echo "-----------------------------------" >> $LOG
echo "" >> $LOG
for i in `ls --color=no | grep -i log | egrep -v .gz$`
do
    if [[ `$LSOF ${i}` ]]; then
        echo "" >> $LOG
	echo "found $i OPEN - not handling" >> $LOG
	echo "" >> $LOG
    else
        if [[ `find . -name ${i} -mtime +${AGE}` ]]; then
	    if [[ -f ${i}.gz ]]; then
	        echo "found gzipped file of the same name:  ${i}.gz - not gzipping ${i}..." >> $LOG
	    else
	        echo "gzipping $i as it is older than $AGE..." >> $LOG
	        $GZIP $i >> $LOG 2>&1
	    fi
	else
	    echo "$i is not open but is newer than $AGE, so NOT being gzipped" >> $LOG
        fi
    fi
done
echo "completed cleanup of application logs..." >> $LOG

#
# Continue cleanup with web server logs
#

export LOGDIR=/var/logs/http
if [[ -d $LOGDIR ]]; then # these may not exist on some apps...
    cd ${LOGDIR} >> $LOG 2>&1
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR cleanup" >> $LOG
    echo "-----------------------------------" >> $LOG
    echo "" >> $LOG
    for i in `ls --color=no | grep -i log | egrep -v .gz$ | egrep -v error_log$ | egrep -v access_log$`
    do
        if [[ `$LSOF ${i}` ]]; then
	    echo "" >> $LOG
            echo "found $i OPEN - not handling" >> $LOG
	    echo "" >> $LOG
        else
	    if [[ `find . -name ${i} -mtime +${AGE}` ]]; then
		if [[ -f ${i}.gz ]]; then
	            echo "found gzipped file of the same name:  ${i}.gz - not gzipping ${i}..." >> $LOG
	        else
                    echo "gzipping $i as it is older than $AGE..." >> $LOG
                    $GZIP $i >> $LOG 2>&1
		fi
	    else
                echo "$i is not open but is newer than $AGE, so NOT being gzipped" >> $LOG
            fi
        fi
    done
    echo "completed cleanup of apache logs..." >> $LOG
else
    echo "-----------------------------------">> $LOG
    echo "start $LOGDIR cleanup" >> $LOG
    echo "-----------------------------------" >> $LOG
    echo "" >> $LOG
    echo "??? failed to find $LOGDIR - PROBLEM???" >> $LOG
    echo "" >> $LOG
fi

#
# Cleanup logs once processed...
#

# these logs are post-processing - no need to check for lsof and no need to retain unzipped...

export LOGDIR=/var/logs/app
if [[ -d $LOGDIR ]]; then
    cd ${LOGDIR} >> $LOG 2>&1
    echo "" >> $LOG
    echo "-----------------------------------">> $LOG
    echo "start app $LOGDIR cleanup" >> $LOG
    echo "-----------------------------------" >> $LOG
    echo "" >> $LOG
    for i in `ls --color=no | grep open0`
    do
	echo "looking through $LOGDIR/${i}/processed..." >> $LOG
        echo "" >> $LOG
        cd $LOGDIR/${i}/processed
        for j in `ls --color=no | egrep -v .gz$`
	do
	    if [[ -f ${j}.gz ]]; then
                echo "found gzipped file of the same name:  ${j}.gz - not gzipping ${j}..." >> $LOG
            else
                echo "gzipping $j app log..." >> $LOG
                    $GZIP $j >> $LOG 2>&1
            fi
	done
        echo "COMPLETED $LOGDIR/${i}/processed..." >> $LOG
	echo "" >> $LOG
    done
    echo "" >> $LOG
    echo "Processing ${LOGDIR}/processed app logs..." >> $LOG
    echo "" >> $LOG
    cd ${LOGDIR}/processed
    for k in `ls --color=no | egrep -v .gz$`
    do
        if [[ -f ${k}.gz ]]; then
            echo "found gzipped file of the same name:  ${k}.gz - not gzipping ${k}..." >> $LOG
        else
            echo "gzipping $k app log..." >> $LOG
            $GZIP $k >> $LOG 2>&1
        fi
    done
    echo "completed app logs cleanup..." >> $LOG
else
    echo "" >> $LOG
    echo "failed to find $LOGDIR - skipping cleanup of application logs..." >> $LOG
    echo "" >> $LOG
fi 

#### Changelog ####
dsm - edited for broad issue

 

My brother Dennis, my youngest brother, the baby of the family for so long, died Saturday 24th April 2010, very suddenly. 

He had a huge loving heart, and leaves an overwhelming hole behind him.  He was there for me without any question or reservation and I hope he begins to know how much he is loved and how much he is missed.  How proud I am of him.  He was a very good man.

My daughter is in remission from AML type M4.   She is coming off a ventilator she's been on for 38 days now, in an ICU in California. 

I'm leaving tomorrow after six weeks out with her.  That's good – I miss being home, I miss my family in MA.  I am so glad my daughter is better enough that I can see leaving.

But leaving isn't easy, either. 

–doug

Not something you hope to have to hope for…

My daughter has visually more healthy bone marrow than she had 36 days ago when she came into the hospital in crisis.   She has at least managed to knock the cancer back and rebuild a bone marrow that is not diseased.  More detailed tests will look for any trace at all of cancerous marrow.  I'm hoping there will be none to find at all.  That would be remission, the first step toward a life outside of a hospital room.

–doug

I've been willing my daughter to breathe for the last 24 days. 

She came down with leukemia.   Acute, dangerous and sudden leukemia that tried very hard to kill her.  Right now she is on an oscillating ventilator, with two chest tubes, daily dialysis, and a wall of IVs you would have to see to believe.  And she is still breathing. 

I have a log rotation script I put together that works for my new employer, Constant Contact.  I'll put that up soon enough. 

As soon as she is out the other side of all of this.

–doug

I found a piece of code in a nagios alerting script that returns “success” matter what is happening with the jboss application it is checking.

This script had been perpetuated as a service alerting script for years in the environs I work in, edited and passed on as working. It reads:

if [failure code here]; then
# return failure to nagios nrpe daemon
else
# return success to nrpe
fi

The test for a failure failed to match, even when the test was looking for a jboss instance not present on the server. Because the code falls through to success, everything looks just fine, all of the time.

The test for failure being incorrect struck me first when I looked through the script. The more basic flaw in logic hit me after. I think it would be true in an alerting script you would NEVER drop through a loop to a final success. The test would be for success, the fall-through to failure. This would have prevented a false sense of security, and the failure to detect success would have been dealt with immediately.

Something like:

if [success code here]; then
# return success to nagios nrpe daemon
else
# return failure nrpe
fi

Obvious. But an epiphany anyway.

–doug