I created an automagically configured Jenkins controller. It spun up in Amazon Web Services, it grabbed and installed its private keys for access, and it was stateless. But… But the Jenkins AMI came with a Jenkins config.xml configured with agent AMI ids when it was built. As development moved forward (CICD, after all), the config.xml baked into the Jenkins Ami fell behind. It needed a method to stay up to date.
The section for the agent instance in config.xml looks like
ami-number ec2-agent us-west-1a sg-number T2Medium false ec2-agent builder apply_setup NORMAL echo "init script goes here" 1 ec2-user subnet-number 52 arn:aws:iam::account:instance-profile/profile false false 2147483647 false Name Jenkins Agent t2.medium false false false 2220 2147483647 false false
The description tag follow the ami id. Originally I had a single slave config, but as this was used by developers I found we needed specialty agents – agents with ruby and brent, agents with maven set up and pre-cached, ready to roll, an agent with docker installed and running, etc. Ansible is capable of finding and replacing this kind of entry but only in sequence. Ansible has the lineinfile module, which can use “insert before:” or “insert after:” – but not both. Ansible turned out incapable of doing a global search and replace without tricking it, and especially unable when the unique identifier comes AFTER the line you want to change. This went to bash scripting, and sed…
#! /bin/bash
# ansible can't parse this, so, bash it is...
# read config.xml into an array...
declare -a configarray
#readarray configarray < /var/lib/jenkins/config.xml # Include newline.
IFS=''
readarray -t configarray < /var/lib/jenkins/config.xml # Exclude newline.
# hold value
PREVIOUS=""
# config file
CONFDIR="/var/lib/jenkins"
# tmp config
TMPCONF="/tmp/tmp_config.xml"
# remove if exists
rm -f ${TMPCONF}
# find region...
RAWREG=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`
#echo $RAWREG
REGION=`echo $RAWREG | sed s'/.$//'`
#echo $REGION
# find the AMI values
# nebula base slaveami
NEBBASEAMI=`aws ec2 describe-images --filters "Name=tag:App,Values=agentami" "Name=tag:update_cloud,Values=1" --region ${REGION} | jq -r '.Images | sort_by(.CreationDate) | last(.[]).ImageId'`
#echo $NEBBASEAMI
# ruby nebula base rubyslaveami
RUBYSLAVE=`aws ec2 describe-images --filters "Name=tag:App,Values=rubyagentami" "Name=tag:update_cloud,Values=1" --region ${REGION} | jq -r '.Images | sort_by(.CreationDate) | last(.[]).ImageId'`
#echo $RUBYSLAVE
# node nebula base nodeslaveami
NODESLAVE=`aws ec2 describe-images --filters "Name=tag:App,Values=nodeagentami" "Name=tag:update_cloud,Values=1" --region ${REGION} | jq -r '.Images | sort_by(.CreationDate) | last(.[]).ImageId'`
#echo $NODESLAVE
# maven nebula base mavenslaveami
MAVENSLAVE=`aws ec2 describe-images --filters "Name=tag:App,Values=mavenagentami" "Name=tag:update_cloud,Values=1" --region ${REGION} | jq -r '.Images | sort_by(.CreationDate) | last(.[]).ImageId'`
#echo $MAVENSLAVE
# docker nebula base dockerslaveami
DOCKERSLAVE=`aws ec2 describe-images --filters "Name=tag:App,Values=dockeragentami" "Name=tag:update_cloud,Values=1" --region ${REGION} | jq -r '.Images | sort_by(.CreationDate) | last(.[]).ImageId'`
#echo $DOCKERSLAVE
# go find and edit the lines before each tag...
for CURRENT in "${configarray[@]}"
do
if [[ $CURRENT =~ ^.*\ec2-agent.*$ ]]; then
if [ "x${NEBBASEAMI}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find slave AMI for ec2-slave label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-slave... $CURRENT"
#echo "previous: ${PREVIOUS}"
# replace. ok, this is $CURRENT. We did not place PREVIOUS (last CURRENT) into the file because it was
# so we create a new $PREVIOUS
NEW_PREVIOUS=" ${NEBBASEAMI} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ $CURRENT =~ ^.*\ec2-t2large.*$ ]]; then
if [ "x${NEBBASEAMI}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find agent AMI for ec2-t2large label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-t2large entry... $CURRENT"
#echo "previous: $PREVIOUS"
NEW_PREVIOUS=" ${NEBBASEAMI} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ $CURRENT =~ ^.*\ec2-t2xlarge.*$ ]]; then
if [ "x${NEBBASEAMI}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find agent AMI for ec2-t2xlarge label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-t2xlarge entry... $CURRENT"
#echo "previous: $PREVIOUS"
NEW_PREVIOUS=" ${NEBBASEAMI} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ $CURRENT =~ ^.*\ec2-ruby.*$ ]]; then
if [ "x${RUBYSLAVE}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find agent AMI for ec2-ruby label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-ruby entry... $CURRENT"
#echo "previous: $PREVIOUS"
NEW_PREVIOUS=" ${RUBYSLAVE} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ $CURRENT =~ ^.*\ec2-node.*$ ]]; then
if [ "x${NODESLAVE}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find agent AMI for ec2-node label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-node entry... $CURRENT"
#echo "previous: $PREVIOUS"
NEW_PREVIOUS=" ${NODESLAVE} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ $CURRENT =~ ^.*\ec2-maven.*$ ]]; then
if [ "x${MAVENSLAVE}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find agent AMI for ec2-maven label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-maven entry... $CURRENT"
#echo "previous: $PREVIOUS"
NEW_PREVIOUS=" ${MAVENSLAVE} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ $CURRENT =~ ^.*\ec2-docker.*$ ]]; then
if [ "x${DOCKERSLAVE}" = "x" ]; then
logger -p syslog.warn -t slaveAmiEditConfig.sh "failed to find agent AMI for ec2-docker label"
echo ${PREVIOUS} >> $TMPCONF
else
#echo "found ec2-node entry... $CURRENT"
#echo "previous: $PREVIOUS"
NEW_PREVIOUS=" ${DOCKERSLAVE} "
echo $NEW_PREVIOUS >> $TMPCONF
fi
# now catch up and drop in current...
echo ${CURRENT} >> $TMPCONF
elif [[ ! $CURRENT =~ ^.*\.*\<\/ami\>$ ]]; then
echo $CURRENT >> $TMPCONF
fi
PREVIOUS=${CURRENT}
done
# backup the config.xml to tmp...
DATE=`date +%Y%m%d%H%M%S`
mkdir -p /tmp/updateAgentBackups
cp $CONFDIR/config.xml /tmp/updateAgentBackups/config.xml.$DATE
cp $TMPCONF $CONFDIR/config.xml
# reload the config
/var/lib/jenkins/scripts/reload-config.sh
This runs at boot, immediately updating the config.xml file and then reloading jenkins (NOT restarting – the script called does a reload config rather than a restart). The “update_cloud” tag ensures that only tested AMIs are brought into the config. Valid working AMIs get tagged update_cloud = 1.
It also runs in cron nightly, allowing the continual move forward of Jenkins behavior as development moves forward.
— doug