troubleshooting puppet variables

This is a practical use of searching for strings and finding and setting a missing puppet variable.

Error from a puppet run:

err: Could not retrieve catalog from remote server: Error 400 on SERVER:
sysloghost variable must be defined in app_site module at
/etc/puppet/environments/directory/modules/dstributed/app/manifests/log4j.pp:11
on node servernamed
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
[root@servernamed ~]#

This is likely because the site addressed is secondary production (DR) and has fallen out of sync with production.
Production will have the variable set correctly, and someone failed to set it for the secondary site.

I ran, on my working directory, checked out from svn:

for i in `du -ak | egrep -v \.svn | awk '{ print $2 }'`
do
  if [[ `grep -i sysloghost $i` ]]; then
    echo ""
    echo __________________________________________
    echo $i
    echo __________________________________________
    grep -i sysloghost $i
    echo ""
  fi
done

I get, for the file referenced in the error:

__________________________________________
./modules/dist/app/manifests/log4j.pp
__________________________________________
define jbillapp::log4j ($ccenv, $ensure='installed', $sysloghost) {
  if ! $sysloghost {fail ("sysloghost variable must be defined in servername_site module" )}

and down further I find the servername_site module referred to:

__________________________________________
./modules/site/app_site/manifests/staged/servername_app.pp
__________________________________________
    sysloghost      => "${sysloghost}",


__________________________________________
./modules/site/app_site/manifests/staged/serverapp.pp
__________________________________________
    sysloghost      => "${sysloghost}",


__________________________________________
./modules/site/app_site/manifests/staged/serverhostname.pp
__________________________________________
    sysloghost       => "${sysloghost}",


__________________________________________
./modules/site/app_site/manifests/site.pp
__________________________________________
# $sysloghost=['ip', 'ip']

The “aha!” moment is seeing the variable commented out in site.pp…
The question is, what should it be set to?

OK, checking the commented out IP address, that’s local to production, which would be wrong…
I altered the files search to only get files from the secondary site:

for i in `du -ak | egrep -v \.svn | awk '{ print $2 }' | grep site1`

and I find

__________________________________________
./modules/dist/pages/files/envconfig/staged-app.properties
__________________________________________
log4j.appender.SYSLOG.syslogHost=ip
log4j.appender.SYSLOG2.syslogHost=ip


__________________________________________
./modules/dist/pages/files/envconfig/staged-app2.properties
__________________________________________
log4j.appender.SYSLOG.syslogHost=ip
log4j.appender.SYSLOG2.syslogHost=ip


__________________________________________
./modules/dist/pages/files/envconfig/staged-app3.properties
__________________________________________
log4j.appender.SYSLOG.syslogHost=ip
log4j.appender.SYSLOG2.syslogHost=ip

so I am consistently finding the same two IP addresses, and nslookup confirms this is the loghost for this secondary site.

Set that in staged.pp, and try again. Since there are yet other variables commented out in that file, I expect a different error, which I’ll then have to debug…