Intuitive Engineering

SED (stream editor)

doug — Wed, 16 May 2012 11:14:15 +0000

SED is the coolest UNIX command ever. In the class I had at WPI that covered SED the professor had no idea how powerful and useful this command can become.

I need to add an exception to apache.conf files that allows checking a status jsp directly through port 80. In the config files for these applications there is a re-write command that pushes all traffic through https.

It looks like:

    #Force all traffic through secure http
    RewriteCond %{HTTP:SSLClientCipher} !^..*$
    RewriteCond %{HTTP_HOST} !^localhost* [NC]
    RewriteCond %{REQUEST_URI} !/fw3?\.jsp
    RewriteCond %{REQUEST_URI} !/d\.jsp
    RewriteCond %{REQUEST_URI} !^/support/Data\.jsp
    RewriteCond %{REQUEST_URI} !^.*(\r|\n|%0A|%0D).* [NC]
    RewriteRule ^(.*) https://%{HTTP_HOST}%{REQUEST_URI} [NC,NE,R,L]

What I want is:

    #Force all traffic through secure http
    RewriteCond %{REQUEST_URI} !^/jmx/app_status.jsp
    RewriteCond %{HTTP:SSLClientCipher} !^..*$
    RewriteCond %{HTTP_HOST} !^localhost* [NC]
    RewriteCond %{REQUEST_URI} !/fw3?\.jsp
    RewriteCond %{REQUEST_URI} !/d\.jsp
    RewriteCond %{REQUEST_URI} !^/support/Data\.jsp
    RewriteCond %{REQUEST_URI} !^.*(\r|\n|%0A|%0D).* [NC]
    RewriteRule ^(.*) https://%{HTTP_HOST}%{REQUEST_URI} [NC,NE,R,L]

This is being changed in the build – each application carries it's own apache.conf file within the RPM – better to do this in puppet, but that has not been pulled out yet. That deployment of the rpm and config file will ultimately put the change out as releases move forward.

But meanwhile I need this change out to 73 servers and 42 applications now. Within the next hour actually.

I worked through the SED command on a single box, and then tried it on that same box and then another box before running through func – the flip side of sed is that you can very quickly mess up a file completely.

This command takes a backup. Running it twice would be bad. You would need to add checking for the line "RewriteCond %{REQUEST_URI} !^/jmx/app_status.jsp" already present if that was a risk, and not run the sed command if already present.


[root@p1-qaut1 ~]# func p1-vm167* call command run ' for i in /etc/httpd/conf/ConstantContact*; do cp ${i} ${i}.bak; sed "s/#Force all traffic through secure http/#Force all traffic through secure http\n    RewriteCond \%\{REQUEST_URI\} \!\^\/jmx\/app_status.jsp/" < ${i} > ${i}.2; mv ${i}.2 ${i}; done'
('p1-vm167.ad.prodcc.net', [0, '', ''])

Okay, I had to do it – here’s the bulletproof version (running more than once creates no effect).


for i in ConstantContact*.conf; do if [[ `egrep "RewriteCond \%\{REQUEST_URI\} \!\^\/jmx\/app_status.jsp" ${i}` ]]; then echo $i > /dev/null; else cp ${i} ${i}.bak; sed "s/#Force all traffic through secure http/#Force all traffic through secure http\n    RewriteCond \%\{REQUEST_URI\} \!\^\/jmx\/app_status.jsp/" < ${i} > ${i}.2; mv ${i}.2 ${i}; fi; done

and in func…


[root@p1-qaut1 ~]# func p1-vm167* call command run 'for i in /etc/httpd/conf/ConstantContact*.conf; do if [[ `egrep "RewriteCond \%\{REQUEST_URI\} \!\^\/jmx\/app_status.jsp" ${i}` ]]; then echo $i > /dev/null; else cp ${i} ${i}.bak; sed "s/#Force all traffic through secure http/#Force all traffic through secure http\n    RewriteCond \%\{REQUEST_URI\} \!\^\/jmx\/app_status.jsp/" < ${i} > ${i}.2; mv ${i}.2 ${i}; fi; done'
('p1-vm167.ad.prodcc.net', [0, '', ''])
[root@p1-qaut1 ~]#

Good luck. TEST CAREFULLY before running anything like this against servers. You can do great harm.

— doug

A/B Testing – of course…

doug — Wed, 09 May 2012 16:29:25 +0000

I skimmed a Wired.com article, and then stopped.

I read it more closely. The article was A/B Test: Inside the Technology That’s Changing the Rules of Business.

Mentioned in the article was Google Web Optimizer. I found

The ideas are:

Choose everything.

Don't make a decision – take every idea (or at least two of 'em) and test

and see which creates the effect you are trying to achieve -

in the case of websites this is traffic patterns, clicking, purchases…

Data makes the call.

Not "HiPPO", not ”highest-paid person’s opinion.” Rather, let actual measurable results make the decision.

The risk is making only tiny improvements.

The success of this model can reinforce making only incremental changes -

and failing to make big changes when they are needed.

The model can apply to bigger changes, larger chunks of code, but that tendency to narrow focus has to be

taken into account. It is so easy, and so effective to make those incremental changes,

the bigger changes needed over time can get lost…

Data can make the very idea of lessons obsolete.

You can get the right results without knowing what the lesson there was – the testing and measuring have become real time.

This has huge implications for not just web design, but in where real world concrete physical design goes.

Prototype on your 3D printer, including the electrical internals, and see which product succeeds,

then push marketing and development behind that to move it further toward ideas that do well.

Again, you would have to watch for a narrowing of genius and those products that are

not incremental changes in existing structure would have to have a route beyond the

tight narrow framework that could come about.

This also affects hiring – find and hire people who are creative rather than have a specific expertise,

and that also makes sense looking over my experiences both hiring and being hired.

The jobs that work well are not necessarily the specific job I was hired for -

it is the job the challenges evolved into…

— doug

test facebook to wordpress app integration…

doug — Fri, 04 May 2012 18:10:02 +0000

testing “add link to facebook” plugin… Aha! It worked. I used to use “Crosspress” but it is no longer being developed. This one seems the most secure, simplest and most supportable going forward…… Read More →

a dreamhost review

doug — Thu, 03 May 2012 11:00:36 +0000

Four stars…

Mostly good. Glitches are few and they are far between.

They are better by far than Hostgator. Hostgator squats on the https side of your domain, advertising on it. If you want that side of your domain clean, you have to pay them to stop by setting up https. Dreamhost does not, the access to https fails as it should.

The support is adequate. But. To communicate with support you open a ticket. To reply to that ticket you open another ticket within the web interface – there’s no provision to replay within a ticket, to update, or to escalate in any way. They usually respond within 3 hours, but I’ve seen a response take 17 hours.

The ticketing system is sometimes sending out an email acknowledgement and sometimes not. I think you can replay to the email and maybe get it included in the original ticket but I haven’t actually tested that.

I’ve been with them four years. I’m not intending to move somewhere else as yet.

I spent seven hours yesterday down because I deleted a unique IP address I didn’t need and had accidentally set up through the web interface domain management section, and that I thought I had already deleted. The DNS for the removed IP address persisted… And persisted… And persisted…

I checked locally on the server on which I was hosting.


[dsm@muffy dougmunsinger.com] $ nslookup dougmunsinger.com
Server:	 66.33.216.208
Address:	66.33.216.208#53

Non-authoritative answer:
Name:	dougmunsinger.com
Address: 67.205.50.249

[dsm@muffy dougmunsinger.com] $ nslookup 67.205.50.249
Server:	 66.33.216.208
Address:	66.33.216.208#53

Non-authoritative answer:
249.50.205.67.in-addr.arpa	name = dougmunsinger.com.

The specific and individual IP is still in place. Also - this is on the dreamhost server schultz, so propagation of the DNS change is not an issue.

Contrast this with a non-specific ip result:

[dsm@muffy dougmunsinger.com] $ nslookup myappportfolio.com
Server:	 66.33.216.208
Address:	66.33.216.208#53

Non-authoritative answer:
Name:	myappportfolio.com
Address: 69.163.143.189

[dsm@muffy dougmunsinger.com] $ nslookup 69.163.143.189
Server:	 66.33.216.208
Address:	66.33.216.208#53

Non-authoritative answer:
189.143.163.69.in-addr.arpa	name = apache2-yelp.schultz.dreamhost.com.

Eventually the site came back, about 7 PM that evening. I received a reply this morning at 6:17 AM, saying all looked to be good and the caching DNS looks to have cleared.

Like I said, mostly good.

—doug

Func: Fedora Unified Network Controller

doug — Wed, 02 May 2012 16:18:29 +0000

This is a replacement for remote command execution on servers via ssh. It has some serious security advantages – it is never a shell, for example. In managing five environments and over 1400 servers and 92 applications, tools are key.

The install and configuration is at https://fedorahosted.org/func.

Once installed, you can verify that you have connectivity with


[root@p1-util05 ~] # func p1-i53[0-5]* ping
[ ok ... ] p1-i535.ad.prodcc.net
[ ok ... ] p1-i534.ad.prodcc.net
[ ok ... ] p1-i531.ad.prodcc.net
[ ok ... ] p1-i532.ad.prodcc.net
[ ok ... ] p1-i530.ad.prodcc.net
[ ok ... ] p1-i533.ad.prodcc.net
[root@p1-util05 ~] #

and check app status with something like


[root@p1-util05 ~] # func p1-i53[0-5]* call command run 'for i in `ls /etc/init.d | grep jboss-`; do /sbin/service $i status; done'
('p1-i534.ad.prodcc.net', [0, 'landingapp is running\n\n', ''])
('p1-i531.ad.prodcc.net', [0, 'landingapp is running\n\n', ''])
('p1-i532.ad.prodcc.net', [0, 'landingapp is running\n\n', ''])
('p1-i530.ad.prodcc.net', [0, 'landingapp is running\n\n', ''])
('p1-i535.ad.prodcc.net', [0, 'landingapp is running\n\n', ''])
('p1-i533.ad.prodcc.net', [0, 'landingapp is running\n\n', ''])
[root@p1-util05 ~] #

and deploy with puppet using something like


[root@sc-util5 ~]# func sca1-jbwebsite* ping
[ ok ... ] sc-website1001.ad.prodcc.net
[ ok ... ] sc-website1002.ad.prodcc.net
[ ok ... ] sc-website1000.ad.prodcc.net
[ ok ... ] sc-website1003.ad.prodcc.net
[root@sc-util5 ~]# func sc-website* call command run '/usr/sbin/puppetd --server=sca1-puppet1.ad.prodcc.net --onetime --no-daemonize --no-usecacheonfailure --ignorecache --no-noop'
('sc-website1002.ad.prodcc.net', [0, '', ''])
('sc-website1001.ad.prodcc.net', [0, '', ''])
('sc-website1000.ad.prodcc.net', [0, '', ''])
('sc-website1003.ad.prodcc.net', [0, '', ''])
[root@sc-util5 ~]#

“0″ means success… “1″ means there was an error.

Without this kind of blanket ad hoc command line tool, managing repetitive tasks can quickly become very difficult and time consuming…

— doug

FUNC: which apps redirect from port 80 to 443?

doug — Sat, 17 Mar 2012 14:25:00 +0000

I am migrating the load balancer pools from one F5 load balancer to another.
Along with that move is a change to the monitor used to control automated load balancer status –
when the app is stopped we have a sequence in the stop function which drains and then offlines the
application in the load balancer, allowing for a graceful removal of the app from the pool.
On start, the app remains offline, and an online argument changes the pool status to “enabled”
bringing the app into the pool after verifying that it is accessible and active.

The first step in this was to find which apps are redirecting from port 80 to port 443, This will
allow us to migrate the nodes as port 443 in the load balancer where they are actually running on port 443.
In the original F5 they could be installed as port 80 nodes, and because of the redirect in most
cases everything would work. But there is an extra hop. We also have a new monitoring check which uses
http or https to directly verify load balancer pool status. The F5 won’t follow a redirect in the monitor.
Where before it was arbitrary which port the node could be installed in, now it will not be.

The configuration to accomplish the redirect is:


    #Force all traffic through secure http
    RewriteCond %{HTTP:SSLClientCipher} !^..*$
    RewriteCond %{HTTP_HOST} !^localhost* [NC]
    RewriteCond %{REQUEST_URI} !/fwtf2?\.jsp
    RewriteCond %{REQUEST_URI} !/d\.jsp
    RewriteCond %{REQUEST_URI} !^/support/supportData\.jsp
    RewriteCond %{REQUEST_URI} !^.*(\r|\n|%0A|%0D).* [NC]
    RewriteCond %{REQUEST_URI} !^/distui/bootstrap/.*
    RewriteCond %{REQUEST_URI} !^/healthcheck$
    RewriteCond %{REQUEST_URI} !^/lb_status$
    RewriteRule ^(.*) https://%{HTTP_HOST}%{REQUEST_URI} [NC,NE,R,L]

We could approach this by exempting lb_status from the redirect, thus making the check
itself accessible over http. But that’s not what we want right now.

What I need right now is a list of the hosts that have this rewrite rule in place,
and are running on port 443.

The string I used is

 RewriteRule ^(.*) https://

I ran func to get a list of hosts set up as jboss and as minions for the environment using:

func p1-vml1jb* ping

I created the list, sorted it, removed any “[FAILED]” hosts after verifying they should fail.
An application no longer used, or a server not actually in use right now.

To cycle through the list and pull out the information needed I ran:

[root@p1-qau01 ~]# for i in `cat list_sort`
do
  if [[ `func ${i}* call command run 'egrep " RewriteRule \^\(\.\*\) https:\/\/" /etc/httpd/conf/*' | egrep "\[0,"` ]]; then
    echo ""
    echo $i
    echo _________________________________________________
    func ${i}* call command run 'egrep " RewriteRule \^\(\.\*\) https:\/\/" /etc/httpd/conf/*'
  fi
done

p1-vml1jb108.cc.net
_________________________________________________
('p1-vml1jb108.cc.net',
 [0,
  '/etc/httpd/conf/CtCtRepConsole.conf:    RewriteRule ^(.*) https://%{HTTP_HOST}/$1 [NC,R,L]\n',
  ''])

p1-vml1jb109.cc.net
_________________________________________________
('p1-vml1jb109.cc.net',
 [0,
  '/etc/httpd/conf/CtCtRepConsole.conf:    RewriteRule ^(.*) https://%{HTTP_HOST}/$1 [NC,R,L]\n',
  ''])

p1-vml1jb110.cc.net
_________________________________________________
('p1-vml1jb110.cc.net',
 [0,
  '/etc/httpd/conf/CtCtsvc.conf:    RewriteRule ^(.*) https://%{HTTP_HOST}%{REQUEST_URI} [NC,NE,R,L]\n/etc/httpd/conf/CtCtCust.conf:    RewriteRule ^(.*) https://%{HTTP_HOST}/$1 [NC,R,L]\n/etc/httpd/conf/CtCtPart.conf:    RewriteRule ^(.*) https://%{HTTP_HOST}/$1 [NC,R,L]\n',
  ''])

What this does is check if the content of the string exists, then if it does (a “0″ result from the func command) then re-execute it getting not just the end code but the full result and print it to the screen in a readable format.

Once this completed I had a list of all of the apps using this redirect. The names of the config files correspond to the application, making this relatively easy to create a list from these results.

— doug

adding HP jetdirect printer to windows XP

doug — Sat, 10 Mar 2012 13:44:45 +0000

About every three to six months IT moves printers, upgrades printer and otherwise makes printer changes necessary.

Most of our printers are HP with jetdirect cards (port 9100). There is a printer server. In UNIX this means translating windows shared printers, naming conventions and security features, through a rather cryptic graphic interface. Doable once in a long while, not worth it every three months…

I tend to connect directly to the printer’s IP address at 9100 and let jetdirect sort out queues and print management.

This also works on windows XP laptops. What you are doing is configuring a local printer, connected to a TCP/IP port.

To do this , open Control Panel -> Printers and Faxes. Right-click and select “Add Printer”. Click “Next”. Select “Local printer” .

From this click “Next”

Then select “Create a new port” and from the drop down menu, type of port is “Standard TCP/IP Port”.

You will hit the opening screen for the “Add Standard TCP/IP Port Wizard”. Click “Next” and then enter the IP address for the printer. You can use the DNS name, I tend not to, I ususally use an IP address. How do you find the IP address? Most modern HP printers have a menu on the printer that allows you to go to “Information” -> “Print Configuration”. The IP address will be in that configuration.

Click “Next”.. This step seems to take a few minutes. Once it returns you get back a message that the detected device type cannot be determined. Drop down and select “Hewlett Packard Jet Direct” and then click next…

Select “Next”. From here the installation is basically the same as every other printer – Select manufacturer “HP” and then scroll to the closest match driver for your printer. I’ve had good luck using an HP Color Laserject 4500 driver for an HP4525 printer for example.

windows printers and print jobs refuse to delete

doug — Mon, 05 Mar 2012 13:23:39 +0000

Occasionally I’ll run into a situation where I have to adjust the printers on a laptop. Delete one, add a new and changed one, basically bring the printer configuration up to date with what is currently present on the network.

Today I brought in my netbook, and I needed to print a few pages of a puppet cookbook. I kicked off the print job, and then realized that the printer configured no longer existed.

I deleted the print job, and went to the printers and faxes folder to see if I could leave the printer, configure a new IP address, and change the driver to the replacement printer. A very simple operation in UNIX, you can edit a file or go through a graphic interface to do it – transparent, nothing hidden from the administrator or even from the user. Windows refused first to delete the print job and then to adjust or change the printer, or delete any other obsolete printers.

Windows can’t do anything to change printers if the Print Spooler service has a print job ready to print and can’t print or delete that job.

The resolution was to first stop the Print Spooler service – go to Start, then Run and type in services.msc. Scroll down to the Print Spooler service, right-click on it and choose Stop

print spooler service - stop service

Don’t close the service window yet. Go into C:\WINDOWS\system32\spool\PRINTERS, and delete any files in this directory.

Go back to the services pane, restart the Print Spooler service (right-click and choose start).

Now your print job is deleted, you have the print spooler service running, you can delete printer, then add the correct printer.

—doug

mapping the memory within a java process

doug — Fri, 02 Mar 2012 18:46:53 +0000

At the system level, you can look at not just how much memory a process is using, but where it is using the memory, in detail. Use pmap…

First find the process id:

[root@p2-jbtest ~]# ps -ef | grep java

bbod     11987 11929  0 Feb29 ?        00:04:03 /opt/jdk/bin/java
-Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxPermSize=256m
-Dcom.test.sso.cookiedomain=s1.test.com -Dtangosol.coherence.mode=prod
-XX:+DisableExplicitGC -XX:+PrintGCDetails -XX:NewRatio=3
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/etc/heap -Dsun.rmi.dgc.client.gcInterval=1800000
-Dsun.rmi.dgc.server.gcInterval=1800000
-Dtangosol.coherence.log.level=2 -Dtangosol.coherence.management=all
-Dtangosol.coherence.management.remote=true
-Dtangosol.coherence.management.readonly=true -verbose:gc
-Xloggc:/opt/logs/gc.log.ui.201202291332 -XX:+PrintGCDateStamps
-XX:+PrintGCDetails -Dorg.apache.catalina.STRICT_SERVLET_COMPLIANCE=false
-Djava.net.preferIPv4Stack=true
-Djava.endorsed.dirs=/opt/jboss/lib/endorsed -classpath
/opt/jboss/bin/run.jar:/opt/jdk/lib/tools.jar org.jboss.Main
-c ui -P /opt/jboss/server/ui/lib/db.properties -Djboss.server.log.dir=/opt/cc/logs
-b ui.p2-jbtest.bsd.prodcc.net
-Djboss.bind.address=ui.p2-jbtest.bsd.prodcc.net
-Djboss.svr.name=p2-jbtest.bsd.prodcc.net
-Djboss.env=p2-bsd -Djboss.app=ui

The run pmap as follows against that process ID:

[root@p2-jbtest ~]# pmap -x 11987
11987:   /opt/jdk/bin/java -Dprogram.name=run.sh -server -Xms2048m -Xmx2048m -XX:MaxPermSize=256m
-Dcom.roving.sso.cookiedomain=s1.constantcontact.com -Dtangosol.coherence.mode=prod -XX:+DisableExplicitGC
-XX:+PrintGCDetails -XX:NewRatio=3 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/cc/etc/ui/heap
-Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -Dtangosol.coherence.log.level=2
-Dtangosol.coherence.management=all -Dtangosol.coherence.management.remote=true -Dtangosol.coheren
Address           Kbytes     RSS   Dirty Mode   Mapping
0000000040000000      36      36       0 r-x--  java
0000000040108000       8       8       8 rwx--  java
000000004010a000       4       0       0 -----    [ anon ]
000000004010b000    1024      12      12 rwx--    [ anon ]
000000004020b000      12       0       0 -----    [ anon ]
000000004020e000    1016      36      36 rwx--    [ anon ]
00000000403c6000      12       0       0 -----    [ anon ]
00000000403c9000    1016      32      32 rwx--    [ anon ]
0000000040505000      12       0       0 -----    [ anon ]
0000000040508000    1016      44      44 rwx--    [ anon ]
0000000040634000      12       0       0 -----    [ anon ]
0000000040637000    1016     104     104 rwx--    [ anon ]
0000000040735000      12       0       0 -----    [ anon ]
0000000040738000    1016      36      36 rwx--    [ anon ]
0000000040836000      12       0       0 -----    [ anon ]
0000000040839000    1016      32      32 rwx--    [ anon ]
0000000040937000      12       0       0 -----    [ anon ]
000000004093a000    1016      44      44 rwx--    [ anon ]
0000000040a38000      12       0       0 -----    [ anon ]
0000000040a3b000    1016      32      32 rwx--    [ anon ]
0000000040c34000       4       0       0 -----    [ anon ]
0000000040c35000    1024      12      12 rwx--    [ anon ]
0000000040d35000       4       0       0 -----    [ anon ]
0000000040d36000    1024      12      12 rwx--    [ anon ]
0000000040e68000      12       0       0 -----    [ anon ]
0000000040e6b000    1016      32      32 rwx--    [ anon ]
0000000040f76000      12       0       0 -----    [ anon ]
0000000040f79000    1016      32      32 rwx--    [ anon ]
000000004107a000       4       0       0 -----    [ anon ]
000000004107b000    1024      12      12 rwx--    [ anon ]
000000004127a000      12       0       0 -----    [ anon ]
000000004127d000    1016      52      52 rwx--    [ anon ]
000000004137b000      12       0       0 -----    [ anon ]
000000004137e000    1016      36      36 rwx--    [ anon ]
00000000414a7000      12       0       0 -----    [ anon ]
00000000414aa000    1016       8       8 rwx--    [ anon ]
00000000415a8000      12       0       0 -----    [ anon ]
00000000415ab000    1016      44      44 rwx--    [ anon ]
00000000416c5000      12       0       0 -----    [ anon ]
00000000416c8000    1016      56      56 rwx--    [ anon ]
00000000417c6000      12       0       0 -----    [ anon ]
00000000417c9000    1016      36      36 rwx--    [ anon ]
000000004190b000      12       0       0 -----    [ anon ]
000000004190e000    1016      32      32 rwx--    [ anon ]
0000000041a70000      12       0       0 -----    [ anon ]
0000000041a73000    1016      36      36 rwx--    [ anon ]
0000000041b75000       4       0       0 -----    [ anon ]
0000000041b76000    1024      12      12 rwx--    [ anon ]
0000000041c76000       4       0       0 -----    [ anon ]
0000000041c77000    1024      12      12 rwx--    [ anon ]
0000000041d77000       4       0       0 -----    [ anon ]
0000000041d78000    1024      12      12 rwx--    [ anon ]
0000000041e78000       4       0       0 -----    [ anon ]
0000000041e79000    1024      12      12 rwx--    [ anon ]
0000000041f79000       4       0       0 -----    [ anon ]
0000000041f7a000    1024      12      12 rwx--    [ anon ]
000000004207a000       4       0       0 -----    [ anon ]
000000004207b000    1024      12      12 rwx--    [ anon ]
000000004217b000       4       0       0 -----    [ anon ]
000000004217c000    1024      12      12 rwx--    [ anon ]
000000004227c000       4       0       0 -----    [ anon ]
000000004227d000    1024      12      12 rwx--    [ anon ]
000000004237d000       4       0       0 -----    [ anon ]
000000004237e000    1024      12      12 rwx--    [ anon ]
000000004247e000       4       0       0 -----    [ anon ]
000000004247f000    1024      12      12 rwx--    [ anon ]
000000004257f000      12       0       0 -----    [ anon ]
0000000042582000    1016      52      52 rwx--    [ anon ]
0000000042680000      12       0       0 -----    [ anon ]
0000000042683000    1016      48      48 rwx--    [ anon ]
0000000042781000      12       0       0 -----    [ anon ]
0000000042784000    1016       8       8 rwx--    [ anon ]
0000000042882000       4       0       0 -----    [ anon ]
0000000042883000    1024       8       8 rwx--    [ anon ]
0000000042983000      12       0       0 -----    [ anon ]
0000000042986000    1016      92      92 rwx--    [ anon ]
0000000042a84000      12       0       0 -----    [ anon ]
0000000042a87000    1016      32      32 rwx--    [ anon ]
0000000042b85000      12       0       0 -----    [ anon ]
0000000042b88000    1016      36      36 rwx--    [ anon ]
0000000042c86000      12       0       0 -----    [ anon ]
0000000042c89000    1016      36      36 rwx--    [ anon ]
0000000042d87000      12       0       0 -----    [ anon ]
0000000042d8a000    1016      44      44 rwx--    [ anon ]
0000000042e88000      12       0       0 -----    [ anon ]
0000000042e8b000    1016      36      36 rwx--    [ anon ]
0000000042f89000      12       0       0 -----    [ anon ]
0000000042f8c000    1016      68      68 rwx--    [ anon ]
000000004308a000      12       0       0 -----    [ anon ]
000000004308d000    1016      32      32 rwx--    [ anon ]
000000004318b000      12       0       0 -----    [ anon ]
000000004318e000    1016      60      60 rwx--    [ anon ]
000000004328c000      12       0       0 -----    [ anon ]
000000004328f000    1016      36      36 rwx--    [ anon ]
000000004338d000      12       0       0 -----    [ anon ]
0000000043390000    1016      32      32 rwx--    [ anon ]
000000004348e000      12       0       0 -----    [ anon ]
0000000043491000    1016      60      60 rwx--    [ anon ]
000000004358f000      12       0       0 -----    [ anon ]
0000000043592000    1016      36      36 rwx--    [ anon ]
0000000043690000      12       0       0 -----    [ anon ]
0000000043693000    1016      36      36 rwx--    [ anon ]
0000000043791000      12       0       0 -----    [ anon ]
0000000043794000    1016      44      44 rwx--    [ anon ]
0000000043892000      12       0       0 -----    [ anon ]
0000000043895000    1016      36      36 rwx--    [ anon ]
0000000043993000      12       0       0 -----    [ anon ]
0000000043996000    1016      36      36 rwx--    [ anon ]
0000000043a94000      12       0       0 -----    [ anon ]
0000000043a97000    1016      48      48 rwx--    [ anon ]
0000000043b95000      12       0       0 -----    [ anon ]
0000000043b98000    1016      40      40 rwx--    [ anon ]
0000000043c96000      12       0       0 -----    [ anon ]
0000000043c99000    1016      52      52 rwx--    [ anon ]
0000000043d97000      12       0       0 -----    [ anon ]
0000000043d9a000    1016      32      32 rwx--    [ anon ]
0000000043e98000      12       0       0 -----    [ anon ]
0000000043e9b000    1016      36      36 rwx--    [ anon ]
0000000043f99000      12       0       0 -----    [ anon ]
0000000043f9c000    1016      36      36 rwx--    [ anon ]
000000004409a000      12       0       0 -----    [ anon ]
000000004409d000    1016      36      36 rwx--    [ anon ]
000000004419b000      12       0       0 -----    [ anon ]
000000004419e000    1016      96      96 rwx--    [ anon ]
000000004429c000      12       0       0 -----    [ anon ]
000000004429f000    1016      36      36 rwx--    [ anon ]
000000004439d000      12       0       0 -----    [ anon ]
00000000443a0000    1016      44      44 rwx--    [ anon ]
000000004449e000      12       0       0 -----    [ anon ]
00000000444a1000    1016      76      76 rwx--    [ anon ]
000000004459f000      12       0       0 -----    [ anon ]
00000000445a2000    1016      52      52 rwx--    [ anon ]
00000000446a0000      12       0       0 -----    [ anon ]
00000000446a3000    1016      56      56 rwx--    [ anon ]
00000000447a1000      12       0       0 -----    [ anon ]
00000000447a4000    1016      48      48 rwx--    [ anon ]
00000000448a2000      12       0       0 -----    [ anon ]
00000000448a5000    1016      56      56 rwx--    [ anon ]
00000000449a3000      12       0       0 -----    [ anon ]
00000000449a6000    1016      48      48 rwx--    [ anon ]
0000000044aa4000      12       0       0 -----    [ anon ]
0000000044aa7000    1016      36      36 rwx--    [ anon ]
0000000044ba5000      12       0       0 -----    [ anon ]
0000000044ba8000    1016      36      36 rwx--    [ anon ]
0000000044ca6000      12       0       0 -----    [ anon ]
0000000044ca9000    1016      44      44 rwx--    [ anon ]
0000000044da7000      12       0       0 -----    [ anon ]
0000000044daa000    1016      76      76 rwx--    [ anon ]
0000000044ea8000      12       0       0 -----    [ anon ]
0000000044eab000    1016      52      52 rwx--    [ anon ]
0000000044fa9000      12       0       0 -----    [ anon ]
0000000044fac000    1016      56      56 rwx--    [ anon ]
00000000450aa000      12       0       0 -----    [ anon ]
00000000450ad000    1016      40      40 rwx--    [ anon ]
00000000451ab000      12       0       0 -----    [ anon ]
00000000451ae000    1016      44      44 rwx--    [ anon ]
00000000452ac000      12       0       0 -----    [ anon ]
00000000452af000    1016      36      36 rwx--    [ anon ]
00000000453ad000      12       0       0 -----    [ anon ]
00000000453b0000    1016      32      32 rwx--    [ anon ]
00000000454ae000      12       0       0 -----    [ anon ]
00000000454b1000    1016      36      36 rwx--    [ anon ]
00000000455af000      12       0       0 -----    [ anon ]
00000000455b2000    1016      36      36 rwx--    [ anon ]
00000000456b0000      12       0       0 -----    [ anon ]
00000000456b3000    1016     116     116 rwx--    [ anon ]
00000000457b1000      12       0       0 -----    [ anon ]
00000000457b4000    1016     116     116 rwx--    [ anon ]
00000000458b2000      12       0       0 -----    [ anon ]
00000000458b5000    1016      32      32 rwx--    [ anon ]
00000000459b3000      12       0       0 -----    [ anon ]
00000000459b6000    1016      60      60 rwx--    [ anon ]
0000000045ab4000      12       0       0 -----    [ anon ]
0000000045ab7000    1016      48      48 rwx--    [ anon ]
0000000045bb5000      12       0       0 -----    [ anon ]
0000000045bb8000    1016      44      44 rwx--    [ anon ]
0000000045cb6000      12       0       0 -----    [ anon ]
0000000045cb9000    1016      44      44 rwx--    [ anon ]
00000000461df000  673112  660568  660568 rwx--    [ anon ]
0000000770000000  159104  159072  159072 rwx--    [ anon ]
0000000779b60000  103040       0       0 rwx--    [ anon ]
0000000780000000 2097152  989472  989472 rwx--    [ anon ]
0000003471400000     112     104       0 r-x--  ld-2.5.so
000000347161b000       4       4       4 r-x--  ld-2.5.so
000000347161c000       4       4       4 rwx--  ld-2.5.so
0000003471800000    1336     556       0 r-x--  libc-2.5.so
000000347194e000    2048       0       0 -----  libc-2.5.so
0000003471b4e000      16      16       8 r-x--  libc-2.5.so
0000003471b52000       4       4       4 rwx--  libc-2.5.so
0000003471b53000      20      20      20 rwx--    [ anon ]
0000003471c00000       8       8       0 r-x--  libdl-2.5.so
0000003471c02000    2048       0       0 -----  libdl-2.5.so
0000003471e02000       4       4       4 r-x--  libdl-2.5.so
0000003471e03000       4       4       4 rwx--  libdl-2.5.so
0000003472000000      88      64       0 r-x--  libpthread-2.5.so
0000003472016000    2044       0       0 -----  libpthread-2.5.so
0000003472215000       4       4       4 r-x--  libpthread-2.5.so
0000003472216000       4       4       4 rwx--  libpthread-2.5.so
0000003472217000      16       4       4 rwx--    [ anon ]
0000003472400000     520      20       0 r-x--  libm-2.5.so
0000003472482000    2044       0       0 -----  libm-2.5.so
0000003472681000       4       4       4 r-x--  libm-2.5.so
0000003472682000       4       4       4 rwx--  libm-2.5.so
0000003473400000      28      20       0 r-x--  librt-2.5.so
0000003473407000    2048       0       0 -----  librt-2.5.so
0000003473607000       4       4       4 r-x--  librt-2.5.so
0000003473608000       4       4       4 rwx--  librt-2.5.so
0000003473800000      84      24       0 r-x--  libnsl-2.5.so
0000003473815000    2044       0       0 -----  libnsl-2.5.so
0000003473a14000       4       4       4 r-x--  libnsl-2.5.so
0000003473a15000       4       4       4 rwx--  libnsl-2.5.so
0000003473a16000       8       0       0 rwx--    [ anon ]
0000003473c00000      68      52       0 r-x--  libresolv-2.5.so
0000003473c11000    2048       0       0 -----  libresolv-2.5.so
0000003473e11000       4       4       4 r-x--  libresolv-2.5.so
0000003473e12000       4       4       4 rwx--  libresolv-2.5.so
0000003473e13000       8       4       0 rwx--    [ anon ]
00002aaaaaaab000       8       8       0 r-xs-  run.jar
00002aaaaaaad000       4       4       0 r-xs-  getopt.jar
00002aaaaaab8000      52      52       0 r-x--  libverify.so
00002aaaaaac5000    1020       0       0 -----  libverify.so
00002aaaaabc4000      12      12      12 rwx--  libverify.so
00002aaaaabc7000     164     148       0 r-x--  libjava.so
00002aaaaabf0000    1020       0       0 -----  libjava.so
00002aaaaacef000      28      20      20 rwx--  libjava.so
00002aaaaacf6000       4       4       0 r-x--    [ anon ]
00002aaaaacf7000       4       4       4 rwx--    [ anon ]
00002aaaaacf8000      28      28       0 r-x--  libhpi.so
00002aaaaacff000    1028       0       0 -----  libhpi.so
00002aaaaae00000       8       8       8 rwx--  libhpi.so
00002aaaaae02000       4       0       0 rwx--    [ anon ]
00002aaaaae03000      32      32      16 rwxs-  11987
00002aaaaae0f000      40      28       0 r-x--  libnss_files-2.5.so
00002aaaaae19000    2044       0       0 -----  libnss_files-2.5.so
00002aaaab018000       4       4       4 r-x--  libnss_files-2.5.so
00002aaaab019000       4       4       4 rwx--  libnss_files-2.5.so
00002aaaab01a000      56      48       0 r-x--  libzip.so
00002aaaab028000    1032       0       0 -----  libzip.so
00002aaaab12a000      12      12      12 rwx--  libzip.so
00002aaaab12d000   14340   14160   14160 rwx--    [ anon ]
00002aaaabf2e000   34816       0       0 rwx--    [ anon ]
00002aaaae12e000     224     224     224 rwx--    [ anon ]
00002aaaae166000     544       0       0 rwx--    [ anon ]
00002aaaae1ee000     312     312     312 rwx--    [ anon ]
00002aaaae23c000     200       0       0 rwx--    [ anon ]
00002aaaae26e000    7484    7484    7484 rwx--    [ anon ]
00002aaaae9bd000     200       0       0 rwx--    [ anon ]
00002aaaae9ef000     160     160     160 rwx--    [ anon ]
00002aaaaea17000      76      76       0 r-xs-  xalan.jar
00002aaaaea2a000       8       8       0 r-xs-  activation.jar
00002aaaaea2c000      12      12       0 r-xs-  jaxb-api.jar
00002aaaaea2f000       8       8       0 r-xs-  jbossws-native-jaxws-ext.jar
00002aaaaea31000       8       8       0 r-xs-  jbossws-native-jaxrpc.jar
00002aaaaea33000       8       8       0 r-xs-  stax-api.jar
00002aaaaea35000      92      92       0 r-xs-  xercesImpl.jar
00002aaaaea4c000      16      16       0 r-xs-  serializer.jar
00002aaaaea50000       8       8       0 r-xs-  jbossws-native-jaxws.jar
00002aaaaea52000       8       8       0 r-xs-  jbossws-native-saaj.jar
00002aaaaea54000       8       8       0 r-xs-  resolver.jar
00002aaaaea56000    1632    1632       0 r-xs-  rt.jar
00002aaaaebee000   14600    2656    2656 rwx--    [ anon ]
00002aaaafa30000   55108      44       0 r-x--  locale-archive
00002aaab3001000     380     380       0 r-xs-  tools.jar
00002aaab3060000      28      28       0 r-xs-  concurrent.jar
00002aaab3067000      16      16       0 r-xs-  log4j-boot.jar
00002aaab306b000       8       4       0 r-xs-  jboss-logging-spi.jar
00002aaab306d000       8       8       0 r-xs-  jboss-logging-log4j.jar
00002aaab306f000      12      12       0 r-xs-  jboss-logging-jdk.jar
00002aaab3072000      44      44       0 r-xs-  jboss-common-core.jar
00002aaab307d000      40      40       0 r-xs-  jboss-xml-binding.jar
00002aaab3087000       8       8       0 r-xs-  jboss-bootstrap.jar
00002aaab3089000      36      36       0 r-xs-  javassist.jar
00002aaab3092000      20      20       0 r-xs-  jboss-reflect.jar
00002aaab3097000      20      20       0 r-xs-  jboss-mdr.jar
00002aaab309c000      12      12       0 r-xs-  jboss-dependency.jar
00002aaab309f000      60      60       0 r-xs-  jboss-kernel.jar
00002aaab30ae000      12      12       0 r-xs-  jboss-metatype.jar
00002aaab30b1000      16      16       0 r-xs-  jboss-managed.jar
00002aaab30b5000      24      24       0 r-xs-  jboss-vfs.jar
00002aaab30bb000       4       4       0 r-xs-  jboss-classloading-spi.jar
00002aaab30bc000      16      16       0 r-xs-  jboss-classloader.jar
00002aaab30c0000      12      12       0 r-xs-  jboss-classloading.jar
00002aaab30c3000       8       8       0 r-xs-  jboss-classloading-vfs.jar
00002aaab30c5000      88      88       0 r-xs-  jboss-aop.jar
00002aaab30db000      20      20       0 r-xs-  jboss-aop-mc-int.jar
00002aaab30e0000      32      32       0 r-xs-  trove.jar
00002aaab30e8000      76      72       0 r-x--  libnet.so
00002aaab30fb000    1028       0       0 -----  libnet.so
00002aaab31fc000      12      12      12 rwx--  libnet.so
00002aaab31ff000    3072      16      16 rwx--    [ anon ]
00002aaab3511000      24      24       0 r-x--  libmanagement.so
00002aaab3517000    1020       0       0 -----  libmanagement.so
00002aaab3616000       8       8       8 rwx--  libmanagement.so
00002aaab3646000      12      12       0 r-xs-  jce.jar
00002aaab3649000      28      28       0 r-xs-  jsse.jar
00002aaab3650000      12      12       0 r-xs-  jce.jar
00002aaab3653000      16      16       0 r-x--  libnss_dns-2.5.so
00002aaab3657000    2044       0       0 -----  libnss_dns-2.5.so
00002aaab3856000       4       4       4 r-x--  libnss_dns-2.5.so
00002aaab3857000       4       4       4 rwx--  libnss_dns-2.5.so
00002aaab3867000       4       4       4 rwx--    [ anon ]
00002aaab3868000      16      16       0 r-xs-  sunpkcs11.jar
00002aaab3879000      12      12       0 r-xs-  sunjce_provider.jar
00002aaab3957000      32      32       0 r-xs-  resources.jar
00002aaab3979000     328     328       0 r-xs-  coherence.jar
00002aaab39d0000      56      56       0 r-xs-  charsets.jar
00002aaab39de000      56      56       0 r-xs-  charsets.jar
00002aaab3b79000     328     328       0 r-xs-  coherence.jar
00002aaab4000000   65524   54276   54276 rwx--    [ anon ]
00002aaab7ffd000      12       0       0 -----    [ anon ]
00002aaab8000000   65536   65516   65516 rwx--    [ anon ]
00002aaabc1e6000      28      28       0 r-x--  libnio.so
00002aaabc1ed000    1020       0       0 -----  libnio.so
00002aaabc2ec000       8       8       8 rwx--  libnio.so
00002aaabc47a000     584     184       0 r-x--  libawt.so
00002aaabc50c000    1020       0       0 -----  libawt.so
00002aaabc60b000     100      56      56 rwx--  libawt.so
00002aaabc624000     148       8       8 rwx--    [ anon ]
00002aaabc681000       4       4       0 r-x--  librmi.so
00002aaabc682000    1020       0       0 -----  librmi.so
00002aaabc781000       4       4       4 rwx--  librmi.so
00002aaabc782000      20      16       0 r-x--  libmawt.so
00002aaabc787000    1020       0       0 -----  libmawt.so
00002aaabc886000       8       8       8 rwx--  libmawt.so
00002aaac0000000   65520   65520   65520 rwx--    [ anon ]
00002aaac3ffc000      16       0       0 -----    [ anon ]
00002aaac4000000   65524   50896   50896 rwx--    [ anon ]
00002aaac7ffd000      12       0       0 -----    [ anon ]
00002aaac8000000   65508   65476   65476 rwx--    [ anon ]
00002aaacbff9000      28       0       0 -----    [ anon ]
00002aaacc000000   65536   65512   65512 rwx--    [ anon ]
00002aaad0000000   65528   65504   65504 rwx--    [ anon ]
00002aaad3ffe000       8       0       0 -----    [ anon ]
00002aaad4000000   65508   65488   65488 rwx--    [ anon ]
00002aaad7ff9000      28       0       0 -----    [ anon ]
00002aaad8000000   65528   65528   65528 rwx--    [ anon ]
00002aaadbffe000       8       0       0 -----    [ anon ]
00002aaadc000000   65520   65492   65492 rwx--    [ anon ]
00002aaadfffc000      16       0       0 -----    [ anon ]
00002aaae0000000   65528   65500   65500 rwx--    [ anon ]
00002aaae3ffe000       8       0       0 -----    [ anon ]
00002aaae8000000   65520   65484   65484 rwx--    [ anon ]
00002aaaebffc000      16       0       0 -----    [ anon ]
00002aaaec000000   64092   64036   64036 rwx--    [ anon ]
00002aaaefe97000    1444       0       0 -----    [ anon ]
00002aaaf0000000   14944    9344    9344 rwx--    [ anon ]
00002aaaf0e98000   50592       0       0 -----    [ anon ]
00002b7717d75000       8       8       8 rwx--    [ anon ]
00002b7717d83000      28      16       0 r-x--  libjli.so
00002b7717d8a000    1028       0       0 -----  libjli.so
00002b7717e8b000       8       8       8 rwx--  libjli.so
00002b7717e8d000      12      12      12 rwx--    [ anon ]
00002b7717e90000    8440    6392       0 r-x--  libjvm.so
00002b77186ce000    1020       0       0 -----  libjvm.so
00002b77187cd000    1616     564     564 rwx--  libjvm.so
00002b7718961000     228     156     156 rwx--    [ anon ]
00007fffce650000      84      44      44 rwx--    [ stack ]
00007fffce6aa000      16       4       0 r-x--    [ anon ]
ffffffffff600000    8192       0       0 -----    [ anon ]
----------------  ------  ------  ------
total kB         4228376 2684036 2672320
[root@p2-jbtest ~]#

And you have in detail WHERE that memory is being allocated.

—doug

recreating a xen VM that has died

doug — Fri, 10 Feb 2012 12:22:30 +0000

This is the process I use to re-create a VM that has hung or otherwise become unresponsive.

1. Find which xen server the VM (e.g., p2-test) is on:

in the svn checkout for puppet configurations (you have puppetized all of your servers, including xen. right?), go to
the location for the configuration of your xen servers, and search for the VM:

$ cd ~/svn/puppet/trunk/modules/site/base/manifests/p2/xenserver
$ find . -exec grep -H p2-test {} \;
./f2/p2-xen101.pp:  xen::xenguests{ 'p2-test':

2. ssh to p2-xen101.

3. Become root:

$ sudo su -

4 Very important, cd to /etc/xen/auto

# cd /etc/xen/auto

5. Optional: to see what Vms are running:

# xm list

6. Optional: to connect to the Vms console to see what went wrong (ctrl-] to get out of the console):

# xm console p2-test

7. Destroy the VM, if it’s running:

# xm destroy p2-test

8. Boot up the VM:

# xm create p2-test