intuitive engineering

One of the features of subversion + apache2 is the ability to list repositories - natively. 

SVNParentPath /apps/repos
SVNListParentPath on

Unfortunately, once you restrict the httpd.conf to individual repositories and start handling permissions separately you lose that.  Both of these permissions, set at the top level, at the parent path to the repositories, have to be commented out to have the individual permissions on the directories below take effect.

And listing just the repositories wasn’t enough for what I had in mind - I wanted a read-only table showing an individual developer or a team lead for a project who has what permissions for the subversion repositories. All the users, all the repositories.

Just your basic cgi.

 and the script:

#! /bin/perl -w

sub DisplayTable {
    my $ref_repos = shift;  #@repos
    my $ref_tabledata = shift; # %tabledata
    my $ref_users = shift; # @users
    # dereference
    my @repos = @$ref_repos;
    my %tabledata = %$ref_tabledata;
    my @users = @$ref_users;
    # repos across the top, users down, R or RW for permissions
    # $tabledata{$rep}{$user} = "readwrite";
    # start table
    print ("<table>\n<tbody>\n");
    # table header
    my $cols = ($#repos + 1);
    print ("<tr><td>Users</td><td colspan=\"$cols\">Repositories</td></tr>\n");
    print ("<div id=\"repotitles\"><tr>\n<td>&nbsp;</td>");
    foreach my $rep (@repos) {
        print ("<td>$rep</td>");
    }
    print ("\n</tr>\n</div>\n");
    foreach my $user (@users) {
        print ("<tr>\n<td>$user</td>");
        foreach my $repo (@repos) {
            if ($tabledata{$repo}{$user}) {
                if ($tabledata{$repo}{$user} eq "read") {
                    print ("<td>R</td>");
                } elsif ($tabledata{$repo}{$user} eq "readwrite") {
                    print ("<td>RW</td>");
                }
            } else {
                print ("<td>&nbsp; - &nbsp;</td>");
            }
        }
        print ("\n</tr>\n");
    }
    # finish table
    print ("</tbody>\n</table>\n");
}

Tags: CGI.pm, script, subversion, svn.

One of the tedious tasks in repository administration is managing users over repositories. Who has access to what repository and to what degree (read-only, or write). Subversion over apache2 allows a tremendous amount of control, down to individual directories within the repository. (see "Per Directory Access Control" in the subversion book).

So far I haven’t placed that but I have set up a default deny and then separate htpasswd files for read, or write access permission. This does at times cause TWO passwords to need to be used to first read and then again write to a repository. However, these are cached, so we’ll see how much developers find to complain about in that.

The script svnadmin.pl assumes that a naming convention for the relationship bewtween htpasswd access files and subversion repositories is set so that the htpasswd file is named <repository_name>_read and <repository_name>_write. You need to setthe path to htpasswd, the path to the htpasswd files, and a location for the top level directory of subversion - the parent directory for all repositories. Within the script you’ll also need to set the paths for the css file ($css) and for the header image ($headerimg). This image should be roughly 420 px wide x 200 px high. I used my corporate logo.

The script also uses a username-as-corporate-ID assumption (begins with "a" or "d" or "x", contains up to eight characters), and a reasonable password assumption (at least 8 charcaters, nor more than 12, must contain at least one capital letter and one digit). This is for internal use, not to be exposed, so if you are going to do something like this on ethe internet, you would want to revisit that and lock it down further.

svnadmin.pl

main.css

To run the script - save-file-as and then change the first line to "#! /usr/bin/perl" (or the appropriate path for the perl you want to use). This is basically removing the extra "#" mark. Rename the script to "svnadmin.pl", put in your cgi-bin location, and put the main.css file in an appropriate location. Edit the variables as above.

Screenshots:

Really fairly simple. This started with a script called "htpasswd.pl" which I downloaded and reworked, then adapted to use CGI.pm for multi-screen form presentation. That framework alone was work rediscovering. I used a similar framework with perl DBI and mysql to do fluid reporting on large system installations 9 years ago. It recurs.

— doug

Tags: apache, perl, script, subversion, svn.

One of the issues with social sites, networking and the co-mingling of work and life - I can have several different “sites”, but I will never have the time to work with all of them. I will update this blog. I won’t necessarily ever get around to its copy on blogger.

IF I can update here, in one place, and as much as possible have that flow through any other sites, at least to keep them showing current links, I haven’t left them dead on the vine, and I haven’t turned my life over to them either.

This post is testing:

• CrossPress
• LiveJournal Crossposter Remix
• TwitterTools
• LinkedIn - which actually runs a feed slurp from LinkedIn’s end…
• WordBook - Facebook plug in.

Ready, computer?

Tags: testing, tweet, website.

Most Open Source software projects invite persons using their product to experience the joys of compiling the product. Subversion does not.

from INSTALL in the 1.5.6 subversion source code:

This document is written for people who intend to build Subversion from source code. Normally, the only people who do this are Subversion developers and package maintainers.

If neither of these labels fits you, we recommend you find an appropriate binary package of Subversion and install that.

why non-privileged?

I work on Solaris 10, in a large financial environment, with separation of responsibilities and a restricted availability of “root”. Most software packages assume that you will be installing as root and they use common and accessible directory locations like “/usr/local”. Most software will build easily or with slight difficulty on linux, and can be much more difficult on Solaris. You will be missing tools and libraries common to Linux.

I currently use CVS, and that’s got some shortcomings - it doesn’t handle binaries easily or consistently, you can’t move a directory without losing history, and it is no longer under development. I have a new container, a Solaris 10 sparse container, in which I wanted to compile and install apache2 and subversion as a non-privileged user. No root. No access to install extra needed packages on /usr/local. On the other hand, complete control over the application without having to consult with the group responsible for root on the container…

what worked

Here’s what worked. Not the dead ends, but what actually resulted in apache and subversion installed, along with apr, apr-util and serf, in a filesystem /apps. This compilation uses serf and FSFS in subversion - it blocks out Berkeley DB and neon.

Berkeley was installed in /usr/local, but I couldn’t get the configure script to find it - and in reading further FSFS is the default in subversion, and has stabilized and become a better solution than Berkeley DB. If you reboot in an unstable state, FSFS will simply recover, where Berkeley DB can require a DBA to recover the database before the system returns to useful.

neon seemed from some errors I encountered to have filesystem access issues (I got a long dump at one point of memory addresses during the make process, all locations for neon libraries and files). Once I eliminated neon, I also got an svnadmin binary that could create a repository without causing “BUS ERROR: segmentation fault”. Of course, I also dropped back from subversion 1.6.0 code to 1.5.6, and compiled and pointed to apr and apr-util from subversion-deps for 1.5.6, and serf from google. It could be any one of those changes.

This was not as simple as compile and go - this was running “configure” and resolving errors, then “make” and resolve yet more errors, and finally, building mod_dav_svn in subversion successfully, installing and finding a “BUS Error: segmentation fault” on running “./svnadmin create /apps/repos/testme”.

I compiled subversion fairly quickly to use svnserver. That compile and install worked, and didn’t create a core file, didn’t seg fault. But it left some problems.

svn + apache2

Many of my developers work from windows. Creating secure access over ssh and svn (svn+ssh://) using eclipse SDK and subclipse (subversion plugin for eclipse) was the initial idea, from a quick pass over subversion, but once svnserver was up and running and I tried it, this wasn’t as simple. There are several articles on integrating subversion, ssh and windows. None of the solutions looked simple or elegant. And wide open unsecured traffic was not acceptable.

I determined best practice pointed us to subversion access through web_dav_svn -> web_dav -> apache2 (https://). This brought authentication away from centralized auth and allowed fine grained control over the access granted via htpasswd and .htaccess files. It allowed us to restrict unsecured access and redirect http to https, and encrypt using SSL. It kept us from having to work ssh onto windows for each developer, instead we could have a simple eclipse/subclipse plugin access demonstrated and from that point they were able to customize on their own.

What I wanted was:

• untar subversion-1.5.6 and subversion-deps-1.5.6 in /apps/src (they layer over each other)
• first build and install the -deps apr and apr-util into /apps/local
• then build apache2 against /apps/local/apr and /apps/local/apr-util
• install apache2
• build and install serf
• remove serf, apr, and apr-util subdirectories and source code from within subversion-1.5.6
• build 1.5.6 against apxs in apache, without Berkeley DB, without neon, and specifying /apps/local/apr, /apps/local/apr-util, and /apps/local/serf
• install and test

process

• create subdirectory /apps/src, place all tarballs in this directory
• untar subversion & subversion-deps version 1.5.6 (these tar onto each other)
• cd subversion-1.5.6/apr
• ./configure –prefix=/apps/local/apr
• make && make install
• cd ../apr-util
• ./configure –prefix=/apps/local/apr-util –with-apr=/apps/local/apr
• make && make install
• apache2: untar httpd-2.2.11
• ./configure –enable-dav –enable-so –prefix=/apps/apache2_2.2.11 –enable-ssl –with-ssl=/usr/local/ssl –with-apr=/apps/local/apr –with-apr-util=/apps/local/apr-util –enable-modules –enable-mods-shared=”most ssl dav”
• make && make install

  This way apache2 builds against apr and apr-util compatible with subversion 1.5.6
  and then build subversion against it as well..

• ln -s /apps/apache2_2.2.11 /apps/apache2
• compile and install serf
• ./configure –prefix=/apps/local/serf –with-openssl=/usr/local/ssl
• make && make install
• remove serf, pr and apr-util from subversion

compile subversion

./configure –prefix=/apps/svn –with-ssl –with-libs=/usr/local/ssl –without-berkeley-db –with-apxs=/apps/apache2/bin/apxs –with-openssl=/usr/local/ssl –without-neon –with-serf=/apps/local/serf –with-apr=/apps/local/apr –with-apr-util=/apps/local/apr-util

make && make install

test:


[appadmin@sccvsapp01p bin] $ ./svnadmin create /apps/repos/testtoo

[appadmin@sccvsapp01p bin] $

no core dump…

ONWARD to configure and test apache2 and subversion…

httpd.conf:

• change all references to apache2_2.2.11 to apache2 (makes the httpd.conf generic rather than subject to needing a migration after a point release upgrade…)
• change port 80 to a non-priveliged port (8080)
• check for
  • LoadModule dav_module modules/mod_dav.so
  • LoadModule dav_module modules/mod_dav.so
  • LoadModule ssl_module modules/mod_ssl.so
• add in ServerName hostname.domain.com:8443 Some of the apache level sanity validation requires a statement of the local host.
• add in SSL stuff (this IS httpd from source - the default httpd.conf had the ssl-module load statement, but no explicit SSL configuration
  
#
# Note: The following must must be present to support
# starting without SSL on platforms with no /dev/random equivalent
# but a statically compiled-in mod_ssl.
#


<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>


# =================================================
# SSL/TLS settings
# =================================================


Listen 0.0.0.0:8080
Listen 0.0.0.0:8443


SSLEngine on
#SSLOptions +StrictRequire


#<Directory />
# SSLRequireSSL
#</Directory>


SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM


SSLMutex file:/apps/apache2/logs/ssl_mutex


SSLRandomSeed startup file:/dev/urandom 1024
SSLRandomSeed connect file:/dev/urandom 1024


SSLSessionCache shm:/apps/apache2/logs/ssl_cache_shm
SSLSessionCacheTimeout 600


SSLPassPhraseDialog builtin
SSLCertificateFile /apps/apache2/conf/ssl.crt/server.crt
SSLCertificateKeyFile /apps/apache2/conf/ssl.key/server.key


SSLVerifyClient none
SSLProxyEngine off


<IfModule mime.c>
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
</IfModule>

  
  
• create ssl certificate (self-signed). I installed in /apps/apache2/conf/ssl.crt and ssl.key, naming the .crt and ,key files for the server hostname and then symbolically linking them to the generic “server.crt” and “server.key”.
• restart apache2 and test https://host:8443/ - you should get the “It works!” apache test page, thus validating the SSL certificate and setup from a browser level
• parent directory for svn
  
  
<Location /svn>
DAV svn
SVNParentPath /apps/repos
</Location>

  
  

Restart apache2, and go look at the log - should now show the svn component:


[Tue Mar 31 11:01:52 2009] [notice] Digest: generating secret for digest authentication …
[Tue Mar 31 11:01:52 2009] [notice] Digest: done
[Tue Mar 31 11:01:52 2009] [notice] Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8d configured — resuming normal ope
rations
[Tue Mar 31 15:15:44 2009] [notice] caught SIGTERM, shutting down
[Tue Mar 31 15:15:48 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Tue Mar 31 15:15:51 2009] [notice] Digest: generating secret for digest authentication …
[Tue Mar 31 15:15:51 2009] [notice] Digest: done
[Tue Mar 31 15:15:51 2009] [notice] Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8d SVN/1.5.6 configured — resuming
normal operations


validate

I used eclipe with subclipse installed in it - the url for the repository was https://host:8443/svn/testme

On the command line

[dsm@dali ~] $ svn checkout https://hostname:8443/svn/testme
Error validating server certificate for 'https://hostname:8443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: hostname.domain.com
 - Valid: from Wed, 01 Apr 2009 14:44:42 GMT until Thu, 01 Apr 2010 14:44:42 GMT
 - Issuer: issuer, Massachusetts, US
 - Fingerprint: ec:74:42:f4:98:0b:5c:62:14:34:85:14:60:38:73:1b:bc:8d:18:27
(R)eject, accept (t)emporarily or accept (p)ermanently? p
Checked out revision 0.
[dsm@dali ~] $

more…

From here, I need security, and then to import existing cvs repositories into subversion one by one, in liaison with the developers. I also need to restrict access to SSL only, install a third-party certificate, and work further to validate subclipse and eclipse, to where I can roll up a development environs they can just unzip and use.

— doug

Tags: apache2, compile, eclipse, subversion, svn.

I’ve been working with Remote System Explorer (RSE), a terminal and ssh session manager that runs in the Eclipse SDK. This is very cool - the thing is, eclipse runs on Solaris, Linux, Windows, MAC OSX, IBM AIX (of course - IBM created the eclipse framework). It as close to an OS-agnostic framework as anything I’ve ever experienced.

I had primarily intended to use it on windows to add ssh terminal capacity - replacing console2 and cygwin - to connect to UNIX servers. But the interface is so good, I find I’m using it on both UNIX and windows - the konsole terminal is better in some ways at history and at cut-and-paste using mouse buttons directly. So far the context menu from a right-click is the only cut-and-paste that functions within the terminal windows themselves.

On the other hand - in konsole each separate terminal window is a separate authentication - password-response - whereas in eclipse RSE, one authentication can be kept and spawn as many terminal sessions as needed. The organization is better. You can close the terminal sessions down, and leave a connection still intact. Then bring back terminal sessions as needed. That ability alone makes it worth losing the double-click select and middle-button paste from konsole sessions. I’m not yet convinced I can’t find a way to get that working either, that and an unlimited history or at least 10000 lines…

If I can get that back somehow in the terminal sessions it will be not just slightly better, but a huge amount better than any other session management I’ve used.

Install

You need java. On windows that can be an issue. On UNIX, not so much, pretty much there by default.

Download RSE and eclipse SDK.

Untar or unzip the eclipse package. In windows I unzip to c:\eclipse_3.4.2. In UNIX /usr/local/eclipse.

Unzip the RSE package layered over the eclipse install.

In windows I point a shortcut to c:\eclipse_3.4.2. In UNIX I move /usr/local/eclipse to /usr/local/eclipse_3.4.2, and create a symbolic link /usr/local/eclipse -> /usr/local/eclipse_3.4.2.

Open eclipse. Go to window -> open perspective -> other and open Remote System Explorer.

Configurations are stored in the workspace folder. Right-click in the left-hand pane and select “New Connection to create connections. Within eclipse you can add software respositories and update and add software from Help -> Software Updates.

— doug

Tags: eclipse, RSE, Solaris, UNIX, windows.

I used Firefox 2 with about seven plugins that I considered must-haves - things like Adblock Plus and Flashblock and Colorful Tabs and Tabbrowser and Remember Mismatched Domains. I recommended it highly, and if it had occasional crashes, they were few and far between enough that I never paid them a lot of attention.

Firefox 3. Not so much. I’ve been waiting since the product first came out for a resolution to whatever memory leak or looping code causes it to just lock up the entire desktop and churn away until I kill it off.

I’ve been using Opera, which I just find a bit odd, but mostly faster and mostly stable. I’ve downloaded Chromium for linux, Codeweaver’s proof-of-concept of running Chrome under wine on linux. And I downloaded and installed opera, chrome and even Apple’s safari on windows.

This firefox misbehavior seems to happen on BOTH operating systems. Here’s a shot of the current bahavior:


Tasks: 180 total, 2 running, 178 sleeping, 0 stopped, 0 zombie
Cpu(s): 70.1%us, 2.7%sy, 0.0%ni, 27.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2066104k total, 2023092k used, 43012k free, 296832k buffers
Swap: 1646620k total, 39812k used, 1606808k free, 972692k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17146 dsm 20 0 373m 180m 35m R 143 8.9 71:52.02 firefox
4995 root 20 0 387m 53m 4744 S 2 2.7 137:35.74 Xorg
7409 dsm 20 0 32592 15m 11m S 1 0.8 0:09.48 konsole
5343 root 20 0 3304 1044 908 S 0 0.1 2:40.47 hald-addon-stor
7793 dsm 20 0 53516 32m 2236 S 0 1.6 9:09.60 synergys
1 root 20 0 2844 1688 544 S 0 0.1 0:01.84 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:03.72 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:01.32 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:04.90 events/0
10 root 15 -5 0 0 0 S 0 0.0 0:03.88 events/1
11 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
46 root 15 -5 0 0 0 S 0 0.0 0:00.28 kblockd/0
47 root 15 -5 0 0 0 S 0 0.0 0:00.30 kblockd/1
50 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
51 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
160 root 20 0 0 0 0 S 0 0.0 0:07.48 pdflush
161 root 20 0 0 0 0 S 0 0.0 0:02.20 pdflush
162 root 15 -5 0 0 0 S 0 0.0 0:02.10 kswapd0


and then after killing it off and restarting:


root@dali:/home/dsm/Desktop/programmes# killall firefox
root@dali:/home/dsm/Desktop/programmes# top
top - 14:46:04 up 10 days, 23:08, 1 user, load average: 0.96, 1.31, 1.25
Tasks: 180 total, 3 running, 177 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.9%us, 0.6%sy, 0.0%ni, 92.5%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2066104k total, 1955112k used, 110992k free, 296928k buffers
Swap: 1646620k total, 39812k used, 1606808k free, 953204k cached


PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17650 dsm 20 0 248m 126m 28m R 17 6.2 0:25.31 firefox
4995 root 20 0 387m 53m 4520 S 1 2.7 137:42.25 Xorg
7366 dsm 20 0 29972 11m 8668 S 0 0.6 1:29.68 kwin
7370 dsm 20 0 36764 18m 13m S 0 0.9 11:19.00 kicker
7793 dsm 20 0 53516 32m 2236 S 0 1.6 9:11.19 synergys
1 root 20 0 2844 1688 544 S 0 0.1 0:01.84 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:03.72 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:01.32 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root 15 -5 0 0 0 S 0 0.0 0:04.92 events/0
10 root 15 -5 0 0 0 S 0 0.0 0:03.88 events/1
11 root 15 -5 0 0 0 S 0 0.0 0:00.00 khelper
46 root 15 -5 0 0 0 S 0 0.0 0:00.28 kblockd/0
47 root 15 -5 0 0 0 S 0 0.0 0:00.30 kblockd/1
50 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
51 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpi_notify
126 root 15 -5 0 0 0 S 0 0.0 0:00.00 kseriod
160 root 20 0 0 0 0 S 0 0.0 0:07.48 pdflush
161 root 20 0 0 0 0 S 0 0.0 0:02.20 pdflush
162 root 15 -5 0 0 0 S 0 0.0 0:02.10 kswapd0


Yes - that first figure - 143% of the cpu. Dropping back to 17% of the CPU with the same tabs and sites loaded.

Whatever. I don’t WANT to pay attention to the browser - the whole point of firefox was to NOT have to pay attention, and they have completely ruined that in version 3.

There is something seriously wrong with firefox, that causes it over time to start misbehaving and to require a reload from scratch. And this seems to occur on both windows and Linux. It is no longer useful. It also seems to being misbehavior once it has downloaded just about any kind of update whether for the browser or a plugin.

I’ve been waiting for firefox three to fix this. Recently firefox 3.07 came out. I just got the window informing me that it has upgraded me to the latest!!!. Just this morning. And yet… And yet - it is still BROKEN.

When you go onto the internet and search for Firefox 3 performance or cpu or memory issues - the first ten or fifteen google results laud firefox three for its blazing performance. True - for awhile. Eventually it will force you to pay attention to it by hanging and chewing up your cpu to where your computer is no longer working.

I wish they’d fix it. But whether they do or not,

I can’t wait for Chrome to come out for linux…

–doug

Tags: browsers, firefox 3, internet.

“java.lang.OutOfMemoryError: PermGen space”

First the application was acting really slow and sluggish, then stopped responding - in looking at the log we saw “java.lang.OutOfMemoryError: PermGen space” as soon as the application was accessed.

The resolution was to set “-XX:MaxPermSize=128m” in the startup script.

to the string sizing the jBoss server is now: “-server -Xms256m -Xmx768m -XX:MaxPermSize=128m”, and that error is gone.

In looking for more data I found:

One more interesting flavor of the same error message, less common but hence even more troublesome is: “java.lang.OutOfMemoryError: PermGen space”. Most of the memory profiler tools are unable to detect this problem, so it is even more troublesome and therefor - interesting.

To understand this error message and fix it, we have to remember that, for optimized, more efficient garbage-collecting Java Heap is managed in generations - memory segments holding objects of different ages.

Garbage collection algorithms in each generation are different. Objects are allocated in a generation for younger objects - the Young Generation, and because of infant mortality most objects die there. When the young generation fills up it causes a Minor Collection. Assuming high infant mortality, minor collections are garbage- collected frequently.

Some surviving objects are moved to a Tenured Generation. When the Tenured Generation needs to be collected there is a Major Collection that is often much slower because it involves all live objects. Each generation contains variables of different length of life and different GC policies are applied to them.

There is a third generation too - Permanent Generation. The permanent generation is special because it holds meta-data describing user classes (classes that are ot part of the Java language). Examples of such meta-data are objects describing classes and methods and they are stored in the Permanent Generation.

Applications with large code-base can quickly fill up this segment of the heap which will cause java.lang.OutOfMemoryError: PermGen no matter how high your -Xmx and how much memory you have on the machine.

Sun JVMs allow you to resize the different generations of the heap, including the permanent generation. On a Sun JVM (1.3.1 and above) you can configure the initial permanent generation size and the maximum permanent generation size.

To set a new initial size on Sun JVM use the -XX:PermSize=64m option when starting the virtual machine. To set the maximum permanent generation size use -XX:MaxPermSize=128m option. If you set the initial size and maximum size to equal values you may be able to avoid some full garbage collections that may occur if/when the permanent generation needs to be resized. The default values differ from among different versions but for Sun JVMs upper limit is typically 64MB.

— dsm

Tags: java, jboss.

This was one of the search terms that found an article here… I hadn’t addressed this directly, but I use Nagios to monitor my company’s server environment, and specifically implemented that monitoring for IBM Websphere MQ.

For MQ, I run nagios monitoring against queue depth and processes. I installed three plugins to run against WebSphere. Of these one was developed for my company’s needs (qdepth), one was changed slightly (channels) and the last debugged, found not to actually measure accurately, and not resolved (message age).

Here’s the nagios console for the websphere MQ server. “message age” in the second qdepth check service title is deceptive - actually checking qdepth…

This is the commands section from the nrpe.cfg file on the WebSphere MQ server.


command[check_mq_channel]=/usr/local/nagios/libexec/check_mq_channel.sh $ARG1$ $ARG2$
command[check_mq_msgage]=/usr/local/nagios/libexec/check_mq_msgage.sh $ARG1$ $ARG2$ $ARG3$ $ARG4$
command[wmq_check_qdepth]=/usr/local/nagios/libexec/wmq_check_qdepth.pl $ARG1$ $ARG2$ $ARG3$


Of these we only really using qdepth monitoring. The channels come up triggered, so an inactive state is fine, and the plugin as written only tests for “running”. The message age plugin, as I mentioned, doesn’t actually work.

When I first looked at setting this messaging up and then monitoring it, I searched for “nagios monitoring MQ webshere” and found several pre-written plugins. I took each plugin and tested it for usability and for accurate results and for meeting what we needed for monitoring.

The message age plugin, in testing, actually returned a hard-coded result rather than actually testing and returning a valid answer. I started to fix it, set it aside and haven’t resolved it. I don’t recall the source for the plugin. Check each piece of code you download from the internet - it may have gone through extensive development and testing, or it could just as easily have been hacked together in an hour. Your mileage may seriously vary and I would highly recommend you verify any of this before you bet your job on it.

Here’s the qdepth plugin - I think I wrote or re-wrote this pretty much from scratch, but the original concept for parsing runmcsc came from one of the plugins I downloaded, written by Kyle O’Donnell - the channel plugin has his original author credit intact. This plugin has alerted once to an increasing qdepth, which turned out to be an issue with an SSL certificate.

## wmq_check_qdepth.pl
#
# nrpe (nagios) script to check websphere qdepth

# uses runmqsc binary
#
# display queue ('APP.REQUEST')
# 8 : display queue ('APP.REQUEST')
# AMQ8409: Display Queue details.
# QUEUE(APP.REQUEST) TYPE(QLOCAL)
# ACCTQ(QMGR) ALTDATE(2008-01-22)
# ALTTIME(14.18.23) BOQNAME( )
# BOTHRESH(0) CLUSNL( )
# CLUSTER( ) CLWLPRTY(0)
# CLWLRANK(0) CLWLUSEQ(QMGR)
# CRDATE(2008-01-22) CRTIME(14.18.23)
# CURDEPTH(0) DEFBIND(OPEN)
# DEFPRTY(0) DEFPSIST(NO)
# DEFSOPT(SHARED) DEFTYPE(PREDEFINED)
# DESCR( ) DISTL(NO)
# GET(ENABLED) HARDENBO
# INITQ( ) IPPROCS(0)
# MAXDEPTH(5000) MAXMSGL(4194304)
# MONQ(QMGR) MSGDLVSQ(PRIORITY)
# NOTRIGGER NPMCLASS(NORMAL)
# OPPROCS(0) PROCESS( )
# PUT(ENABLED) QDEPTHHI(80)
# QDEPTHLO(20) QDPHIEV(DISABLED)
# QDPLOEV(DISABLED) QDPMAXEV(ENABLED)
# QSVCIEV(NONE) QSVCINT(999999999)
# RETINTVL(999999999) SCOPE(QMGR)
# SHARE STATQ(QMGR)
# TRIGDATA( ) TRIGDPTH(1)
# TRIGMPRI(0) TRIGTYPE(FIRST)
# USAGE(NORMAL)

### Variables ###

# test values set if this flag is true (1)
### THIS MUST BE SET TO 0 IN PRODUCTION!!! ###
my $test = 0;

# debug flag (adds messages)
my $debug = 0;
my $LOG = "/tmp/wmq_check_qdepth.pl.log";

# runmqsc binary
my $MQSC = "/opt/mqm/bin/runmqsc";

### ARGS ###

# first argument is warn level
my $WARN = shift;
# second arg is crtitical level
my $CRIT = shift;

# third arg is queue name
my $QUEUE = shift;

# set for dev purposes
if ($test) {
$WARN = 5;
$CRIT = 10;
$QUEUE = "1A33.EVG.REQUEST";
}

# validate
# WARN and CRIT must be greater than 0 and CRIT must be greater than WARN
unless (($WARN > 0) && ($CRIT > 0)) {
print ("Command Failed: WARN and CRIT levels must be greater than 0\n");
exit 3;
}
unless ($CRIT > $WARN) {
print ("Command Failed: CRIT must be greater than WARN\n");
exit 4;
}

### Subs ###

### MAIN ###

# run query
my $result = `echo "display queue ('${QUEUE}')" | $MQSC | grep CURDEPTH`;
print ("result: $result\n") if $debug;
# parse result
my @lines = split ("\n", $result); # divide into an array by end of line...
# each element of the array will contain a single line
# set variables
my ($PARAM, $VALUE);

for my $line (@lines) {
# each line is one or two elements like "QDPLOEV(DISABLED) QDPMAXEV(ENABLED)"
# divide those...
my ($first, $discard) = split (' ', $line);
print ("\$first: $first \$discard $discard\n") if $debug;
($PARAM, $VALUE) = split ('\(', $first);
$VALUE =~ s/\)//;
print ("\$PARAM: $PARAM \$VALUE: $VALUE\n") if $debug;
}

# testing value
$VALUE = 13 if $test;
# check for $WARN and $CRIT levels, exit 0 as OK, 1 as warn or 2 as critical
if ($VALUE == 0) {
print ("OK: found qdepth for $QUEUE at 0\n");
exit 0;
} elsif ($VALUE < $WARN) {
print ("OK: found qdepth for $QUEUE at $VALUE\n");
exit 0;
} elsif (($VALUE >= $WARN) && ($VALUE < $CRIT)) {
print ("WARN: qdepth of $QUEUE is at $VALUE: exceeds WARN thresh of $WARN\n");
exit 1;
} elsif ($VALUE >= $CRIT) {
print (”CRITICAL: qdepth for $QUEUE at $VALUE: exceeds CRITICAL thresh of $CRIT\n”);
exit 2;
}


This is the channel status plugin - I may have re-written the original data gathering runmssc string, but the majority of the plugin remained intact…

Here’s the server.cfg file for the Websphere MQ machine on the nagios server:


define service {
use generic-service
host_name mq1
service_description Host Alive
check_period 24x7
contact_groups unix-administrators
notification_period 24x7
check_command check-host-alive
}

define service {
use generic-service
host_name mq1
service_description Sonic Bridge java process
check_period 24x7
contact_groups esb-administrators
notification_period 24x7
check_command check_unix_proc!mqm!1!java
}

define service {
use generic-service
host_name mq1
service_description SSB queue depth EVGPQM01.DEAD.QUEUE message age
check_period 24x7
contact_groups systems-services,help_desk
notification_period 24x7
check_command wmq_check_qdepth!1!3!QMGR01!QMGR01.DEAD.QUEUE
}

define service {
use generic-service
host_name mq1
service_description server queue depth APPLICATION.RESPONSE
check_period 24x7
contact_groups systems-services,help_desk
notification_period 24x7
check_command wmq_check_qdepth!5!10!APPLICATION.RESPONSE
}

define service {
use generic-service
host_name mq1
service_description server queue depth OPPOSITE-QMGR
check_period 24x7
contact_groups systems-services,help_desk
notification_period 24x7
check_command wmq_check_qdepth!5!10!OPPOSITE-QMGR
}

define service {
use generic-service
host_name mq1
service_description WMQ command server
check_period 24x7
contact_groups systems-services,help_desk
notification_period 24x7
check_command check_unix_proc!mqm!1!amqpcsea
}

define service {
use generic-service
host_name mq1
service_description WMQ Critical process manager
check_period 24x7
contact_groups systems-services,help_desk
notification_period 24x7
check_command check_unix_proc!mqm!1!amqzmuc0
}

The strategy is to monitor qdepth and processes specific to IBM WebSphere MQ on the Websphere MQ server, along with the normal UNIX processes and disk space.

— dsm

Tags: monitoring, nagios, websphere.