apache2 | intuitive engineering

Archive for the ‘apache2’ Category

* another tool for SVN – list_repositories.pl

Posted on May 11th, 2009 by doug. Filed under Solaris, apache2, eclipse, subversion, tools, websphere.

One of the features of subversion + apache2 is the ability to list repositories – natively.

SVNParentPath /apps/repos
SVNListParentPath on

Unfortunately, once you restrict the httpd.conf to individual repositories and start handling permissions separately you lose that. Both of these permissions, set at the top level, at the parent path to the repositories, have to be commented out to have the individual permissions on the directories below take effect.

And listing just the repositories wasn’t enough for what I had in mind – I wanted a read-only table showing an individual developer or a team lead for a project who has what permissions for the subversion repositories. All the users, all the repositories.

Just your basic cgi.

and the script:

#! /bin/perl -w



# script to parse and display users and repositories and rights (R, RW)
use strict;

use CGI qw(:standard);

use CGI::Carp qw(fatalsToBrowser);

##----------------------------

## Variables

##----------------------------

#debug

my $debug = 0; #set to 0 to turn off, 1 (true) to turn on 
# title for page

my $PageTitle = "List of SVN Repositories";
## repository directory

my $SVN = "/apps/repos";

## location for htpasswd files

my $HTDIR = "/apps/apache2/conf/htpasswd";

## Set untainted path

$ENV{PATH} = '/apps/apache2/bin:/bin:/usr/bin:/usr/local/bin';

$ENV{IFS} = "" if $ENV{IFS} ne "";
# css 

my $css = "http://<your server name>/css/main.css";

my $headerimg = "http://<your server name>/css/roger_rabbit_120.jpg";

##------------------------------

## MAIN

##------------------------------
&standard_header;

my ($ref_repos, $ref_tabledata, $ref_users) = &CreateTableSpace;

&DisplayTable ($ref_repos, $ref_tabledata, $ref_users);

&standard_footer;

exit;
##------------------------------

## subs

##------------------------------
sub standard_header {

    print header();

    print start_html(-Title => "$PageTitle", -BGCOLOR=>"White", 

        -style => {

            -src => "${css}"

        }

    );

    print ("<div id=\"header\">\n");

    print p("<img src=\"${headerimg}\" title=\"Wells logo\" alt=\"wells logo\"/>\n");

    print ("</div>\n");

    print ("<div id=\"headertitle\">\n");

    print h3("Repositories<br/>\nusers | read (R) | read & write (RW)\n"); # start_multipart_form() if file upload

}

sub standard_footer { 

    print end_html(); 

}

sub CreateTableSpace {

    my $ref_repos = &GetBlankRepos;

    my @repos = @$ref_repos; # dereference

    my %tabledata = (); # hash to hold table data

    my @users = (); # list of users

    my %seen = (); 

    foreach my $rep (@repos) {

        open(FILE, "$HTDIR/${rep}_read") || croak "Failed to open $HTDIR/${rep}_read for reading...";

        my @filelines = <FILE>;

        close FILE;

        foreach my $line (@filelines) {

            # lines are user:passwd

            my ($user, $pass) = split (":", $line);

            unless ($seen{$user}) {

                $seen{$user} = 1; # save as seen

                push (@users, $user); # save the user to a list

            }

            $tabledata{$rep}{$user} = "read";

        }

        open (FILE, "$HTDIR/${rep}_write") || croak "Failed to open $HTDIR/${rep}_write for reading";

        my @file_lines = <FILE>;

        close FILE;

        foreach my $line (@file_lines) {

            # lines are user:passwd

            my ($user, $pass) = split (":", $line);

            unless ($seen{$user}) {

                $seen{$user} = 1; # save as seen

                push (@users, $user); # save the user to a list

            }

            $tabledata{$rep}{$user} = "readwrite";

        }

    }

    return (\@repos, \%tabledata, \@users);

}

sub GetBlankRepos {

    my @repos = ();

    # list $SVN

    opendir (DIR, $SVN) || croak "Failed to open directory $SVN for reading...";

    while (defined(my $file = readdir(DIR))) {

        # skip ".", ".." and .<hidden> files...

        if ($file =~ /^\./) {

            next;

        } else {

            push(@repos, $file);

        }

    }

    return (\@repos);

}

sub DisplayTable { my $ref_repos = shift; #@repos my $ref_tabledata = shift; # %tabledata my $ref_users = shift; # @users # dereference my @repos = @$ref_repos; my %tabledata = %$ref_tabledata; my @users = @$ref_users; # repos across the top, users down, R or RW for permissions # $tabledata{$rep}{$user} = "readwrite"; # start table print ("<table>\n<tbody>\n"); # table header my $cols = ($#repos + 1); print ("<tr><td>Users</td><td colspan=\"$cols\">Repositories</td></tr>\n"); print ("<div id=\"repotitles\"><tr>\n<td> </td>"); foreach my $rep (@repos) { print ("<td>$rep</td>"); } print ("\n</tr>\n</div>\n"); foreach my $user (@users) { print ("<tr>\n<td>$user</td>"); foreach my $repo (@repos) { if ($tabledata{$repo}{$user}) { if ($tabledata{$repo}{$user} eq "read") { print ("<td>R</td>"); } elsif ($tabledata{$repo}{$user} eq "readwrite") { print ("<td>RW</td>"); } } else { print ("<td>  -  </td>"); } } print ("\n</tr>\n"); } # finish table print ("</tbody>\n</table>\n"); }

share: These icons link to social bookmarking sites where readers can share and discover new web pages.

Tags: CGI.pm, script, subversion, svn.

No Comments

* svnadmin.pl – perl cgi script to manage svn over apache

Posted on May 8th, 2009 by doug. Filed under apache2, perl, subversion.

One of the tedious tasks in repository administration is managing users over repositories. Who has access to what repository and to what degree (read-only, or write). Subversion over apache2 allows a tremendous amount of control, down to individual directories within the repository. (see "Per Directory Access Control" in the subversion book).

So far I haven’t placed that but I have set up a default deny and then separate htpasswd files for read, or write access permission. This does at times cause TWO passwords to need to be used to first read and then again write to a repository. However, these are cached, so we’ll see how much developers find to complain about in that.

The script svnadmin.pl assumes that a naming convention for the relationship bewtween htpasswd access files and subversion repositories is set so that the htpasswd file is named <repository_name>_read and <repository_name>_write. You need to setthe path to htpasswd, the path to the htpasswd files, and a location for the top level directory of subversion – the parent directory for all repositories. Within the script you’ll also need to set the paths for the css file ($css) and for the header image ($headerimg). This image should be roughly 420 px wide x 200 px high. I used my corporate logo.

The script also uses a username-as-corporate-ID assumption (begins with "a" or "d" or "x", contains up to eight characters), and a reasonable password assumption (at least 8 charcaters, nor more than 12, must contain at least one capital letter and one digit). This is for internal use, not to be exposed, so if you are going to do something like this on ethe internet, you would want to revisit that and lock it down further.

svnadmin.pl

main.css

To run the script – save-file-as and then change the first line to "#! /usr/bin/perl" (or the appropriate path for the perl you want to use). This is basically removing the extra "#" mark. Rename the script to "svnadmin.pl", put in your cgi-bin location, and put the main.css file in an appropriate location. Edit the variables as above.

Screenshots:

Really fairly simple. This started with a script called "htpasswd.pl" which I downloaded and reworked, then adapted to use CGI.pm for multi-screen form presentation. That framework alone was work rediscovering. I used a similar framework with perl DBI and mysql to do fluid reporting on large system installations 9 years ago. It recurs.

— doug

share: These icons link to social bookmarking sites where readers can share and discover new web pages.

Tags: apache, perl, script, subversion, svn.

No Comments

* subversion compile and install as non-privileged user…

Posted on April 1st, 2009 by doug. Filed under apache2, compile, eclipse, subversion.

Most Open Source software projects invite persons using their product to experience the joys of compiling the product. Subversion does not.

from INSTALL in the 1.5.6 subversion source code:

This document is written for people who intend to build Subversion from source code. Normally, the only people who do this are Subversion developers and package maintainers.

If neither of these labels fits you, we recommend you find an appropriate binary package of Subversion and install that.

why non-privileged?

I work on Solaris 10, in a large financial environment, with separation of responsibilities and a restricted availability of “root”. Most software packages assume that you will be installing as root and they use common and accessible directory locations like “/usr/local”. Most software will build easily or with slight difficulty on linux, and can be much more difficult on Solaris. You will be missing tools and libraries common to Linux.

I currently use CVS, and that’s got some shortcomings – it doesn’t handle binaries easily or consistently, you can’t move a directory without losing history, and it is no longer under development. I have a new container, a Solaris 10 sparse container, in which I wanted to compile and install apache2 and subversion as a non-privileged user. No root. No access to install extra needed packages on /usr/local. On the other hand, complete control over the application without having to consult with the group responsible for root on the container…

what worked

Here’s what worked. Not the dead ends, but what actually resulted in apache and subversion installed, along with apr, apr-util and serf, in a filesystem /apps. This compilation uses serf and FSFS in subversion – it blocks out Berkeley DB and neon.

Berkeley was installed in /usr/local, but I couldn’t get the configure script to find it – and in reading further FSFS is the default in subversion, and has stabilized and become a better solution than Berkeley DB. If you reboot in an unstable state, FSFS will simply recover, where Berkeley DB can require a DBA to recover the database before the system returns to useful.

neon seemed from some errors I encountered to have filesystem access issues (I got a long dump at one point of memory addresses during the make process, all locations for neon libraries and files). Once I eliminated neon, I also got an svnadmin binary that could create a repository without causing “BUS ERROR: segmentation fault”. Of course, I also dropped back from subversion 1.6.0 code to 1.5.6, and compiled and pointed to apr and apr-util from subversion-deps for 1.5.6, and serf from google. It could be any one of those changes.

This was not as simple as compile and go – this was running “configure” and resolving errors, then “make” and resolve yet more errors, and finally, building mod_dav_svn in subversion successfully, installing and finding a “BUS Error: segmentation fault” on running “./svnadmin create /apps/repos/testme”.

I compiled subversion fairly quickly to use svnserver. That compile and install worked, and didn’t create a core file, didn’t seg fault. But it left some problems.

svn + apache2

Many of my developers work from windows. Creating secure access over ssh and svn (svn+ssh://) using eclipse SDK and subclipse (subversion plugin for eclipse) was the initial idea, from a quick pass over subversion, but once svnserver was up and running and I tried it, this wasn’t as simple. There are several articles on integrating subversion, ssh and windows. None of the solutions looked simple or elegant. And wide open unsecured traffic was not acceptable.

I determined best practice pointed us to subversion access through web_dav_svn -> web_dav -> apache2 (https://). This brought authentication away from centralized auth and allowed fine grained control over the access granted via htpasswd and .htaccess files. It allowed us to restrict unsecured access and redirect http to https, and encrypt using SSL. It kept us from having to work ssh onto windows for each developer, instead we could have a simple eclipse/subclipse plugin access demonstrated and from that point they were able to customize on their own.

What I wanted was:

untar subversion-1.5.6 and subversion-deps-1.5.6 in /apps/src (they layer over each other)
first build and install the -deps apr and apr-util into /apps/local
then build apache2 against /apps/local/apr and /apps/local/apr-util
install apache2
build and install serf
remove serf, apr, and apr-util subdirectories and source code from within subversion-1.5.6
build 1.5.6 against apxs in apache, without Berkeley DB, without neon, and specifying /apps/local/apr, /apps/local/apr-util, and /apps/local/serf
install and test

process

create subdirectory /apps/src, place all tarballs in this directory
untar subversion & subversion-deps version 1.5.6 (these tar onto each other)
cd subversion-1.5.6/apr
./configure –prefix=/apps/local/apr
make && make install
cd ../apr-util
./configure –prefix=/apps/local/apr-util –with-apr=/apps/local/apr
make && make install
apache2: untar httpd-2.2.11
./configure –enable-dav –enable-so –prefix=/apps/apache2_2.2.11 –enable-ssl –with-ssl=/usr/local/ssl –with-apr=/apps/local/apr –with-apr-util=/apps/local/apr-util –enable-modules –enable-mods-shared=”most ssl dav”
make && make install
This way apache2 builds against apr and apr-util compatible with subversion 1.5.6
and then build subversion against it as well..
ln -s /apps/apache2_2.2.11 /apps/apache2
compile and install serf
./configure –prefix=/apps/local/serf –with-openssl=/usr/local/ssl
make && make install
remove serf, pr and apr-util from subversion

compile subversion

./configure –prefix=/apps/svn –with-ssl –with-libs=/usr/local/ssl –without-berkeley-db –with-apxs=/apps/apache2/bin/apxs –with-openssl=/usr/local/ssl –without-neon –with-serf=/apps/local/serf –with-apr=/apps/local/apr –with-apr-util=/apps/local/apr-util

make && make install

test:

[appadmin@sccvsapp01p bin] $ ./svnadmin create /apps/repos/testtoo [appadmin@sccvsapp01p bin] $

no core dump…

ONWARD to configure and test apache2 and subversion…

httpd.conf:

change all references to apache2_2.2.11 to apache2 (makes the httpd.conf generic rather than subject to needing a migration after a point release upgrade…)
change port 80 to a non-priveliged port (8080)
check for
- LoadModule dav_module modules/mod_dav.so
- LoadModule dav_module modules/mod_dav.so
- LoadModule ssl_module modules/mod_ssl.so
add in ServerName hostname.domain.com:8443 Some of the apache level sanity validation requires a statement of the local host.
add in SSL stuff (this IS httpd from source – the default httpd.conf had the ssl-module load statement, but no explicit SSL configuration
# # Note: The following must must be present to support # starting without SSL on platforms with no /dev/random equivalent # but a statically compiled-in mod_ssl. #
<IfModule ssl_module> SSLRandomSeed startup builtin SSLRandomSeed connect builtin </IfModule>
# ================================================= # SSL/TLS settings # =================================================
Listen 0.0.0.0:8080 Listen 0.0.0.0:8443
SSLEngine on #SSLOptions +StrictRequire
#<Directory /> # SSLRequireSSL #</Directory>
SSLProtocol -all +TLSv1 +SSLv3 SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
SSLMutex file:/apps/apache2/logs/ssl_mutex
SSLRandomSeed startup file:/dev/urandom 1024 SSLRandomSeed connect file:/dev/urandom 1024
SSLSessionCache shm:/apps/apache2/logs/ssl_cache_shm SSLSessionCacheTimeout 600
SSLPassPhraseDialog builtin SSLCertificateFile /apps/apache2/conf/ssl.crt/server.crt SSLCertificateKeyFile /apps/apache2/conf/ssl.key/server.key
SSLVerifyClient none SSLProxyEngine off
<IfModule mime.c> AddType application/x-x509-ca-cert .crt AddType application/x-pkcs7-crl .crl </IfModule>
create ssl certificate (self-signed). I installed in /apps/apache2/conf/ssl.crt and ssl.key, naming the .crt and ,key files for the server hostname and then symbolically linking them to the generic “server.crt” and “server.key”.
restart apache2 and test https://host:8443/ – you should get the “It works!” apache test page, thus validating the SSL certificate and setup from a browser level
parent directory for svn

<Location /svn> DAV svn SVNParentPath /apps/repos </Location>

Restart apache2, and go look at the log – should now show the svn component:

[Tue Mar 31 11:01:52 2009] [notice] Digest: generating secret for digest authentication ... [Tue Mar 31 11:01:52 2009] [notice] Digest: done [Tue Mar 31 11:01:52 2009] [notice] Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8d configured -- resuming normal ope rations [Tue Mar 31 15:15:44 2009] [notice] caught SIGTERM, shutting down [Tue Mar 31 15:15:48 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache] [Tue Mar 31 15:15:51 2009] [notice] Digest: generating secret for digest authentication ... [Tue Mar 31 15:15:51 2009] [notice] Digest: done [Tue Mar 31 15:15:51 2009] [notice] Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8d SVN/1.5.6 configured -- resuming normal operations

validate

I used eclipe with subclipse installed in it – the url for the repository was https://host:8443/svn/testme

On the command line

[dsm@dali ~] $ svn checkout https://hostname:8443/svn/testme
Error validating server certificate for 'https://hostname:8443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: hostname.domain.com
 - Valid: from Wed, 01 Apr 2009 14:44:42 GMT until Thu, 01 Apr 2010 14:44:42 GMT
 - Issuer: issuer, Massachusetts, US
 - Fingerprint: ec:74:42:f4:98:0b:5c:62:14:34:85:14:60:38:73:1b:bc:8d:18:27
(R)eject, accept (t)emporarily or accept (p)ermanently? p
Checked out revision 0.
[dsm@dali ~] $

more…

From here, I need security, and then to import existing cvs repositories into subversion one by one, in liaison with the developers. I also need to restrict access to SSL only, install a third-party certificate, and work further to validate subclipse and eclipse, to where I can roll up a development environs they can just unzip and use.

— doug

share: These icons link to social bookmarking sites where readers can share and discover new web pages.

Tags: apache2, compile, eclipse, subversion, svn.

(2) Comments

What I'm Doing...

flying back to Boston tomorrow, and watching my daughter come off a ventilator and breathe on her own... 2 weeks ago
writing a startup and shutdown sc ript for all of jboss-land 2009-08-25
finished and deployed svnadmin.pl cgi, documented it and checked into subversion... next is more log4j edits, and deploy jsvn (java svn) 2009-05-08
More updates...

Posting tweet...

intuitive engineering

doug munsinger

Archive for the ‘apache2’ Category

* another tool for SVN – list_repositories.pl

* svnadmin.pl – perl cgi script to manage svn over apache

svnadmin.pl

main.css

* subversion compile and install as non-privileged user…

why non-privileged?

what worked

svn + apache2

process

compile subversion

httpd.conf:

validate

more…

recent posts

What I'm Doing...

categories

archives

tag cloud

admin