Posts Tagged ‘svn’
* another tool for SVN – list_repositories.pl
Posted on May 11th, 2009 by doug. Filed under Solaris, apache2, eclipse, subversion, tools, websphere.
One of the features of subversion + apache2 is the ability to list repositories – natively.
SVNParentPath /apps/repos
SVNListParentPath on
Unfortunately, once you restrict the httpd.conf to individual repositories and start handling permissions separately you lose that. Both of these permissions, set at the top level, at the parent path to the repositories, have to be commented out to have the individual permissions on the directories below take effect.
And listing just the repositories wasn’t enough for what I had in mind – I wanted a read-only table showing an individual developer or a team lead for a project who has what permissions for the subversion repositories. All the users, all the repositories.
Just your basic cgi.
and the script:
# script to parse and display users and repositories and rights (R, RW)
use strict;
use CGI qw(:standard);
use CGI::Carp qw(fatalsToBrowser);
##----------------------------
## Variables
##----------------------------
#debug
my $debug = 0; #set to 0 to turn off, 1 (true) to turn on
# title for page
my $PageTitle = "List of SVN Repositories";
## repository directory
my $SVN = "/apps/repos";
## location for htpasswd files
my $HTDIR = "/apps/apache2/conf/htpasswd";
## Set untainted path
$ENV{PATH} = '/apps/apache2/bin:/bin:/usr/bin:/usr/local/bin';
$ENV{IFS} = "" if $ENV{IFS} ne "";
# css
my $css = "http://<your server name>/css/main.css";
my $headerimg = "http://<your server name>/css/roger_rabbit_120.jpg";
##------------------------------
## MAIN
##------------------------------
&standard_header;
my ($ref_repos, $ref_tabledata, $ref_users) = &CreateTableSpace;
&DisplayTable ($ref_repos, $ref_tabledata, $ref_users);
&standard_footer;
exit;
##------------------------------
## subs
##------------------------------
sub standard_header {
print header();
print start_html(-Title => "$PageTitle", -BGCOLOR=>"White",
-style => {
-src => "${css}"
}
);
print ("<div id=\"header\">\n");
print p("<img src=\"${headerimg}\" title=\"Wells logo\" alt=\"wells logo\"/>\n");
print ("</div>\n");
print ("<div id=\"headertitle\">\n");
print h3("Repositories<br/>\nusers | read (R) | read & write (RW)\n"); # start_multipart_form() if file upload
}
sub standard_footer {
print end_html();
}
sub CreateTableSpace {
my $ref_repos = &GetBlankRepos;
my @repos = @$ref_repos; # dereference
my %tabledata = (); # hash to hold table data
my @users = (); # list of users
my %seen = ();
foreach my $rep (@repos) {
open(FILE, "$HTDIR/${rep}_read") || croak "Failed to open $HTDIR/${rep}_read for reading...";
my @filelines = <FILE>;
close FILE;
foreach my $line (@filelines) {
# lines are user:passwd
my ($user, $pass) = split (":", $line);
unless ($seen{$user}) {
$seen{$user} = 1; # save as seen
push (@users, $user); # save the user to a list
}
$tabledata{$rep}{$user} = "read";
}
open (FILE, "$HTDIR/${rep}_write") || croak "Failed to open $HTDIR/${rep}_write for reading";
my @file_lines = <FILE>;
close FILE;
foreach my $line (@file_lines) {
# lines are user:passwd
my ($user, $pass) = split (":", $line);
unless ($seen{$user}) {
$seen{$user} = 1; # save as seen
push (@users, $user); # save the user to a list
}
$tabledata{$rep}{$user} = "readwrite";
}
}
return (\@repos, \%tabledata, \@users);
}
sub GetBlankRepos {
my @repos = ();
# list $SVN
opendir (DIR, $SVN) || croak "Failed to open directory $SVN for reading...";
while (defined(my $file = readdir(DIR))) {
# skip ".", ".." and .<hidden> files...
if ($file =~ /^\./) {
next;
} else {
push(@repos, $file);
}
}
return (\@repos);
}
sub DisplayTable {
my $ref_repos = shift; #@repos
my $ref_tabledata = shift; # %tabledata
my $ref_users = shift; # @users
# dereference
my @repos = @$ref_repos;
my %tabledata = %$ref_tabledata;
my @users = @$ref_users;
# repos across the top, users down, R or RW for permissions
# $tabledata{$rep}{$user} = "readwrite";
# start table
print ("<table>\n<tbody>\n");
# table header
my $cols = ($#repos + 1);
print ("<tr><td>Users</td><td colspan=\"$cols\">Repositories</td></tr>\n");
print ("<div id=\"repotitles\"><tr>\n<td> </td>");
foreach my $rep (@repos) {
print ("<td>$rep</td>");
}
print ("\n</tr>\n</div>\n");
foreach my $user (@users) {
print ("<tr>\n<td>$user</td>");
foreach my $repo (@repos) {
if ($tabledata{$repo}{$user}) {
if ($tabledata{$repo}{$user} eq "read") {
print ("<td>R</td>");
} elsif ($tabledata{$repo}{$user} eq "readwrite") {
print ("<td>RW</td>");
}
} else {
print ("<td> - </td>");
}
}
print ("\n</tr>\n");
}
# finish table
print ("</tbody>\n</table>\n");
}
* svnadmin.pl – perl cgi script to manage svn over apache
Posted on May 8th, 2009 by doug. Filed under apache2, perl, subversion.
One of the tedious tasks in repository administration is managing users over repositories. Who has access to what repository and to what degree (read-only, or write). Subversion over apache2 allows a tremendous amount of control, down to individual directories within the repository. (see "Per Directory Access Control" in the subversion book).
So far I haven’t placed that but I have set up a default deny and then separate htpasswd files for read, or write access permission. This does at times cause TWO passwords to need to be used to first read and then again write to a repository. However, these are cached, so we’ll see how much developers find to complain about in that.
The script svnadmin.pl assumes that a naming convention for the relationship bewtween htpasswd access files and subversion repositories is set so that the htpasswd file is named <repository_name>_read and <repository_name>_write. You need to setthe path to htpasswd, the path to the htpasswd files, and a location for the top level directory of subversion – the parent directory for all repositories. Within the script you’ll also need to set the paths for the css file ($css) and for the header image ($headerimg). This image should be roughly 420 px wide x 200 px high. I used my corporate logo.
The script also uses a username-as-corporate-ID assumption (begins with "a" or "d" or "x", contains up to eight characters), and a reasonable password assumption (at least 8 charcaters, nor more than 12, must contain at least one capital letter and one digit). This is for internal use, not to be exposed, so if you are going to do something like this on ethe internet, you would want to revisit that and lock it down further.
svnadmin.pl
main.css
To run the script – save-file-as and then change the first line to "#! /usr/bin/perl" (or the appropriate path for the perl you want to use). This is basically removing the extra "#" mark. Rename the script to "svnadmin.pl", put in your cgi-bin location, and put the main.css file in an appropriate location. Edit the variables as above.
Screenshots:
 |
 |
 |
 |
Really fairly simple. This started with a script called "htpasswd.pl" which I downloaded and reworked, then adapted to use CGI.pm for multi-screen form presentation. That framework alone was work rediscovering. I used a similar framework with perl DBI and mysql to do fluid reporting on large system installations 9 years ago. It recurs.
— doug
* subversion compile and install as non-privileged user…
Posted on April 1st, 2009 by doug. Filed under apache2, compile, eclipse, subversion.
Most Open Source software projects invite persons using their product to experience the joys of compiling the product. Subversion does not.
from INSTALL in the 1.5.6 subversion source code:
This document is written for people who intend to build Subversion from source code. Normally, the only people who do this are Subversion developers and package maintainers.
If neither of these labels fits you, we recommend you find an appropriate binary package of Subversion and install that.
why non-privileged?
I work on Solaris 10, in a large financial environment, with separation of responsibilities and a restricted availability of “root”. Most software packages assume that you will be installing as root and they use common and accessible directory locations like “/usr/local”. Most software will build easily or with slight difficulty on linux, and can be much more difficult on Solaris. You will be missing tools and libraries common to Linux.
I currently use CVS, and that’s got some shortcomings – it doesn’t handle binaries easily or consistently, you can’t move a directory without losing history, and it is no longer under development. I have a new container, a Solaris 10 sparse container, in which I wanted to compile and install apache2 and subversion as a non-privileged user. No root. No access to install extra needed packages on /usr/local. On the other hand, complete control over the application without having to consult with the group responsible for root on the container…
what worked
Here’s what worked. Not the dead ends, but what actually resulted in apache and subversion installed, along with apr, apr-util and serf, in a filesystem /apps. This compilation uses serf and FSFS in subversion – it blocks out Berkeley DB and neon.
Berkeley was installed in /usr/local, but I couldn’t get the configure script to find it – and in reading further FSFS is the default in subversion, and has stabilized and become a better solution than Berkeley DB. If you reboot in an unstable state, FSFS will simply recover, where Berkeley DB can require a DBA to recover the database before the system returns to useful.
neon seemed from some errors I encountered to have filesystem access issues (I got a long dump at one point of memory addresses during the make process, all locations for neon libraries and files). Once I eliminated neon, I also got an svnadmin binary that could create a repository without causing “BUS ERROR: segmentation fault”. Of course, I also dropped back from subversion 1.6.0 code to 1.5.6, and compiled and pointed to apr and apr-util from subversion-deps for 1.5.6, and serf from google. It could be any one of those changes.
This was not as simple as compile and go – this was running “configure” and resolving errors, then “make” and resolve yet more errors, and finally, building mod_dav_svn in subversion successfully, installing and finding a “BUS Error: segmentation fault” on running “./svnadmin create /apps/repos/testme”.
I compiled subversion fairly quickly to use svnserver. That compile and install worked, and didn’t create a core file, didn’t seg fault. But it left some problems.
svn + apache2
Many of my developers work from windows. Creating secure access over ssh and svn (svn+ssh://
I determined best practice pointed us to subversion access through web_dav_svn -> web_dav -> apache2 (https://
What I wanted was:
- untar subversion-1.5.6 and subversion-deps-1.5.6 in /apps/src (they layer over each other)
- first build and install the -deps apr and apr-util into /apps/local
- then build apache2 against /apps/local/apr and /apps/local/apr-util
- install apache2
- build and install serf
- remove serf, apr, and apr-util subdirectories and source code from within subversion-1.5.6
- build 1.5.6 against apxs in apache, without Berkeley DB, without neon, and specifying /apps/local/apr, /apps/local/apr-util, and /apps/local/serf
- install and test
process
- create subdirectory /apps/src, place all tarballs in this directory
- untar subversion & subversion-deps version 1.5.6 (these tar onto each other)
- cd subversion-1.5.6/apr
- ./configure –prefix=/apps/local/apr
- make && make install
- cd ../apr-util
- ./configure –prefix=/apps/local/apr-util –with-apr=/apps/local/apr
- make && make install
- apache2: untar httpd-2.2.11
- ./configure –enable-dav –enable-so –prefix=/apps/apache2_2.2.11 –enable-ssl –with-ssl=/usr/local/ssl –with-apr=/apps/local/apr –with-apr-util=/apps/local/apr-util –enable-modules –enable-mods-shared=”most ssl dav”
- make && make install
This way apache2 builds against apr and apr-util compatible with subversion 1.5.6
and then build subversion against it as well.. - ln -s /apps/apache2_2.2.11 /apps/apache2
- compile and install serf
- ./configure –prefix=/apps/local/serf –with-openssl=/usr/local/ssl
- make && make install
- remove serf, pr and apr-util from subversion
compile subversion
./configure –prefix=/apps/svn –with-ssl –with-libs=/usr/local/ssl –without-berkeley-db –with-apxs=/apps/apache2/bin/apxs –with-openssl=/usr/local/ssl –without-neon –with-serf=/apps/local/serf –with-apr=/apps/local/apr –with-apr-util=/apps/local/apr-util
make && make install
test:
[appadmin@sccvsapp01p bin] $ ./svnadmin create /apps/repos/testtoo
[appadmin@sccvsapp01p bin] $
no core dump…
ONWARD to configure and test apache2 and subversion…
httpd.conf:
- change all references to apache2_2.2.11 to apache2 (makes the httpd.conf generic rather than subject to needing a migration after a point release upgrade…)
- change port 80 to a non-priveliged port (8080)
- check for
- LoadModule dav_module modules/mod_dav.so
- LoadModule dav_module modules/mod_dav.so
- LoadModule ssl_module modules/mod_ssl.so
- add in ServerName hostname.domain.com:8443 Some of the apache level sanity validation requires a statement of the local host.
- add in SSL stuff (this IS httpd from source – the default httpd.conf had the ssl-module load statement, but no explicit SSL configuration
#
# Note: The following must must be present to support
# starting without SSL on platforms with no /dev/random equivalent
# but a statically compiled-in mod_ssl.
#
<IfModule ssl_module>
SSLRandomSeed startup builtin
SSLRandomSeed connect builtin
</IfModule>
# =================================================
# SSL/TLS settings
# =================================================
Listen 0.0.0.0:8080
Listen 0.0.0.0:8443
SSLEngine on
#SSLOptions +StrictRequire
#<Directory />
# SSLRequireSSL
#</Directory>
SSLProtocol -all +TLSv1 +SSLv3
SSLCipherSuite HIGH:MEDIUM:!aNULL:+SHA1:+MD5:+HIGH:+MEDIUM
SSLMutex file:/apps/apache2/logs/ssl_mutex
SSLRandomSeed startup file:/dev/urandom 1024
SSLRandomSeed connect file:/dev/urandom 1024
SSLSessionCache shm:/apps/apache2/logs/ssl_cache_shm
SSLSessionCacheTimeout 600
SSLPassPhraseDialog builtin
SSLCertificateFile /apps/apache2/conf/ssl.crt/server.crt
SSLCertificateKeyFile /apps/apache2/conf/ssl.key/server.key
SSLVerifyClient none
SSLProxyEngine off
<IfModule mime.c>
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl
</IfModule>
- create ssl certificate (self-signed). I installed in /apps/apache2/conf/ssl.crt and ssl.key, naming the .crt and ,key files for the server hostname and then symbolically linking them to the generic “server.crt” and “server.key”.
- restart apache2 and test https://host:8443/ – you should get the “It works!” apache test page, thus validating the SSL certificate and setup from a browser level
- parent directory for svn
<Location /svn>
DAV svn
SVNParentPath /apps/repos
</Location>
Restart apache2, and go look at the log – should now show the svn component:
[Tue Mar 31 11:01:52 2009] [notice] Digest: generating secret for digest authentication ...
[Tue Mar 31 11:01:52 2009] [notice] Digest: done
[Tue Mar 31 11:01:52 2009] [notice] Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8d configured -- resuming normal ope
rations
[Tue Mar 31 15:15:44 2009] [notice] caught SIGTERM, shutting down
[Tue Mar 31 15:15:48 2009] [warn] Init: Session Cache is not configured [hint: SSLSessionCache]
[Tue Mar 31 15:15:51 2009] [notice] Digest: generating secret for digest authentication ...
[Tue Mar 31 15:15:51 2009] [notice] Digest: done
[Tue Mar 31 15:15:51 2009] [notice] Apache/2.2.11 (Unix) DAV/2 mod_ssl/2.2.11 OpenSSL/0.9.8d SVN/1.5.6 configured -- resuming
normal operations
validate
I used eclipe with subclipse installed in it – the url for the repository was https://host:8443/svn/testme
On the command line
[dsm@dali ~] $ svn checkout https://hostname:8443/svn/testme
Error validating server certificate for 'https://hostname:8443':
- The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
- Hostname: hostname.domain.com
- Valid: from Wed, 01 Apr 2009 14:44:42 GMT until Thu, 01 Apr 2010 14:44:42 GMT
- Issuer: issuer, Massachusetts, US
- Fingerprint: ec:74:42:f4:98:0b:5c:62:14:34:85:14:60:38:73:1b:bc:8d:18:27
(R)eject, accept (t)emporarily or accept (p)ermanently? p
Checked out revision 0.
[dsm@dali ~] $
more…
From here, I need security, and then to import existing cvs repositories into subversion one by one, in liaison with the developers. I also need to restrict access to SSL only, install a third-party certificate, and work further to validate subclipse and eclipse, to where I can roll up a development environs they can just unzip and use.
— doug
recent posts
- home to Boston, daughter in remission
- visually healthy bone marrow…
- matter of the lungs
- Fall through code to a success…
- another tool for SVN – list_repositories.pl
- svnadmin.pl – perl cgi script to manage svn over apache
- testing Crosspress (plugin)…
- subversion compile and install as non-privileged user…
What I'm Doing...
- flying back to Boston tomorrow, and watching my daughter come off a ventilator and breathe on her own... 2 weeks ago
- writing a startup and shutdown sc ript for all of jboss-land 2009-08-25
- finished and deployed svnadmin.pl cgi, documented it and checked into subversion... next is more log4j edits, and deploy jsvn (java svn) 2009-05-08
- More updates...
Posting tweet...