Subversion, per-repository access

These are my notes on how I configured per-user access rights for my Subversion repositories. There is an excellent article here to help with this.

I run three repositories (for three companies) and need access to all of them. Other users must have access only to the repository for their respective company.

1. Create an access control file called svn_access_control:

jon@raspberrypi ~ $ sudo nano /etc/apache2/svn_access_control

Enter information like this:

# jon belongs to all groups, paul and nigel to one group each
[groups]
TeamName1 = jon, paul
TeamName2 = jon, nigel

# Only admin has access to everything
[/]
admin = rw

# Setup access to a specific repository
[repos_1:/]
@TeamName1 = rw

# Setup access to a specific repository
[repos_2:/]
@Teamname2 = rw

Note: the user names in the above file are case sensitive.

Then edit the dav_svn.conf file and include a reference to the new access control file:

<Location /svn>
 DAV svn
 SVNParentPath /home/jon/svnrepos
 AuthType Digest
 AuthName "Subversion Repo"
 AuthUserFile /etc/apache2/dav_svn.htdigest
 AuthzSVNAccessFile /etc/apache2/svn_access_control
 Require valid-user
</Location>

The following command must be used to enable the new access control:

jon@raspberrypi ~ $ sudo a2enmod authz_svn

Restart the apache service to activate the changes.

Advertisements

Subversion Server on Raspberry Pi, part 3

Automated backup to a Dropbox folder

In parts 1 and 2 I documented how I setup a Raspberry Pi to provide Internet-based access to a Subversion server. The task here is to write a Python script that creates a hot copy of a repository, zips it, and sends to an appropriate place for archiving.

Note: after writing my own scripts I found some example scripts including hot-backup.py. This has more goodies than mine, including cleaning up excess (old) backups.

Python scripting notes: to allow a Python script to be ran directly from the command line set the first line to:

#!/usr/bin/env python

and then make it executable with:

jon@raspberrypi ~ $ sudo chmod u+x your_filename.py

The #! text on the first line is called the shebang and tells the shell what type of interpreter to use for the rest of the script. We’ll need to make system calls from the Python script – this site has lots of cool examples.

Dropbox via Python on the RPi

Note: using the Dropbox API requires a Dropbox account.

An application key is required to use the Python API. Go to the Dropbox app console, agree with Terms and Conditions, and select the following options:

D1
Click Create app. The following screen will appear and includes the application key and application secret strings which we’ll need in our Python scripts:

D2
Then download the Dropbox Python SDK:

jon@raspberrypi ~ $ wget https://www.dropbox.com/static/developers/dropbox-python-sdk-1.6.zip
 --2013-08-28 14:09:38-- https://www.dropbox.com/static/developers/dropbox-python-sdk-1.6.zip
 Resolving www.dropbox.com (www.dropbox.com)... 199.47.217.170
 Connecting to www.dropbox.com (www.dropbox.com)|199.47.217.170|:443... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 812016 (793K) [application/zip]
 Saving to: `dropbox-python-sdk-1.6.zip'
 100%[=======================================================================================>] 812,016 118K/s in 8.3s
 2013-08-28 14:09:53 (96.1 KB/s) - `dropbox-python-sdk-1.6.zip' saved [812016/812016]

Unzip the archive and cleanup:

jon@raspberrypi ~ $ unzip dropbox-python-sdk-1.6.zip
jon@raspberrypi ~ $ rm dropbox-python-sdk-1.6.zip 

Install the Python SetupTools package, then the Dropbox Python SDK :

jon@raspberrypi ~ $ sudo apt-get install python-setuptools
jon@raspberrypi ~ $ cd dropbox-python-sdk-1.6/
jon@raspberrypi ~/dropbox-python-sdk-1.6 $ sudo python setup.py install 

The Dropbox Core API page is useful in developing upload scripts. For this backup we need two scripts:

  1. A temporary script that requires operator interaction to generate a Dropbox access code. This access code will be used by the main script for authentication and is the key to writing non-interactive Dropbox scripts.
  2. A backup script that provides non-interactive repository archiving (via a hotcopy), compression, Dropbox upload, and caretaking.

First the access token: create a new script, db_get_access_token.py (and insert the actual app_key and app_secret strings). (For my scripts I created a new folder, svn_pi, underneath the home folder).

#!/usr/bin/env python
import dropbox
# Get your app key and secret from the Dropbox developer website
app_key = '*** insert ***'
app_secret = '*** insert ***'
 
flow = dropbox.client.DropboxOAuth2FlowNoRedirect(app_key, app_secret)
authorize_url = flow.start()
print '1. Go to: ' + authorize_url
print '2. Click "Allow" (you might have to log in first)'
print '3. Copy the authorization code.'
code = raw_input("Enter the authorization code here: ").strip()
access_token, user_id = flow.finish(code)
client = dropbox.client.DropboxClient(access_token)
 
print 'linked account: ', client.account_info()
print 'Access token ['
print access_token
print ']'

Run the script, copy the URL from the terminal window into a browser, then allow the new app to have access:

D3

D4

Copy the code and paste into the terminal session prompt: ‘linked account‘. If everything works as expected the ‘access token‘ information will be displayed. Otherwise something similar to dropbox.rest.ErrorResponse: [400] u'invalid_grant' will be displayed. Copy the access token string – this is needed to allow dropbox file transfers without operator interaction.

Then create the main backup script, svn_hotcopy_db.py. This will process all the repositories, for each one creating a hot copy, compressing it to a zip file, coping the zip to Dropbox, then finally cleaning up. Replace any *** string sections in the script with the appropriate information before running!

#!/usr/bin/env python

import datetime
import subprocess
import os
import dropbox
import shutil

homePath = '/home/***/'

def makeDateTimeStamp():
	now = datetime.datetime.today()
	nowAsString = now.strftime('%Y_%m_%d_%H_%M_%S')
	return nowAsString

def backupRepository(repoName, db_client):

	# Step 1: make hotcopy
	pathToSourceRepository = homePath + 'svnrepos/' + repoName + '/'
	pathToHotCopyRepository = homePath + 'svn_py/hotcopy/'
	cmdMakeHotCopy = 'svnadmin hotcopy ' + pathToSourceRepository + ' '  + pathToHotCopyRepository
	print cmdMakeHotCopy
	subprocess.call(cmdMakeHotCopy, shell=True)

	# Step 2: zip the hotcopy
	zipFileNameWithoutPath = repoName + '_' + makeDateTimeStamp() + '.zip'
	zipFileName = homePath + 'svn_py/' + zipFileNameWithoutPath
	cmdMakeZip = 'zip -r ' + zipFileName + ' ' + pathToHotCopyRepository
	print cmdMakeZip
	subprocess.call(cmdMakeZip, shell=True)

	# Step 3: send zip file to Dropbox
	print 'Sending zip file to Dropbox'
	with open(zipFileName, 'r') as zipFile:
     	db_client.put_file(zipFileNameWithoutPath, zipFile)

	# Step 4: cleanup
	print 'Cleaning up'
	os.remove(zipFileName)
	shutil.rmtree(pathToHotCopyRepository)

# Configure dropbox access
db_access_token = '***'
db_client = dropbox.client.DropboxClient(db_access_token)

# Setup list of repositories to dump/zip/backup
repositories = ['***', '***', '***']

# Run !
for repository in repositories:
	backupRepository(repository, db_client)

The backup can be scheduled to run automatically. First edit the schedule for the current user:

jon@raspberrypi ~ $ sudo crontab -e

Edit the file and include the following line:

0 2 * * * /home/jon/svn_py/svn_hotcopy_db.py &

Notes:

0 2 * * *: Run at 02:00 every day.
/home/jon/svn_py/svn_hotcopy_db.py: script to run
&: run in background

Subversion Server on Raspberry Pi, part 2

Installing Subversion and Apache

In part 1 I documented how I installed the OS on my Raspberry Pi  –  bread and butter to a seasoned Linux developer but lots of Googling for a Windows programmer, even one using a MacBook Pro.

Now it’s time to install Subversion (for version control) and Apache (to provide secure web access to the Pi and to access the Subversion data).

Using the Mac’s Terminal application and a secure shell (SSH) session we can use apt-get to install the software:

jon@raspberrypi ~ $ sudo apt-get install subversion
jon@raspberrypi ~ $ sudo apt-get install apache2 libapache2-svn

We can test the Apache installation by connecting to the Pi from Safari on the MacBook Pro:

004


Setting up subversion

From the [default] home folder make a new folder to contain all Subversion repositories:

jon@raspberrypi ~ $ mkdir svnrepos

Note: the full path for this folder is: /home/jon/svnrepos

Next we must edit the Subversion/Apache configuration file to set the digest access authorisation mechanism for logging into Subversion via Apache. Digest authentication offers good security as far as user name and passwords are concerned.

The dav_svn.conf file provides configuration information used by Apache when serving Subversion repositories through the HTTP server:

jon@raspberrypi ~ $ sudo nano /etc/apache2/mods-available/dav_svn.conf

Replace the entire contents of the file with the block below:

<Location /svn>
DAV svn
SVNParentPath /home/jon/svnrepos
AuthType Digest
AuthName "Subversion Repo"
AuthUserFile /etc/apache2/dav_svn.htdigest
Require valid-user
</Location>

Note the AutoName property: "Subversion Repo": we’ll need that shortly when we start generating passwords.

Before we can use digest access authentication we must enable it and restart the Apache service:

jon@raspberrypi ~ $ sudo a2enmod auth_digest
jon@raspberrypi ~ $ sudo service apache2 restart

Finally we can set one user and password in the Apache/Subversion digest file:

jon@raspberrypi ~ $ sudo htdigest -c /etc/apache2/dav_svn.htdigest "Subversion Repo" jon

Note 1: the -c parameter: this is used to create a new digest file; remember to omit this once we have the file and want to add/remove/append users and passwords.

Note 2: we use "Subversion Repo" as the realm argument to htdigest; this must match the AuthName used in the dav_svn.conf file.

Make a temporary Subversion repository to help verify that everything is working:

jon@raspberrypi ~ $ svnadmin create svnrepos/pi_test

Use a browser to connect to the Subversion repository using this address:

http://192.168.0.14/svn/pi_test/

The browser will prompt for a user name and password: these must match the same values used when creating the digest authentication file. Once the browser has connected is should display the following:

005

Delete the test repository:

jon@raspberrypi ~ $ rm -rf svnrepos/pi_test/

Note: use -r to recurse sub-directories; use -f to force the deletion of files/folders without asking.


Repository folder ownership – important !

By default the Apache user (www-data, automatically created when Apache is installed) has permissions to read the svnrepos folder, but not to write to it. This means that any Subversion clients will be able to view a repository but not affect any changes. The following command will change the owner of the folder from jon to www-data (but don’t run it):

jon@raspberrypi ~ $ sudo chown -R www-data:www-data /home/jon/svnrepos

This will provide Apache (via the www-data user) with full access to svnrepos but will also  prevent the jon user from being able to access the directory directly. (We could use the chown command again but it’s a bit tedious). A better option would be to have a group with permissions for reading and writing, and to assign this group to the folder. The following commands do this:

jon@raspberrypi ~ $ sudo groupadd svn-super
jon@raspberrypi ~ $ sudo usermod -aG svn-super jon
jon@raspberrypi ~ $ sudo usermod -aG svn-super www-data
jon@raspberrypi ~ $ sudo chown -R :svn-super /home/jon/svnrepos/
jon@raspberrypi ~ $ ls -l

total 4
lrwxrwxrwx 1 jon jon 44 Aug 27 17:46 pistore.desktop -> /usr/share/indiecity/pistore/pistore.desktop
drwxr-xr-x 4 jon svn-super 4096 Aug 28 08:20 svnrepos

Note in the above listing that the user that owns the svnrepos folder is still jon but the group owner is now svn-super.

The new group, svn-super, doesn’t have write permissions so we fix this now, restart the Pi, and review the permissions again

jon@raspberrypi ~ $ sudo chmod -R g+w svnrepos/
jon@raspberrypi ~ $ sudo shutdown -r now

jon@raspberrypi ~ $ ls -l
total 4
lrwxrwxrwx 1 jon jon 44 Aug 27 17:46 pistore.desktop -> /usr/share/indiecity/pistore/pistore.desktop
drwxrwxr-x 4 jon svn-super 4096 Aug 28 08:20 svnrepos

Note the difference in the permissions for the svnrepos directory:

Before: drwxr-xr-x
After: drwxrwxr-x

The format of the above is: TUUUGGGSSS, where:

T: the item type, in this case d for directory.
UUU: user permissions for read, write and execute.
GGG: group permissions for read, write and execute.
SSS: other permissions for read, write and execute.

This link provides more details on the ls -l long format output.


Migrating a repository to the Pi (or restore from a backup)

This section describes how I migrated my PC-based Subversion repositories onto the Pi using the dump and load commands. A hotcopy would probably have been better but at the time I’d gone with dump, mainly because it looked like the simplest option from Subversion Edge.

The migration plan is:

  1. Use SubversionEdge to backup the repositories to .dump files.
  2. Copy the .dump files to the Pi.
  3. Import the each .dump files into a new repository.
  4. Point any existing working copies to the new repository.

Refer to the SubversionEdge Web management tool for backing up a repository to a dump file.

I used SFTP (SSH File Transfer Protocol) to transfer files between the MacBook Pro and the Pi using pre-installed tools I can use SFTP from a terminal session on the Mac to push my .dump files onto the Pi.

Establish a connection:

$ sftp jon@192.168.0.14

Determine the current working directories, first the Pi and then the Mac:

sftp> lpwd
Local working directory: /Users/Jon
sftp> pwd
Remote working directory: /home/jon

Change the local directory to the location of the .dump files:

sftp> lcd "Downloads/Subversion Dumps"
sftp> lpwd
Local working directory: /Users/Jon/Downloads/Subversion Dumps

Push the files from the Mac to the Pi:

sftp> put AmberOptix3-r0_1758-20130827205309.dump.zip
Uploading AmberOptix3-r0_1758-20130827205309.dump.zip to /home/jon/AmberOptix3-r0_1758-20130827205309.dump.zip
AmberOptix3-r0_1758-20130827205309.dump.zip 100% 200MB 2.8MB/s 01:12

Then unzip to get hold of the .dump file:

jon@raspberrypi ~ $ unzip AmberOptix3-r0_1758-20130827205309.dump.zip
Archive: AmberOptix3-r0_1758-20130827205309.dump.zip
inflating: AmberOptix3-r0_1758-20130827205309.dump

Create a new repository and import the .dump file:

jon@raspberrypi ~ $ svnadmin load --force-uuid svnrepos/AmberOptix4/ < AmberOptix3-r0_1758-20130827205309.dump

The --force_uuid parameter will tell svnadmin to use the unique identifiers from the imported repository – this feature will allow users to relocate existing working copies.

Example output from the above:

<<< Started new transaction, based on original revision 1
* adding path : tags ... done.
------- Committed revision 1 >>>
<<< Started new transaction, based on original revision 2
* adding path : trunk ... done.
------- Committed revision 2 >>>
<<< Started new transaction, based on original revision 3
* adding path : trunk/ThirdParty ... done.

Finally we can update the location of the repositories of each working copy. With TortoiseSVN the following steps are required:

  1. Right-click on the top-level folder of the working copy.
  2. Select TortoiseSVN | Relocate, enter the new location:

006
Hit ok and wait the the confirmation:

007

In the final part of this write up I’ll document how I used a Python script and the Dropbox API to make a nightly backup of all my repositories.

Subversion Server on Raspberry Pi, part 1

Introduction

These are my notes on how I configured a Raspberry Pi to run Subversion via an Apache server with secure Internet access and automated daily backups to Dropbox. The image below shows the configuration I was aiming for:

SVN RPi Layout

Until last week I’d been managing a couple of Subversion repositories on an old and noisy PC running the excellent CollabNet SubversionEdge. I’d also been looking for a project for my Raspberry Pi to help me learn a little Linux and some Python scripting. Finally I wanted a new system that could be put together by other people I work with, primarily the imaging specialists at Amber Optix whose Subversion repository I manage.

Environment

* MacBook Pro: although I’m primarily a Windows developer I work on a MBP retina with a solid state drive because, well, it’s just awesome. I run virtual machines, courtesy of VMWare Fusion, manage my notes in Evernote, my files with Dropbox, and OS X for web, email, IOS development and everything else. TortoiseSVN provides the Subversion interface via Windows Explorer.

* Raspberry Pi, Model-B, with 4Gb SD card, connected via Ethernet to my router. (My router is an Apple Time Capsule in Bridge Mode, connected to a Sky ADSL modem/router).

Installing the Operating System

(Another neat article here on how to do this).

Download the latest version of Raspbian “Wheezy” from  http://www.raspberrypi.org/downloads (version 2013-07-26 at the time of writing). Unzip the downloaded file to get the operating system image. E.g.

  • 2013-07-26-wheezy-raspbian.img

Before the OS image can be copied onto the SD card we must find the device name. One technique is to list the free disk space of all devices on the MBP with the SD card removed, then repeat the listing with the SD card inserted:

With the SD card not inserted:

$ df -lH
Filesystem                          Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk0s2                        250G   205G    44G    83% 50217398 10852042   82%   /
localhost:/Ev5H8qvZlfqLy7rY7p0zks   250G   250G     0B   100%        0        0  100%   /Volumes/MobileBackups

With the SD card inserted:

$ df -lH
Filesystem                          Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk0s2                        250G   205G    44G    83% 50217398 10852042   82%   /
localhost:/Ev5H8qvZlfqLy7rY7p0zks   250G   250G     0B   100%        0        0  100%   /Volumes/MobileBackups
/dev/disk2s1 59M 19M 39M 34% 512 0 100% /Volumes/boot

This identifies /dev/disk2s1 as the SD card device, where 2 is the disk number.

We’re going to use the dd command to copy the image to the SD card, but we must first  unmount the Pi it so that we have full access to it:

$ diskutil unmount /dev/disk2s1

Then we copy the OS image to the device using the dd command:

$ sudo dd bs=1m if=./Downloads/2013-07-26-wheezy-raspbian.img of=/dev/rdisk2

Note that the destination is formed of /dev/rdisk{x} where x is is the disk number seen earlier with the df -lH command.

Once the block copy is complete we should results similar to those shown below:

1850+0 records in
1850+0 records out
1939865600 bytes transferred in 268.153202 secs (7234169 bytes/sec)

Insert the card into the Raspberry Pi – don’t power up yet.


Connecting to the Pi without a monitor

The Pi comes with a HDMI slot but in my office I don’t have another monitor and I wanted to be able to do everything with just the MBP and terminal sessions.

This site has useful info on configuring the Pi for a variety of routers. I have a Sky router so I logged into the administration pages and looked for the list of connected devices, first with the Pi disconnected and then with it connected:

001

I then used another administration page to reserve the address 192.168.0.14 for the Pi:

002

Once the Pi’s IP is fixed on the router we can continue. Run a terminal session and use SSH (Secure Shell) to gain access to the Pi:

$ ssh pi@192.168.0.14

[Note: if this procedure has been performed before the initial SSH may fail: in this case use the following to remove all keys belonging to the remote host (the Pi) from the locally stored host information:

$ ssh-keygen -R 192.168.0.14

Repeating the SSH command will now work.]

After the first login the output will be similar to:

$ ssh pi@192.168.0.14
The authenticity of host '192.168.0.14 (192.168.0.14)' can't be established.
RSA key fingerprint is 88:8b:92:32:13:d1:0b:3d:10:a6:a0:d7:08:a5:d1:66.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.14' (RSA) to the list of known hosts.
pi@192.168.0.14's password: {raspberry, the default password}
Linux raspberrypi 3.6.11+ #474 PREEMPT Thu Jun 13 17:14:42 BST 2013 armv6l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
NOTICE: the software on this Raspberry Pi has not been fully configured. Please run 'sudo raspi-config'


Prepare the PI

Before starting on the Subversion and Apache installation it’s worth getting the OS updated as much as possible. From within an SSH session run:

$ sudo raspi-config

The following screen will appear:

003

Select option 1 to expand the file system, then reboot the device.

Tip: to reboot the Pi from the SSH session use:

$ sudo shutdown -r now

Next we’ll change the Pi default user’s password:

$ sudo passwd pi

And then we’ll add a new user for general use so that the Pi user is reserved for purely administrative tasks – not necessary but seems to be the normal pattern for Linux systems. For this walkthrough the user jon will be used as the main user on the Pi and in the Subversion user list.

$ sudo adduser jon

We’ll grant this user the ability to use sudo to perform key tasks. If we don’t do this then any attempt to use sudo from the jon account will result in a message like this:

jon is not in the sudoers file. This incident will be reported.

So we use:

$ sudo usermod -aG sudo jon

Then we log out using logout and connect via ssh again but this time for the jon account:

$ ssh jon@192.168.0.14

Finally we will update the software on the PI using these two commands:

$ sudo apt-get update
$ sudo apt-get upgrade

That’s the basic Pi setup complete. In the next part I’ll cover how I configured Subversion.