Subversion Server on Raspberry Pi, part 3

Automated backup to a Dropbox folder

In parts 1 and 2 I documented how I setup a Raspberry Pi to provide Internet-based access to a Subversion server. The task here is to write a Python script that creates a hot copy of a repository, zips it, and sends to an appropriate place for archiving.

Note: after writing my own scripts I found some example scripts including hot-backup.py. This has more goodies than mine, including cleaning up excess (old) backups.

Python scripting notes: to allow a Python script to be ran directly from the command line set the first line to:

#!/usr/bin/env python

and then make it executable with:

jon@raspberrypi ~ $ sudo chmod u+x your_filename.py

The #! text on the first line is called the shebang and tells the shell what type of interpreter to use for the rest of the script. We’ll need to make system calls from the Python script – this site has lots of cool examples.

Dropbox via Python on the RPi

Note: using the Dropbox API requires a Dropbox account.

An application key is required to use the Python API. Go to the Dropbox app console, agree with Terms and Conditions, and select the following options:

D1
Click Create app. The following screen will appear and includes the application key and application secret strings which we’ll need in our Python scripts:

D2
Then download the Dropbox Python SDK:

jon@raspberrypi ~ $ wget https://www.dropbox.com/static/developers/dropbox-python-sdk-1.6.zip
 --2013-08-28 14:09:38-- https://www.dropbox.com/static/developers/dropbox-python-sdk-1.6.zip
 Resolving www.dropbox.com (www.dropbox.com)... 199.47.217.170
 Connecting to www.dropbox.com (www.dropbox.com)|199.47.217.170|:443... connected.
 HTTP request sent, awaiting response... 200 OK
 Length: 812016 (793K) [application/zip]
 Saving to: `dropbox-python-sdk-1.6.zip'
 100%[=======================================================================================>] 812,016 118K/s in 8.3s
 2013-08-28 14:09:53 (96.1 KB/s) - `dropbox-python-sdk-1.6.zip' saved [812016/812016]

Unzip the archive and cleanup:

jon@raspberrypi ~ $ unzip dropbox-python-sdk-1.6.zip
jon@raspberrypi ~ $ rm dropbox-python-sdk-1.6.zip 

Install the Python SetupTools package, then the Dropbox Python SDK :

jon@raspberrypi ~ $ sudo apt-get install python-setuptools
jon@raspberrypi ~ $ cd dropbox-python-sdk-1.6/
jon@raspberrypi ~/dropbox-python-sdk-1.6 $ sudo python setup.py install 

The Dropbox Core API page is useful in developing upload scripts. For this backup we need two scripts:

  1. A temporary script that requires operator interaction to generate a Dropbox access code. This access code will be used by the main script for authentication and is the key to writing non-interactive Dropbox scripts.
  2. A backup script that provides non-interactive repository archiving (via a hotcopy), compression, Dropbox upload, and caretaking.

First the access token: create a new script, db_get_access_token.py (and insert the actual app_key and app_secret strings). (For my scripts I created a new folder, svn_pi, underneath the home folder).

#!/usr/bin/env python
import dropbox
# Get your app key and secret from the Dropbox developer website
app_key = '*** insert ***'
app_secret = '*** insert ***'
 
flow = dropbox.client.DropboxOAuth2FlowNoRedirect(app_key, app_secret)
authorize_url = flow.start()
print '1. Go to: ' + authorize_url
print '2. Click "Allow" (you might have to log in first)'
print '3. Copy the authorization code.'
code = raw_input("Enter the authorization code here: ").strip()
access_token, user_id = flow.finish(code)
client = dropbox.client.DropboxClient(access_token)
 
print 'linked account: ', client.account_info()
print 'Access token ['
print access_token
print ']'

Run the script, copy the URL from the terminal window into a browser, then allow the new app to have access:

D3

D4

Copy the code and paste into the terminal session prompt: ‘linked account‘. If everything works as expected the ‘access token‘ information will be displayed. Otherwise something similar to dropbox.rest.ErrorResponse: [400] u'invalid_grant' will be displayed. Copy the access token string – this is needed to allow dropbox file transfers without operator interaction.

Then create the main backup script, svn_hotcopy_db.py. This will process all the repositories, for each one creating a hot copy, compressing it to a zip file, coping the zip to Dropbox, then finally cleaning up. Replace any *** string sections in the script with the appropriate information before running!

#!/usr/bin/env python

import datetime
import subprocess
import os
import dropbox
import shutil

homePath = '/home/***/'

def makeDateTimeStamp():
	now = datetime.datetime.today()
	nowAsString = now.strftime('%Y_%m_%d_%H_%M_%S')
	return nowAsString

def backupRepository(repoName, db_client):

	# Step 1: make hotcopy
	pathToSourceRepository = homePath + 'svnrepos/' + repoName + '/'
	pathToHotCopyRepository = homePath + 'svn_py/hotcopy/'
	cmdMakeHotCopy = 'svnadmin hotcopy ' + pathToSourceRepository + ' '  + pathToHotCopyRepository
	print cmdMakeHotCopy
	subprocess.call(cmdMakeHotCopy, shell=True)

	# Step 2: zip the hotcopy
	zipFileNameWithoutPath = repoName + '_' + makeDateTimeStamp() + '.zip'
	zipFileName = homePath + 'svn_py/' + zipFileNameWithoutPath
	cmdMakeZip = 'zip -r ' + zipFileName + ' ' + pathToHotCopyRepository
	print cmdMakeZip
	subprocess.call(cmdMakeZip, shell=True)

	# Step 3: send zip file to Dropbox
	print 'Sending zip file to Dropbox'
	with open(zipFileName, 'r') as zipFile:
     	db_client.put_file(zipFileNameWithoutPath, zipFile)

	# Step 4: cleanup
	print 'Cleaning up'
	os.remove(zipFileName)
	shutil.rmtree(pathToHotCopyRepository)

# Configure dropbox access
db_access_token = '***'
db_client = dropbox.client.DropboxClient(db_access_token)

# Setup list of repositories to dump/zip/backup
repositories = ['***', '***', '***']

# Run !
for repository in repositories:
	backupRepository(repository, db_client)

The backup can be scheduled to run automatically. First edit the schedule for the current user:

jon@raspberrypi ~ $ sudo crontab -e

Edit the file and include the following line:

0 2 * * * /home/jon/svn_py/svn_hotcopy_db.py &

Notes:

0 2 * * *: Run at 02:00 every day.
/home/jon/svn_py/svn_hotcopy_db.py: script to run
&: run in background

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s