Raspberry Pi Desktop virtual machine

This weekend I discovered that there is a Linux distribution, Debian Jessie, that now has the Raspberry Pi Desktop, download from here.

So I installed it as a virtual machine on my Windows 10 PC using VMWare Professsional Pro 14. The only difficulty was getting VMWare Tools working to allow automatic screen resizing and file sharing. Found some good info on the VMWare communities site (search for vmware tools debian jessie). From this thread there is a great perl script to help install VMWare Tools. The only modification I made was to delete –default from the line that runs the VMWare installer, without which the script will suggest an alternative option and abandon the installation.

I made a YouTube video of my virtual machine installation and VMWare tools configuration:

 

The script for VMWare tools:

#!/bin/bash 
 
sudo apt-get update 
sudo apt-get upgrade 
echo "Do go and mount your cdrom from the VMware menu" 
echo "press any key to continue" 
read -n 1 -s 
mount /dev/cdrom 
cd /media/cdrom0 
cp VMwareTools-*.tar.gz /tmp 
cd /tmp 
tar xvzf VMwareTools-*.tar.gz 
cd vmware-tools-distrib/ 
sudo apt-get install --no-install-recommends libglib2.0-0 
sudo apt-get install --no-install-recommends build-essential 
sudo apt-get install --no-install-recommends gcc-4.3 linux-headers-`uname -r` 
sudo ./vmware-install.pl 
 
sudo /etc/init.d/networking stop 
sudo rmmod pcnet32 
sudo rmmod vmxnet 
sudo modprobe vmxnet 
sudo /etc/init.d/networking start

 

 

 

Advertisements

Automated Dynamic DNS updating on Raspberry Pi

I use a Raspberry Pi to host SVN and Git repositories but don’t have a static IP with my BT Infinity account. To support remote connectivity I use a dynamic DNS service from Dyn.

The only feature missing was the ability to detect when the IP address had changed, and then to automatically update it.

So with a bit lot of Googling I’ve put together a Python 3 script which is run hourly via crontabThe script uses Dyn’s IP address utility to find the current IP address. This is compared to the last known IP address which I store in a text file. If the address has changed then Dyn’s Perform Update utility is used to sent the new IP address. Any changes or errors are emailed back to me.

Here’s the script:


#!/usr/bin/env python3

import re
import urllib.request
import os
import sys
import smtplib

# -----------------------------------------------------------------------------
# variable: lastIPAddressFileName
#
# The name of a file used to store the last committed IP address.
# -----------------------------------------------------------------------------
lastIPAddressFileName = '/var/tmp/last_router_ip'

# -----------------------------------------------------------------------------
# function: mailInfo
#
# Emails a message to one or more recipients. This should just be used for
# errors and successful changes of IP. (I.e. not used for regular no-change
# messages).
# -----------------------------------------------------------------------------
def mailInfo(info):
	#print('Sending email [{0}]'.format(info))
	from email.mime.text import MIMEText

	msg = MIMEText(
	"Hello from the Raspberry Pi DNS IP update utility.\n"
	"I have a new message for you:\n\n{0}".format(info))

	msg['Subject'] = 'CDS.RPi DSN IP Utility'
	msg['From'] = 'CDS RPi'
	msg['To'] = 'jon@carpediemsystems.co.uk'

	s = smtplib.SMTP('smtp.gmail.com', 587)
	s.ehlo()
	s.starttls()
	s.ehlo()
	s.login('jon@carpediemsystems.co.uk', '******')
	s.sendmail('jon@carpediemsystems.co.uk', ['jon@carpediemsystems.co.uk'], msg.as_string())
	s.quit()

# -----------------------------------------------------------------------------
# function: findRouterIP
#
# Returns the IP of the router by scraping the results of the DYN DNS check
# IP web site. None is returned if the address cannot be found.
# -----------------------------------------------------------------------------
def findRouterIP():

	ipAddress = None
	webAddress = 'http://checkip.dyndns.org/'
	siteData = urllib.request.urlopen(url = webAddress).read()
	siteAsText = str(siteData)
	ipAddresses = re.findall('\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}', siteAsText)

	if(len(ipAddresses) == 1):
		ipAddress = ipAddresses[0]

	return ipAddress

# -----------------------------------------------------------------------------
# function: dynDnsPerformUpdate
#
# Updates the IP address of a DynDNS host using Dyn's Perform Update request.
# The resulting web page is scraped and the function will return True if one
# of the valid results codes is seen (meaning the IP address was either updated
# or was not changed).
# -----------------------------------------------------------------------------
def dynDnsPerformUpdate(username, password, hostname, ipAddress):

	url = 'http://{0}:{1}@members.dyndns.org/nic/update?hostname={2}&myip={3}&wildcard=NOCHG&mx=NOCHG&backmx=NOCHG'.format(username, password, hostname, ipAddress)
	fancyOpener = urllib.request.FancyURLopener()
	siteData = fancyOpener.open(url).read()
	siteAsText = str(siteData)

	acceptableResults = ['good', 'nochg']
	isOk = len([r for r in acceptableResults if r in siteAsText]) > 0
	return isOk

# -----------------------------------------------------------------------------
# function: checkForNewIPAddress
#
# Finds the current router IP address and compares it to the last address
# saved to the lastIPAddressFileName file (if it exists).
#
# Returns a tuple:
#	Current IP address (string)
#	Is this a new and valid IP address (boolean)
# -----------------------------------------------------------------------------
def checkForNewIPAddress():
	lastIPAddress = None

	if(os.path.exists(lastIPAddressFileName) == True):
		with open(lastIPAddressFileName, 'r') as f:
			lastIPAddress = f.read()

	currentIPAddress = findRouterIP()
	isNewIPAddress = ((currentIPAddress != None) and (currentIPAddress != lastIPAddress))

	return currentIPAddress, isNewIPAddress

# -----------------------------------------------------------------------------
# function: saveNewIPAddress
#
# Sends the new IP address to Dyn DNS. This is also saved to the
# lastIPAddressFileName file if the web operation succeeded.
#
# Returns True if the operation succeeded.
# -----------------------------------------------------------------------------
def saveNewIPAddress(newIPAddress):
	username = '******'
	password = '******'
	host = '******.dnsalias.net'
	didUpdate = dynDnsPerformUpdate(username = username, password = password, hostname = host, ipAddress = newIPAddress)

	if(didUpdate == True):
		with open(lastIPAddressFileName, 'w') as f:
			f.write(newIPAddress)

	return didUpdate

# -----------------------------------------------------------------------------
# Script main
# -----------------------------------------------------------------------------

try:
	currentIPAddress, isNewIPAddress = checkForNewIPAddress()

	if(currentIPAddress == None):
		mailInfo('ERROR: failed to get current IP address')
	elif(isNewIPAddress == False):
		print('IP address is unchanged - nothing to do')
	else:
		saveNewIPAddress(currentIPAddress)
		mailInfo('IP address updated to {0}'.format(currentIPAddress))
except:
   mailInfo("ERROR: An unexpected error was detected in the script:", sys.exc_info()[0])

Note: the scripting presentation in this post was done by using WordPress.com’s sourcecode widget, as described here.

The script was saved to:

 /bin/DNS_IP_Util.py

and made executable with:

sudo chmod +x DNS_IP_Util.py

It was scheduled to run hourly by editing root’s cron file. The command to run the editor is:

sudo crontab -e

The cron file was edited to include the following two lines:

# DNS update, every day, hourly
0 * * * * /bin/DNS_IP_Util.py &

Subversion Server on Raspberry Pi, part 2

Installing Subversion and Apache

In part 1 I documented how I installed the OS on my Raspberry Pi  –  bread and butter to a seasoned Linux developer but lots of Googling for a Windows programmer, even one using a MacBook Pro.

Now it’s time to install Subversion (for version control) and Apache (to provide secure web access to the Pi and to access the Subversion data).

Using the Mac’s Terminal application and a secure shell (SSH) session we can use apt-get to install the software:

jon@raspberrypi ~ $ sudo apt-get install subversion
jon@raspberrypi ~ $ sudo apt-get install apache2 libapache2-svn

We can test the Apache installation by connecting to the Pi from Safari on the MacBook Pro:

004


Setting up subversion

From the [default] home folder make a new folder to contain all Subversion repositories:

jon@raspberrypi ~ $ mkdir svnrepos

Note: the full path for this folder is: /home/jon/svnrepos

Next we must edit the Subversion/Apache configuration file to set the digest access authorisation mechanism for logging into Subversion via Apache. Digest authentication offers good security as far as user name and passwords are concerned.

The dav_svn.conf file provides configuration information used by Apache when serving Subversion repositories through the HTTP server:

jon@raspberrypi ~ $ sudo nano /etc/apache2/mods-available/dav_svn.conf

Replace the entire contents of the file with the block below:

<Location /svn>
DAV svn
SVNParentPath /home/jon/svnrepos
AuthType Digest
AuthName "Subversion Repo"
AuthUserFile /etc/apache2/dav_svn.htdigest
Require valid-user
</Location>

Note the AutoName property: "Subversion Repo": we’ll need that shortly when we start generating passwords.

Before we can use digest access authentication we must enable it and restart the Apache service:

jon@raspberrypi ~ $ sudo a2enmod auth_digest
jon@raspberrypi ~ $ sudo service apache2 restart

Finally we can set one user and password in the Apache/Subversion digest file:

jon@raspberrypi ~ $ sudo htdigest -c /etc/apache2/dav_svn.htdigest "Subversion Repo" jon

Note 1: the -c parameter: this is used to create a new digest file; remember to omit this once we have the file and want to add/remove/append users and passwords.

Note 2: we use "Subversion Repo" as the realm argument to htdigest; this must match the AuthName used in the dav_svn.conf file.

Make a temporary Subversion repository to help verify that everything is working:

jon@raspberrypi ~ $ svnadmin create svnrepos/pi_test

Use a browser to connect to the Subversion repository using this address:

http://192.168.0.14/svn/pi_test/

The browser will prompt for a user name and password: these must match the same values used when creating the digest authentication file. Once the browser has connected is should display the following:

005

Delete the test repository:

jon@raspberrypi ~ $ rm -rf svnrepos/pi_test/

Note: use -r to recurse sub-directories; use -f to force the deletion of files/folders without asking.


Repository folder ownership – important !

By default the Apache user (www-data, automatically created when Apache is installed) has permissions to read the svnrepos folder, but not to write to it. This means that any Subversion clients will be able to view a repository but not affect any changes. The following command will change the owner of the folder from jon to www-data (but don’t run it):

jon@raspberrypi ~ $ sudo chown -R www-data:www-data /home/jon/svnrepos

This will provide Apache (via the www-data user) with full access to svnrepos but will also  prevent the jon user from being able to access the directory directly. (We could use the chown command again but it’s a bit tedious). A better option would be to have a group with permissions for reading and writing, and to assign this group to the folder. The following commands do this:

jon@raspberrypi ~ $ sudo groupadd svn-super
jon@raspberrypi ~ $ sudo usermod -aG svn-super jon
jon@raspberrypi ~ $ sudo usermod -aG svn-super www-data
jon@raspberrypi ~ $ sudo chown -R :svn-super /home/jon/svnrepos/
jon@raspberrypi ~ $ ls -l

total 4
lrwxrwxrwx 1 jon jon 44 Aug 27 17:46 pistore.desktop -> /usr/share/indiecity/pistore/pistore.desktop
drwxr-xr-x 4 jon svn-super 4096 Aug 28 08:20 svnrepos

Note in the above listing that the user that owns the svnrepos folder is still jon but the group owner is now svn-super.

The new group, svn-super, doesn’t have write permissions so we fix this now, restart the Pi, and review the permissions again

jon@raspberrypi ~ $ sudo chmod -R g+w svnrepos/
jon@raspberrypi ~ $ sudo shutdown -r now

jon@raspberrypi ~ $ ls -l
total 4
lrwxrwxrwx 1 jon jon 44 Aug 27 17:46 pistore.desktop -> /usr/share/indiecity/pistore/pistore.desktop
drwxrwxr-x 4 jon svn-super 4096 Aug 28 08:20 svnrepos

Note the difference in the permissions for the svnrepos directory:

Before: drwxr-xr-x
After: drwxrwxr-x

The format of the above is: TUUUGGGSSS, where:

T: the item type, in this case d for directory.
UUU: user permissions for read, write and execute.
GGG: group permissions for read, write and execute.
SSS: other permissions for read, write and execute.

This link provides more details on the ls -l long format output.


Migrating a repository to the Pi (or restore from a backup)

This section describes how I migrated my PC-based Subversion repositories onto the Pi using the dump and load commands. A hotcopy would probably have been better but at the time I’d gone with dump, mainly because it looked like the simplest option from Subversion Edge.

The migration plan is:

  1. Use SubversionEdge to backup the repositories to .dump files.
  2. Copy the .dump files to the Pi.
  3. Import the each .dump files into a new repository.
  4. Point any existing working copies to the new repository.

Refer to the SubversionEdge Web management tool for backing up a repository to a dump file.

I used SFTP (SSH File Transfer Protocol) to transfer files between the MacBook Pro and the Pi using pre-installed tools I can use SFTP from a terminal session on the Mac to push my .dump files onto the Pi.

Establish a connection:

$ sftp jon@192.168.0.14

Determine the current working directories, first the Pi and then the Mac:

sftp> lpwd
Local working directory: /Users/Jon
sftp> pwd
Remote working directory: /home/jon

Change the local directory to the location of the .dump files:

sftp> lcd "Downloads/Subversion Dumps"
sftp> lpwd
Local working directory: /Users/Jon/Downloads/Subversion Dumps

Push the files from the Mac to the Pi:

sftp> put AmberOptix3-r0_1758-20130827205309.dump.zip
Uploading AmberOptix3-r0_1758-20130827205309.dump.zip to /home/jon/AmberOptix3-r0_1758-20130827205309.dump.zip
AmberOptix3-r0_1758-20130827205309.dump.zip 100% 200MB 2.8MB/s 01:12

Then unzip to get hold of the .dump file:

jon@raspberrypi ~ $ unzip AmberOptix3-r0_1758-20130827205309.dump.zip
Archive: AmberOptix3-r0_1758-20130827205309.dump.zip
inflating: AmberOptix3-r0_1758-20130827205309.dump

Create a new repository and import the .dump file:

jon@raspberrypi ~ $ svnadmin load --force-uuid svnrepos/AmberOptix4/ < AmberOptix3-r0_1758-20130827205309.dump

The --force_uuid parameter will tell svnadmin to use the unique identifiers from the imported repository – this feature will allow users to relocate existing working copies.

Example output from the above:

<<< Started new transaction, based on original revision 1
* adding path : tags ... done.
------- Committed revision 1 >>>
<<< Started new transaction, based on original revision 2
* adding path : trunk ... done.
------- Committed revision 2 >>>
<<< Started new transaction, based on original revision 3
* adding path : trunk/ThirdParty ... done.

Finally we can update the location of the repositories of each working copy. With TortoiseSVN the following steps are required:

  1. Right-click on the top-level folder of the working copy.
  2. Select TortoiseSVN | Relocate, enter the new location:

006
Hit ok and wait the the confirmation:

007

In the final part of this write up I’ll document how I used a Python script and the Dropbox API to make a nightly backup of all my repositories.

Subversion Server on Raspberry Pi, part 1

Introduction

These are my notes on how I configured a Raspberry Pi to run Subversion via an Apache server with secure Internet access and automated daily backups to Dropbox. The image below shows the configuration I was aiming for:

SVN RPi Layout

Until last week I’d been managing a couple of Subversion repositories on an old and noisy PC running the excellent CollabNet SubversionEdge. I’d also been looking for a project for my Raspberry Pi to help me learn a little Linux and some Python scripting. Finally I wanted a new system that could be put together by other people I work with, primarily the imaging specialists at Amber Optix whose Subversion repository I manage.

Environment

* MacBook Pro: although I’m primarily a Windows developer I work on a MBP retina with a solid state drive because, well, it’s just awesome. I run virtual machines, courtesy of VMWare Fusion, manage my notes in Evernote, my files with Dropbox, and OS X for web, email, IOS development and everything else. TortoiseSVN provides the Subversion interface via Windows Explorer.

* Raspberry Pi, Model-B, with 4Gb SD card, connected via Ethernet to my router. (My router is an Apple Time Capsule in Bridge Mode, connected to a Sky ADSL modem/router).

Installing the Operating System

(Another neat article here on how to do this).

Download the latest version of Raspbian “Wheezy” from  http://www.raspberrypi.org/downloads (version 2013-07-26 at the time of writing). Unzip the downloaded file to get the operating system image. E.g.

  • 2013-07-26-wheezy-raspbian.img

Before the OS image can be copied onto the SD card we must find the device name. One technique is to list the free disk space of all devices on the MBP with the SD card removed, then repeat the listing with the SD card inserted:

With the SD card not inserted:

$ df -lH
Filesystem                          Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk0s2                        250G   205G    44G    83% 50217398 10852042   82%   /
localhost:/Ev5H8qvZlfqLy7rY7p0zks   250G   250G     0B   100%        0        0  100%   /Volumes/MobileBackups

With the SD card inserted:

$ df -lH
Filesystem                          Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk0s2                        250G   205G    44G    83% 50217398 10852042   82%   /
localhost:/Ev5H8qvZlfqLy7rY7p0zks   250G   250G     0B   100%        0        0  100%   /Volumes/MobileBackups
/dev/disk2s1 59M 19M 39M 34% 512 0 100% /Volumes/boot

This identifies /dev/disk2s1 as the SD card device, where 2 is the disk number.

We’re going to use the dd command to copy the image to the SD card, but we must first  unmount the Pi it so that we have full access to it:

$ diskutil unmount /dev/disk2s1

Then we copy the OS image to the device using the dd command:

$ sudo dd bs=1m if=./Downloads/2013-07-26-wheezy-raspbian.img of=/dev/rdisk2

Note that the destination is formed of /dev/rdisk{x} where x is is the disk number seen earlier with the df -lH command.

Once the block copy is complete we should results similar to those shown below:

1850+0 records in
1850+0 records out
1939865600 bytes transferred in 268.153202 secs (7234169 bytes/sec)

Insert the card into the Raspberry Pi – don’t power up yet.


Connecting to the Pi without a monitor

The Pi comes with a HDMI slot but in my office I don’t have another monitor and I wanted to be able to do everything with just the MBP and terminal sessions.

This site has useful info on configuring the Pi for a variety of routers. I have a Sky router so I logged into the administration pages and looked for the list of connected devices, first with the Pi disconnected and then with it connected:

001

I then used another administration page to reserve the address 192.168.0.14 for the Pi:

002

Once the Pi’s IP is fixed on the router we can continue. Run a terminal session and use SSH (Secure Shell) to gain access to the Pi:

$ ssh pi@192.168.0.14

[Note: if this procedure has been performed before the initial SSH may fail: in this case use the following to remove all keys belonging to the remote host (the Pi) from the locally stored host information:

$ ssh-keygen -R 192.168.0.14

Repeating the SSH command will now work.]

After the first login the output will be similar to:

$ ssh pi@192.168.0.14
The authenticity of host '192.168.0.14 (192.168.0.14)' can't be established.
RSA key fingerprint is 88:8b:92:32:13:d1:0b:3d:10:a6:a0:d7:08:a5:d1:66.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.0.14' (RSA) to the list of known hosts.
pi@192.168.0.14's password: {raspberry, the default password}
Linux raspberrypi 3.6.11+ #474 PREEMPT Thu Jun 13 17:14:42 BST 2013 armv6l
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
NOTICE: the software on this Raspberry Pi has not been fully configured. Please run 'sudo raspi-config'


Prepare the PI

Before starting on the Subversion and Apache installation it’s worth getting the OS updated as much as possible. From within an SSH session run:

$ sudo raspi-config

The following screen will appear:

003

Select option 1 to expand the file system, then reboot the device.

Tip: to reboot the Pi from the SSH session use:

$ sudo shutdown -r now

Next we’ll change the Pi default user’s password:

$ sudo passwd pi

And then we’ll add a new user for general use so that the Pi user is reserved for purely administrative tasks – not necessary but seems to be the normal pattern for Linux systems. For this walkthrough the user jon will be used as the main user on the Pi and in the Subversion user list.

$ sudo adduser jon

We’ll grant this user the ability to use sudo to perform key tasks. If we don’t do this then any attempt to use sudo from the jon account will result in a message like this:

jon is not in the sudoers file. This incident will be reported.

So we use:

$ sudo usermod -aG sudo jon

Then we log out using logout and connect via ssh again but this time for the jon account:

$ ssh jon@192.168.0.14

Finally we will update the software on the PI using these two commands:

$ sudo apt-get update
$ sudo apt-get upgrade

That’s the basic Pi setup complete. In the next part I’ll cover how I configured Subversion.