Deploying and maintaining a website using git

I’ve kind of used git to deploy/ manage php/ static websites for a while now, but in a very luddite way – basically sshing into the webserver, cloning a repo into my site root directory, then hiding the .git folder in the Apache config.

After recently starting a new job and inheriting some web sites and web apps and starting to take the whole “devops” thing more seriously, I was pleased to see a better technique in place than the one i’ve been using.

The basic premise is this – a repo is set up on the web server as a remote outside of the web root, but configured so that the checkout working directory is another directory. The advantage of this is that there is no need to ssh into the webserver to update the code, as a post-update hook can be used to checkout the updated files.

In addition to this, we usually use tools such as gulp to process scss and minify and js, so the post-update hook can also be used to process these on the server after checkout.

Going into more detail, here is the config of the remote git repo on the webserver, this could be somewhere like /var/sites/mysite_git

repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
worktree = /var/sites/mysite
denycurrentbranch = ignore

and in hooks/post-update on the remote, something like:-

git checkout -f
cd /var/sites/mysite
gulp build

Using Vagrant for local LAMP development

I’ve always used a local version of Apache for php dev, either the version provided with OSX, or using something like XAMPP or MAMP. On a recent freelance contract I was introduced to using Vagrant to spin up a tailored virtual machine with specific versions of php, mysql and any other relevant dependencies.

Vagrant will download a base virtual machine, and then run a provisioning script to install dependencies. Vagrant also sets up file sharing from your host machine, and port forwarding so you edit locally in your normal editor, and view locally via a web browser as if you were running a local apache instance.

The advantage of this is that once you are up and running, the Vagrant configuration can be stored in the GIT repo, so that other developers can quickly start developing using exactly the same dev environment.

A typical Vagrantfile looks like this:-

Vagrant.configure(2) do |config| = "hashicorp/precise32"
# Mentioning the SSH Username/Password:
config.ssh.username = "vagrant"
config.ssh.password = "vagrant"
# Begin Configuring
config.vm.define "lamp" do|lamp|
lamp.vm.hostname = "lamp" # Setting up hostname "private_network", ip: "" # Setting up machine's IP Address
lamp.vm.synced_folder "siteroot", "/var/www", owner: "root", group: "root"
#lamp.vm.provision :shell, path: "" # Provisioning with
# End Configuring "forwarded_port", guest: 80, host: 8080, auto_correct: true

Running “vagrant up” in a directory with this file would spin up an ubuntu VM, serving files from the “siteroot” directory. It would also attempt to run on the VM, which can be used to install php/mysql etc.

The getting started guide on the Vagrant site is the best way to get up and running if your are interested.

Olivewood developments – Chil-LED Cold Storage Area Light

Chil-LED cold storage area light

I co-founded Olivewood Technology back in May 2007, when I decided to make the jump to freelancing. I had been developing an EDI web app for a small distribution company, and my co-founder and I decided that we should from a company to market and develop this app.

Seven years on, the app, known as “Supply Portal” is alive and well and is now a mission-critical app for the original company, and installed for another distributor, but it still needs some work to make it suitable as something “off the shelf” that people would buy without requiring modifications to fit their workflow.

In the meantime my co-founder became an expert in LED lighting, and identified a gap in the market – an LED light specifically designed for Cold Storage Areas. Vast amounts of energy are consumed extracting heat from cold storage warehouses, only for the lighting to pump heat back in. Switching to LED lights is an obvious move – apart from the reduced energy needed to run them, they also run cooler than legacy lighting systems.However, high power LED’s still produce significant amounts of heat, and this usually ends up as warm air at the top of the warehouse, resulting in a reduced area of useable storage space higher up in the warehouse and higher energy bills needed to remove that heat from the warehouse..

Chil-LED was designed to extract the heat from the LED chip and transfer it outside of the Cold Store, using a patented system of thermal heat pipes fed through a conduit to heatsinks in the void above the warehouse ceiling.

Chil-LED cold storage area light product development

So this is the main focus for Olivewood moving forward, we still build software, but we are now a technology company, we have developed the Chil-LED light and how now gone to market – exciting times!

For more information on chil-LED see the Chil-LED website

For more information on Olivewood Technology, see the Olivewood Website

Upgrading to OSX Mavericks

I’m a late adopter, but I finally took the plunge and upgraded to OSX Mavericks tonight.

A few gotchas:-

  • I had to reconfigure Apache for PHP and virtualhost support
  • I had to re-add a symlink for the MySQL installation:-

sudo ln -s /usr/local/mysql-5.6.10-osx10.7-x86_64/ /usr/local/mysql

  • I also had to reconfigure ssl
  • I had to reinstall vagrant (because of lost path)
Tags: Uncategorised

The Camputer Part 3 – API hacking

In part 2 I got RasbBMC installed, and worked out how to get vnc server running. The next goal was to get to know how the music player works, how to create playlists and most importantly hack around with the API so that I can work out how to control it from a python script.

raspbmc displaying on small lcd monitor

My original intention was to run the pi entirely headless, but then I remembered I had already bought a small LCD monitor for a reversing camera, that I never got round to fitting. Hooked up to a 12 volt power supply and to the PI using the RCA socket, I now have a tiny monitor, which is handy for keeping an eye on what is going on with the player.

Most of the text is illegible at that size/ resolution, so I upped all the font sizes for the default skin:-

cd .xbmc-current/xbmc-bin/share/xbmc/addons/skin.confluence/720p
sudo nano Font.xml

Now all the line-spacing is wrong, so is more legible in some places, but less so in others. I don’t want to get side-tracked right now into learning how to skin XBMC. Maybe at a later date, it might be a nice project to create (or find) something specifically for very small screens.

IInitially I couldn’t hear any Audio, so I had to change the audio settings in XBMC to use analog audio out rather than HDMI.

I spent ages reading up on the api, the documentation is pretty confusing and lacking in examples, but eventually I found some simple examples on a forum showing how to interact with the JSON-RPC API.

I was expecting to find some examples of using the python API “directly”, but couldn’t really find anything (or at least anything complete enough to understand real-world use). I eventually found this python xbmc json client.

With that library downloaded to the pi, I was able to speak to xbmc from the python prompt:-

>>> from xbmcjson import XBMC
>>> xbmc = XBMC("http://localhost:8080/jsonrpc")
>>> print xbmc.JSONRPC.Ping()
{u'jsonrpc': u'2.0', u'id': 0, u'result': u'pong'}

To open a saved playlist:-


I was then able to play/pause XBMC and move to the next track from the python prompt with:-


Before I worked out how to open a saved playlist via the API, I noticed that the above play/pause command doesn’t work if the player hasn’t been opened yet i.e. when the Pi has been rebooted. I found a way of opening a playlist and pausing at XBMC startup, by creating a file called in /home/pi/.xbmc/userdata with the following in it:-

import xbmc
xbmc.executebuiltin( "XBMC.Action(Pause)" )

This is run by XBMC at startup, and doing so ensures that the player has a playlist loaded and ready to play at startup. Currently not sure where this script imports xbmc from or how I could have this available to my own application scripts and managed by XBMC, but there must be a way. Either way I seem to be able to do most of what I need with the JSON-RPC API currently, with the exception of shuffle. The following command toggles shuffle on and off:-

xbmc.Player.SetShuffle({"playerid":0, "shuffle":"toggle"})

The JSON response sais “OK”, but annoyingly it has no effect on the XBMC player. When operating XBMC manually you can open a saved playlist, and at any point click the shuffle button, to toggle shuffle on and off, and the next track will either be in order, or “random” from a shuffled list. I was hoping that this toggle on the API would have the same effect, but apparently not. I think I may have to consult the XBMC forum, once i’ve put togther a precise list of commands that i’ve tried, and checked exactly which version of XBMC and API I am using (you know how forums can be if you go asking questions without providing enough background info!).

If it turns out that this is a bug and there is no solution for shuffling the current playlist with the API, I can envisage a workaround involving dynamically creating a shuffled version of each of the saved playlists. As I intend to create some kind of playlist management app for this (outside of XBMC), it would be a simple step on from that.

So, besides working out a solution for shuffle, the next step is to connect the Piface digital I/O expansion board, and work out how to trigger API stuff in response to button presses.

The Camputer Part 2 – raspBMC installation

(See Part 1 for details on what the project is about and the kit bought for it so far).

I decided to go with RaspBMC as the operating system, on the basis that I wanted something Raspian/debian based, and this distro is exactly that, but with the benefit of being preconfigured as an XBMC media player.

I ordered a 16gb SD card (I intend to keep my media on an external USB drive) and used the instructions to create a bootable raspBMC installation, using my macbook air, which has an SD card slot.

Then I put the SD card in the Raspberry Pi board that I got from UK Farnell, hooked up to a keyboard, mouse, ethernet cable (connected to our home router), power supply, audio cable to my hifi amp and our TV via HDMI cable.

The bootable SD then takes you through the rest of the installation – it took about 25 minutes.

The Pi then booted into XBMC, which i’ve never actually seen before, and it looks great. I enabled AirPlay in the settings and found that I could stream audio and (non-apple DRMed) movies from itunes and from my wife’s iPhone. I’ll probably set one up specifically as a home media system in the future, and probably others just for audio streaming to amps/speakers around the house.

I then went of at a tangent, trying and failing to get a vnc server installed and working, so that I can control it from another device, as it will likely be run headless in the camper van. After a bit of digging though, I realised it already has a VNC server installed which can be enabled in the system settings – the raspBMC settings rather than XBMC settings. You can also start it from the command line with:-

service vnc start

I couldn’t get it to connect using OSX screen sharing, so I installed Real VNC viewer. It seems very sluggish, but works.

So the next stage is to hook it up to a USB drive with some music on it, create some playlists and investigate how I can interact with the player programatically via the API.

Follow-up posts:

Part 3: The Camputer Part 3 – API hacking

Raspberry Pi project – the Camputer (Part 1)

raspberry Pi kit for my camper van in-vehicle computer

I’m building an in-vehicle computer for my camper van. I wasn’t sure whether to blog about this here or on my Camper van blog, but as the project will initially be about software, i’ll start here, then move over to the campervan blog when I get to the point of installing it.

Why does a campervan need an on-board computer?

Well obviously it doesn’t need a computer, but here are a few things i’d like to do with it:-

Media system

I don’t have many CD’s these days, so I currently use my phone or an ipod to provide music to the stereo system, but it is a bit of a faff, fiddling around with cables and chargers and stuff. I’m also pretty disorganised with my music collection and have several small collections spread out over different iTunes libraries etc. It’s also a pain (not to mention dangerous) to operate a touchscreen while driving. The main requirements for my ideal music system therefore are:-

  1. All my music in one library
  2. Physical buttons to play/stop and skip tracks (a bit like an iPod shuffle)

I have in mind a set of switches and buttons on the dash that I can operate by feel. Aswell as the basic functions listed above, there might be toggle switches for different playlists or genres.

The media system could have a small monitor or LCD display, but should be able to be operated using only the buttons. Other operations (such as creating playlists etc.) I would envisage a web interface, VNC or maybe plugging in a monitor and keyboard.

I haven’t decided what media player software i’ll use, but hopefully there is something with a decent api that I can hook my custom controls to.

I’m thinking mainly music, but with a ceiling mounted pull-down monitor it could also drive a (motor)home cinema system in the future,

Engine diagnostics

I have a bluetooth OBII adapter that I use with the Torque Android app. From a quick google around, people are using them with a Pi, so i’d like to set up some logging.

Security/ Tracking

It would be great to have the Pi send location updates to my server, to tie in with my travel blogging. It could also potentially be useful if the vehicle gets stolen, or I lose it in an unfamiliar town!

Remote switching of the heater

My camper has an on-board propex-heater, so the Pi could potentially be used as a thermostat or to switch the heater on from a web/ phone app.


So far I have:-

So the first phase will be to decide on an OS and media player and work out how to control it using physical switches – watch this space!

Follow-up posts:

Part 2: RaspBMC installation

Part 3: The Camputer Part 3 – API hacking

Search and replace on a large file on OSX using Sed

My code editor of choice, Sublime text 2, crashes when you try to do a search and replace on a large file. This makes it difficult to do a search and replace on a large sql file, as I sometimes need to do when moving a wordpress site from one domain to another

The command-line tool sed can be used instead like this:-

sed -i '.backup' 's/olddomain/newdomain/g' wordpress-backup.sql

This makes a backup of the original file, then updates the original

Rasberry Pi as low power office server

raspberry pi running as low power office server

Part of a system that I look after for a small distribution company includes a linux server based in their offices. The server doesn’t do much – it runs a few cron jobs and a mySQL database, acting as a monitoring system on the internal network and as a middle-man receiving data extracted from Sage Line 50 database on a windows server and posting off data updates to a web-based server. It is however a vital part of the system, and needs to be running 24/7. For years this has been handled by an assortment of dusty old desktop PC’s, the last of which was starting to show signs of needing to be retired. The next old PC in line turned out to have a faulty network port, so I though it might be time to buy some kit specifically for this job. It occured to me that due to the low power requirements, a Rasberry Pi might be ideal.

Being fairly new to the Rasberry Pi, I ordered a starter kit including a Model B Pi, a 4GB SD card with “noobs” installed on it, a case and a power supply. Connected up to the TV at home via an HDMI cable, it booted into noobs, from where I installed raspbian linux. Once raspbian was installed, I found that I could install everything I needed (mysql, PHP5, apache) using apt much as I would on a regular debian-based linux box. One thing that I found I couldn’t do (from googling rather than actually trying) was install dropbox, however this wasn’t essential.

After installing it in the office and monitoring it for a few days, it seems to be coping extremely well. There is one particular php/mysql task that sends the cpu temporarily very high, but I think this can be tamed with a bit of code optimisation. I also wish I had started with a bigger SD card, as there is only just enough “disk” space when everything is installed.

One advantage of this low-power system is that I have it running from a USB cable plugged into another server rather then requiring another port on the office UPS. I’ve considered having it run from a rechargeable USB power pack, kind of like a mini dedicated UPS. I’ll be interested to see how it stands up to this kind of use. Due to the low cost of this system, the plan is to have a spare Rasberry Pi or two and some spare cloned SD cards on standby.

One other thing to note is that to plug the Pi into a monitor you may need an HDMI to DVI cable, unless you have a monitor with an HDMI port, or some HDMI or RCA to VGA solution (that will be another blog post). I only needed to plug it into a monitor to congure the static IP address – after that I ssh’ed into it from another machine.

So why didn’t I just use a virtual machine? A couple of reasons: firstly, one of the tasks for this machine is to monitor the windows server from which the sage line 50 data is sent. Secondly, the aforementioned windows server is the only machine on the network running 24/7 and already struggles a fair amount, and I didn’t want to hog any more RAM or cpu with a virtual machine.

Update 2014-01-22: The Pi has been up and running for nearly a week with no issues, but i’ve been reading about SD card wear that can occur if the disc is being written to frequently, so I plan to move the MySql and log file storage to an external drive, otherwise I suspect the card might “wear out” fairly quickly.

Managing Google App Engine SDK versions on OSX with virtualenv

When I first started developing Django apps on Google App Engine (GAE), I downloaded the google app engine launcher. Installing this will install the latest version of the SDK, and will (if you let it) update your path to point to it. This is fine if you are always using the latest version and don’t need to use an older version for some reason. If that is the case you will need to uninstall GAE launcher, find an old installer for the version you want and reinstall.

A more manageable way is to not use the GAE launcher, but to download the SDK manually, then set up a virtualenv to use a particular SDK. I also use virtualenvwrapper to help me switch easily between environments.

This is the pertinent bit – in the folder where your virtual envs are stored, add the path to the app engine sdk in bin/postactivate. Postactivate gets run when you activate the virualenv through virtualenvwrapper. E.g for something using version 1.8.8 you might have something like:-

# This hook is run after this virtualenv is activated.
export PATH=/Users/rickhurst/gae-sdk/1.8.8/google_appengine:$PATH