Dropbox + Mercurial

Jan 9, 2012

This is a summary of my personal version control system that I told a friend I’d write up a while ago. It’s basically a mutant combination of Mercurial and Dropbox. Each of these systems is somewhat specialised to be very good at approximately half of what I really want out of a versioning system, so the combination works out fairly well. There are some issues though, which I cover towards the end of the post.

 

What I like to have stored in version control pretty much comes down to this: it should be possible to build or run the game straight after checking out. So I like to version all of my source code and projects, along with all the required game assets, the latest build, and – if an artist is working on the project – any of the files they need to have on hand as well.

 

Mercurial is very close to my ideal version control. It’s distributed, which means I can check in changes with or without connection to a server, and easily branch and merge when I want to try out new things. Mercurial deals incredibly well with text files, and is able to deal with merges even when lines get moved or altered. This makes it perfect for versioning source code. Unfortunately it deals pretty badly with binary files like textures, compiled libraries, or executables.

 

Dropbox on the other hand, is ideally suited for binary files. It doesn’t track chunks of changes like Mercurial does, but instead just stores the last few versions of a specific file in full. It will keep track of any changed versions of a file for up to 30 days, although there’s an add-on you can buy which will keep track of them forever. For things like textures, libraries, executables or any other binary file, this is pretty much perfect. So to create the kind of versioning system I want, I create a Mercurial repository inside a Dropbox folder, giving me the best of both worlds. Dropbox will automatically synchronise both the binaries and the repository itself.

 

I imagine this would work perfectly for a single-user project, or even a single-programmer + artist project. I always prefer to have a continuous integration system running as well though. So that means keeping a separate working repository of unsafe/uncompiling code – where I do most of my work – and a repository of checked in code – which is the code used to create the latest build. This is easy enough: use the Dropbox folder as a remote repository and sync it to a different folder on my computer hard drive. Now I can check in half-working code as much as I want, and when I’m ready to trigger an automated build I push my changes into the Dropbox folder. The Dropbox folder on my working machine gets synchronised with the Dropbox folder on the build machine, the build machine notices changes in the repo and triggers a build. I keep binary files in certain subfolders of the Dropbox folder and set Mercurial up to ignore them. Then in my working folder I create symlinks to the Dropbox subfolders, letting me access them in buildscripts etc. as though they are actually subfolders in my working copy.

My working copy ends up looking like this:Working

While the Dropbox folder looks like this:Dropbox

 

For simplicity, I wrote build scripts (on both Windows and OSX) to create the symlinks as needed and to build, run tests and copy the latest executable into the bin folder. You can find some example scripts here.

 

Overall, it’s a decent system. The only issue I have with it is that sometimes I find builds getting triggered prematurely. In order to trigger a build, the following chain of syncs need to happen:

Working folder —(hg)—> My Dropbox folder —(dropbox)—> Build Dropbox Folder

This means the Mercurial repo is actually being synced by Dropbox, which can result in the repo thinking it’s changed and triggering a build while the Dropbox sync is only half completed. Mercurial does not deal well with half-modified repositories. It’s generally not a massive issue, and usually just results in a single build failing. I minimise it by checking the repo for modifications less often, but if it’s too problematic it would be possible to store the hg repo on a server like BitBucket and update the Dropbox repo every build.

 

Hope this post is helpful for someone.

Posted by | Categories: Blog |

Share with others

2 Responses so far | Have Your Say!

  1. Tatham
    January 16th, 2012 at 6:04 AM #

    Oops, got the working directory/dropbox folder images reversed. Fixed.

  2. Jeanne
    April 26th, 2012 at 5:54 AM #

    Nice article — thanks for writing it up. This is exactly what I was hoping to do for our small project, and was looking at various paid repositories. But they charge way too much per user or per project, as you noted. If this works could be a good reason to upgrade the DropBox account…

Leave a Feedback

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current month ye@r day *