To be able to use merge tracking for my existing repositories, I first have to upgrade these. From the svn 1.5 release notes:

The Subversion 1.5 server works with 1.4 and older repositories, and it will not upgrade such repositories to 1.5 unless specifically requested to via the svnadmin upgrade command. This means that some of the new 1.5 features will not become available simply by upgrading your server: you will also have to upgrade your repositories. (We decided not to auto-upgrade repositories because we didn’t want 1.5 to silently make repositories unusable by 1.4 — that step should be a conscious decision on the part of the repository admin.)

The only other machine I’m using these repos from is my laptop, which, running Gentoo, is already at Subversion 1.6.2. So, how do I upgrade?

First, I made a backup of all our SVN repos using a script that Halfgaar made to run from cron. After running that script, I initialized the upgrade procedure:

cd /var/svn
svnadmin upgrade blog.bigsmoke.us
svn-populate-node-origins-index blog.bigsmoke.us

The last command is unimportant, but may speed up the next few operations. The release notes again:

After running svnadmin upgrade, you may wish to also run the svn-populate-node-origins-index program on the repository. Subversion 1.5 maintains a node-origins index for each repository, and builds the index lazily as the information is needed. But for old repositories with lots of revisions, it’s better to create the index in one step, using the aforementioned tool, than to have live queries be slower until the index has built itself. See issue #3024 for details.

With my data safe and the repo upgraded, it’s time to actually test the new merge tracking features:

cd ~/blog.bigsmoke.us # Enter working copy (on the same server)
 
# I'm now in the lichtgekruid branch.
svn switch file:///var/svn/blog.bigsmoke.us/trunk
svn add wp-content/plugins/openid
svn ci -m "I was convinced this was a change which didn't belong to my feature branch" wp-content/plugins/openid
 
svn switch file:///var/svn/blog.bigsmoke.us/branches/lichtgekruid
svn merge file:///var/svn/blog.bigsmoke.us/trunk
# The last command was a real kicker, not having to find and specify the correct range of revision numbers.
 
svn ci -m "Merged changes from trunk to 'lichtgekruid' branch."
svn switch file:///var/svn/blog.bigsmoke.us/trunk
svn merge file:///var/svn/blog.bigsmoke.us/branches/lichtgekruid
# (Another spontaneous orgasm.)
 
svn ci -m "Merged 'lichtgekruid' branch back into trunk."

Ok, I know I could have simply changed all these projects to use Git, and I like Git. I love its simplicity and flexibility. But, call me old-fashioned; I have an SVN fetish, and so far I find its merge tracking quite convincing.

Besides all the gotchas and pointers in the 1.5 release notes, there’s a lot more info about merge tracking in the SVN redbook.

#!/bin/bash
 
usage() {
    cat <<"EOF"
$0 [--dry-run] <svn_dir> <replacement_dir>
 
This script replaces the contents of <svn_dir> with the contents of <replacement_dir>,
where <replacement_dir> is not an svn directory.
 
Copyleft 2010, Rowan Rodrik van der Molen <rowan@bigsmoke.us>
EOF
}
 
fatal_error() {
    message=$1
 
    echo -e "\e[1;31m$message\e[0m"
    exit 1
}
 
usage_error() {
    error="Wrong usage."
 
    if [ -n "$1" ]; then
        error=$1
    fi
 
    echo -e "\e[1;31m$error\e[0m"
    exit 1
}
 
run_command() {
    echo -e "\e[1;34m$1\e[0m"
 
    [ $dry_run == 1 ] || eval $1
}
 
dry_run=0
if [ $1 == '--dry-run' ]; then
  dry_run=1
  shift
fi
 
 
[ $# == 2 ] || usage_error "Wrong number of arguments."
 
svn_dir=`echo "$1"|sed -e 's#/$##'`
replacement_dir=`echo "$2"|sed -e 's#/$##'`
begin_path=$PWD
 
#if [ "${svn_dir:0:1}" != "/" ]; then svn_dir="$PWD/$svn_dir"; fi
#if [ "${replacement_dir:0:1}" != "/" ]; then replacement_dir="$PWD/$replacement_dir"; fi
 
[ -d "$svn_dir" ] || usage_error "$svn_dir is not a directory."
[ -d "$replacement_dir" ] || usage_error "$replacement_dir is not a directory."
 
 
# Create all subdirectories in $svn_dir that do not yet exist
cd $replacement_dir
find . -mindepth 1 -type d -print | sed -e 's#^./##' | while read d; do
    cd $begin_path/$svn_dir
    # Doesn't the destination directory already exist?
    if [ ! -d "$d" ]; then
        run_command "svn mkdir '$d'"
    fi
done
 
# Copy all files from $replacement_dir to $svn_dir
cd $begin_path/$replacement_dir
find . -type f -print | sed -e 's#^./##' | while read f; do
    cd $begin_path
    run_command "cp '$replacement_dir/$f' '$svn_dir/$f'" # FIXME: Quoting problem
done
 
# Remove all files that do no longer exist in $replacement dir
cd $begin_path/$svn_dir
find . -type f -print | grep -v '.svn' | while read f; do
    if [ ! -f "$begin_path/$replacement_dir/$f" ]; then
        run_command "svn rm '$f'"
    fi
done
 
# Remove all subdirs that do no longer exist in $replacement dir
cd $begin_path/$svn_dir
find . -mindepth 1 -type d -print | grep -v '.svn' | while read d; do
    if [ ! -d "$begin_path/$replacement_dir/$d" ]; then
        run_command "svn rm '$d'"
    fi
done
 
exit 0

Using the script is simple:

svn-replace-dir simple-tags new-simple-tags|less -R

It replaces all the contents of the first directory (simple-tags in the example) with those of the second directory and it deletes everything that is no longer present in the second dir. In the process, it does all the necessary calls to svn mkdir, svn rm and (in the next version) svn add.

diff tells me that the script has done its work correctly:

diff -x .svn -ruN simple-tags new-simple-tags
# Emptiness is bliss :-) 

This is another one of these occasions when Git would have made life so much easier. Luckily, at least there’s GitHub to host this script as a Gist. Check there if you want to fetch the newest version of this script.

cat upgrade-wordpress.sh 
#!/bin/bash
 
svn_repo_url=file:///var/svn/blog.omega-research.org/vendor/wordpress/
last_version=$1
new_version=$2
 
tmp_dir=`mktemp -d`
cd $tmp_dir
 
for version in $*; do
    echo "Downloading and extracting WordPress version $version..."
 
    if [ -z `svn ls $svn_repo_url|grep $version` ]; then
        branch=`echo $version |sed -e 's/\.[0-9]\+$//'`
        archive_file="wordpress-$version.tar.gz"
        md5_file="wordpress-$version.md5"
 
        wget "http://wordpress.org/$md5_file"
        wget "http://wordpress.org/$archive_file"
 
        if [ `md5sum $archive_file |cut -f 1 -d ' '` != `cat $md5_file` ]; then
            echo "MD5 sum did not match!" >&2
            exit 1
        fi
 
        tar --extract --ungzip --transform "s/^wordpress/$version/" --file $archive_file
        svn_load_dirs.pl $svn_repo_url -t $version current $version
    fi
done
 
cd -
svn merge "$svn_repo_url$last_version" "$svn_repo_url$new_version" .

I’m actually planning to make both scripts a little bit more generic (in the sense that svn_repo_url becomes an external param) and to track future changes to them using GitHub’s Gist. (How ironic is that, tracking an script for Subversion using Git?)

From before I first deployed the Omega Research Wiki, I already used svn to track changes to my MediaWiki installation. However, for upgrading from one upstream release to the next, I used diff and patch. This isn’t the most reliable of methods as is exemplified by doing a diff comparing a fresh MediaWiki download with the actual files in my repo (which are supposed to belong to the same version).

What’s basically a shortcoming of svn is that I can’t just say:

svn merge http://svn.wikimedia.org/svnroot/mediawiki/tags/REL1_15_0/phase3/ \
http://svn.wikimedia.org/svnroot/mediawiki/tags/REL1_15_1/phase3/ .

This would have been incredibly helpful, because now I’m keeping a vendor branch not because of local modifications to upstream, but just to be able to merge cleanly.

In Subversion, maintaining a vendor branch by hand is quite some work, because you need to do a checkout first before you can import each version. (My working copy is normally a checkout of /trunk, not of /vendor/mediawiki.) Luckily, Subversion is distributed with a handy Perl script, svn_load_dirs.pl, which can do most of the heavy lifting.

Still, I didn’t feel like having to do to many manual steps, such as typing in the painfully long URLs for merging, so I decided to wrap the whole process into a nice little Bash script:

cat upgrade-mediawiki.sh
#!/bin/bash
 
svn_repo_url=file:///var/svn/wiki.omega-research.org/vendor/mediawiki/
 
merge=1
while [ $# -gt 2 ] # Process extra options
do
    case "$1" in
        --no-merge)
            merge=0
            ;;
    esac
    shift
done
 
if [ $# -ne 2 ]; then
    echo "Usage: $0 [--no-merge] version version"
    exit 1
fi
 
last_version=$1
new_version=$2
 
tmp_dir=`mktemp -d`
cd $tmp_dir
 
for version in $*; do
    echo "Downloading and extracting MediaWiki version $version..."
 
    if [ -z `svn ls $svn_repo_url|grep $version` ]; then
        branch=`echo $version |sed -e 's/\.[0-9]\+$//'`
        download_file="mediawiki-$version.tar.gz"
        download_url="http://download.wikimedia.org/mediawiki/$branch/$download_file"
 
        wget $download_url || { echo "Downloading $download_file failed">&2; exit 1; }
        tar --extract --ungzip --transform 's/^mediawiki-//' --file $download_file
        svn_load_dirs.pl $svn_repo_url -t $version current $version
    fi
done
 
cd -
if [ $merge == '1' ]; then
    svn merge "$svn_repo_url$last_version" "$svn_repo_url$new_version" .
fi
 
exit 0

The script only downloads and imports each specified version if that version doesn’t already exist in /vendor/mediawiki/. Also, because it has a --no-merge option, you can download all the old versions of MediaWiki that you’ve ever used, so that you can go back and compare old versions of your installation with the factory version. Of course, this is only useful if you were already tracking your installation in svn at the time, and even then not really.

Anyway, the important thing is that you can use the script to download the version you’re running now and the version you want to upgrade too. I wanted to make a big jump, from 1.11.1 to 1.15.0 (I had put of the upgrade for a long time, because I first wanted to learn more about vendor branches):

Of course, as always, you need to check if the patch went well. My own results were a vivid demonstration of the unreliability of my previous method:

svn st|grep '^C'

C      languages/messages/MessagesKrj.php
C      languages/messages/MessagesWar.php
C      languages/messages/MessagesSe.php
C      languages/messages/MessagesFrc.php

Luckily, these were all files I was sure I hadn’t modified:

for i in `svn st|grep '^C'|sed -e 's/^C //'`; do mv $i.merge-right.r32 $i; svn resolved $i; done

Resolved conflicted state of 'languages/messages/MessagesKrj.php'
Resolved conflicted state of 'languages/messages/MessagesWar.php'
Resolved conflicted state of 'languages/messages/MessagesSe.php'
Resolved conflicted state of 'languages/messages/MessagesFrc.php'

Apparently, a previous upgrade hadn’t turned my MediaWiki installation exactly into 1.11.1.

The raw SVN logs are a bit raw, reason enough to try to turn it in something prettier. Luckily, there’s no need to parse the log files, because svn log has a command-line option, --xml. This option causes the log file to be printed in a simple XML format:

<?xml version="1.0"?>
<log>
<logentry revision="345">
<author>halfgaar</author>
<date>2009-05-25T10:50:07.560914Z</date>
<msg>Added very useful note to index about an awkward sentence.
</msg>
</logentry>
<!-- Snipped the rest of the logentry elements -->
</log>

Now we can use any number of ready-to-use XML tools to process this log, but I figured that, maybe, a very simple solution could work: CSS. Cascasing Stylesheets can be used for more than just styling HTML. One of the few differences is that with non-HTML XML, there are no defaults for the CSS properties (and aren’t we always trying to discover and override the various browser-specific CSS defaults anyway?)

First, we want to add a <?xml-stylesheet?> processing instruction to the log file:

svn log --xml example_file.xhtml | sed -e '/<\?xml / a<?xml-stylesheet type="text/css" media="screen" href="/css/svn-log.css"?>'

The XML file now references the CSS file that we’re going to make:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/css" media="screen" href="svn-log.css"?>
<!-- snip -->

svn-log.css:

log {
  background-color: white;
  background-image: url('http://svn.collab.net/repos/svn/trunk/notes/logo/256-colour/subversion_logo-200x173.png');
  background-attachment: fixed;
  background-position: bottom right;
  background-repeat: no-repeat;
  display: block;
  margin: 2em 204px 2em 5ex;
}
 
logentry {
  display: list-item;
  list-style-type: square;
  margin-bottom: 1em;
  font-family: monospace;
  font-size: normal;
  color: #999;
}
 
author {
  display: none;
}
 
date {
  display: block;
  width: 10ex;
  /* If Firefox would support font families even when you force a font,
     I could use 'overflow: hidden' to hide everything except the date part of <date>. */
  overflow: hidden;
  overflow: visible; 
  font-family: monospace;
  font-weight: bold;
  font-size: 110%;
  color: #7488a7;
}
 
msg {
  display: block;
  white-space: pre-wrap;
  color: black;
}

We now have a nicely formatted log file. Other things you could do:

• Add styles for printing (in a separate stylesheet or by using @media blocks).
• Display the revision author instead of hiding it.

Of course, you could do all this and much more with XSLT, but that’s just all too obvious.

If you want to see the stylesheet in action, take a look at Wiebe’s website and look for a View revision log of this page link in the footer of any of his pages.

XML SVN log styled with CSS

We used http://svn.rubyonrails.org/rails for our svn:external deployment of rails 1.2.6. But that suddenly stopped working. The Ruby on Rails weblog doesn’t contain any information on it (that I can find), going to svn.rubyonrails.org with my web browser yielded just the front page of the weblog and Googling didn’t help much. That is, until I accidentally found http://dev.rubyonrails.org/svn/rails.

So, I changed the svn:externals from “rails http://svn.rubyonrails.org/rails/tags/rel_1-2-6/” to “rails http://dev.rubyonrails.org/svn/rails/tags/rel_1-2-6/” and now it works again.

If, in the pre-2.6 days, you wanted to fetch your WordPress updates through SVN, the docs would advice you to do an svn checkout from the official WP SVN repo in your working dir and then do an svn update whenever you want to update WordPress. This works because svn update leaves local modifcations alone. However, this means that you’ll be unable to commit your local changes (configuration, themes, plugins, etc.) if you choose this route.

I used my own subversion repository for my blogs and thus had to upgrade the old fashioned way with each release (although I prefer diff/patch over rm/cp). (I could have used vendor branches, but, clearly, I hadn’t thought about that at the time.) This was pretty much a royal pain in the ass, so I was glad when I could move WordPress into a separate directory with its 2.6 release.

This process consisted of removing everything except wp-content/, wp-config.php, .htaccess. (I also kept robots.txt, favicon.ico and some other personal files.) Then, I added the current WordPress release as an svn:external.

svn propset svn:externals 'wp-factory http://svn.automattic.com/wordpress/tags/2.6.1' .

.htaccess changes

In the WordPress codex, it is then suggested to copy index.php to the root dir and to change it to require wp-factory/wp-blog-header.php instead of ./wp-blog-header.php. I preferred adding some mod_rewrite voodoo of my own to .htaccess, so I did:

<IfModule mod_rewrite.c>
# This way I don't need directory indices
RewriteRule ^$ /wp-factory/index.php [L]
 
# This way WordPress can manage its own block without doing any harm
RewriteRule ^index.php /wp-factory/index.php [L]
 
# Allow easier access to /wp-factory/wp-admin/
RewriteRule ^wp-admin http://%{HTTP_HOST}/wp-factory/wp-admin/ [L,R=301]
</IfModule>

The middle rule performs most of the magic. It redirects all the requests to /index.php to the factory default index.php. This means that I can let WordPress pretend that index.php does live in the root, so I don’t have to modify the rewrite rules that are managed by WordPress itself:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# END WordPress 

wp-config.php changes

In Giving WordPress Its Own Directory in the WordPress Codex, it is suggested to change the “siteurl” and “home” options through the administration panel. In my case they would have to be changed to “http://blog.bigsmoke.us/wp-factory/” and http://blog.bigsmoke.us“. I couldn’t do this because I override these with WP_SITEURL and WP_HOME in my wp-config.php. This is because I configured WordPress to support a development environment separate from the live production environment.

Ignoring the customizations for my development environment, these are the relevant settings in wp-config.php:

define('WP_HOME', 'http://blog.bigsmoke.us');
define('WP_SITEURL', WP_HOME . '/wp-factory');
 
define('WP_CONTENT_DIR', dirname(__FILE__) . '/wp-content');
define('WP_CONTENT_URL', WP_HOME . '/wp-content');

BTW: I really like it how WordPress disables the form controls for siteurl and home when you override these settings in wp-config.php. Kudos for that, devs!

Next time: git

In the end, this is all quite a bit of pain to compensate for what is essentially a version management problem. That’s why, on my newer projects, I’m now using git which makes forking and tracking an upstream repo absolutely trivial.

A few references

• Editing wp-config.php in the WordPress Codex
• Giving WordPress its Own Directory While Leaving Your Blog in the Root Directory in the WordPress Codex
• Vendor branches in the Subversion Redbook

When you install a big-ass web application such as WordPress or MediaWiki, you usually end with a bunch of configuration files and customizations (skins/themes, extension/plugins, uploads, etc.). This makes upgrading the files that come with the application a bit tricky. There’s a simple solution, however, which work regardless of whether you use a revision control system or not.

First of all, you do, of course, always need a revision control system. I personally recommend Git or Subversion, which are both excellent tools. But, that’s not what this post is about. I’m going to use two simple tools which are uniformly available on all (Unixy) platforms: diff and patch.

The procedure is simple:

1. Download the version of the application which you’re currently running. For our example, we pretend that this version is extracted into the directory webapp-1.4.3.

2. Then, download the version to which you’d like to upgrade. (We’re assuming that this version is extracted into the webapp-1.6.2 directory.)

3. Compare the two versions to create a patch file:

   $ diff --unified --recursive --new-file webapp-1.4.3 webapp-1.6.2 > webapp-upgrade.diff
   

4. Apply the patch to the installation of said web app:

   $ cd webapp-installed
   $ patch --strip=1 --remove-empty-files < ../webapp-upgrade.diff || echo "Some failures!"
   

Check if everything was patched perfectly

Now, if the patch command returned a non-zero status (printing Some failures! in the above example), it's time to check which chunks of which files failed. Get a summary by searching all files with an .rej or a .orig suffix:

$ find . -name "*.rej"

After manually applying any failed hunks, what's left is to compare your directory containing the patched application to the directory with the contents of the new application archive which you've used to create the patch:

$ cd ..
$ diff --unified --recursive --new-file webapp-1.6.2 webapp-installed

Version management

Your upgrade is done. Now, if your using a revision control system, you just need to check in new files and check out deleted files. In Subversion, I do this quickly using the following command sequence:

$ svn status|sed -e '/^\?/!d; s/^\?//'|xargs svn add
$ svn status|sed -e '/^\!/!d; s/^\!//'|xargs svn del

If you'd been using Git, you could do this all a little bit more sophisticatedly, but my Git skills are not advanced enough to go around giving others advice. Also, it's nice to learn a generic method before learning more specific tools.

Importing the tags from Ultimate Tag Warrior worked fine. Before upgrading and importing, I quickly patched my local version of WP with a little help from Subversion:

$ svn diff http://svn.automattic.com/wordpress/tags/2.1/ http://svn.automattic.com/wordpress/tags/2.3.1/ > wp.diff
$ cd blog.bigsmoke.us
$ patch --remove-empty-files -p0 < ../wp.diff
$ svn revert wp-config.php
$ svn add `svn status|grep '^?'|sed -e 's/\?//'`
$ svn rm `svn status|grep '^!'|sed -e 's/!//'`

Then, after a few changes to my template files to play nice with WP’s new built-in tagging system, everything was running again.

Subversion’s merge tracking is non-existent. This means that, when I climb from branch to trunk and back again a lot, I have to manually keep track of all the changes in trunk/ that I merged into the branch. Every one such change, once merged, loses much of its meaningful history unless I painstakingly merge all the commit messages of the patch into the message of the commit that I do after the merge.

Today, after having maintained a branch for months to keep it somewhat in sync with an every-changing trunk, I’m at the point of having to merge the branch back into trunk. This is rather nightmarish because there are bound to be the many merge conflicts that I already suffered whenever merging changes from the trunk into the branch and then multiplied some.

To avoid torture, I decided I’d rather just replace the trunk with my feature branch. This is especially attractive because I then retain the history of the branch which is a little more useful to me than the history of the trunk.

I googled around a bit and could find one thread discussing a similar problem. The solution proposed there seemed to involve a few too many steps for my taste, so I did the following:

# From the working copy of my branch:
$ svn del file:///repos/trunk -m "Temporarily deleted trunk."
$ svn mv file:///repos/branches/my_branch file:///repos/trunk -m "Moved /branches/my_branch to /trunk"
$ svn switch file:///repos/trunk 

That worked perfectly fine. (Except that I still want automatic merge tracking, dammit!)