Page 40 of 52

Tracking MediaWiki in a Subversion vendor branch

July 20, 2009 / Rowan Rodrik

Vendor branches are the proper™ way of merging upstream changes in your web application installations. In Subversion, managing vendor branches isn’t so easy as it is in Git. Still, vendor branches make it much easier to track upstream.

From before I first deployed the Omega Research Wiki, I already used svn to track changes to my MediaWiki installation. However, for upgrading from one upstream release to the next, I used diff and patch. This isn’t the most reliable of methods as is exemplified by doing a diff comparing a fresh MediaWiki download with the actual files in my repo (which are supposed to belong to the same version).

What’s basically a shortcoming of svn is that I can’t just say:

svn merge http://svn.wikimedia.org/svnroot/mediawiki/tags/REL1_15_0/phase3/ \
http://svn.wikimedia.org/svnroot/mediawiki/tags/REL1_15_1/phase3/ .

This would have been incredibly helpful, because now I’m keeping a vendor branch not because of local modifications to upstream, but just to be able to merge cleanly.

In Subversion, maintaining a vendor branch by hand is quite some work, because you need to do a checkout first before you can import each version. (My working copy is normally a checkout of /trunk, not of /vendor/mediawiki.) Luckily, Subversion is distributed with a handy Perl script, svn_load_dirs.pl, which can do most of the heavy lifting.

Still, I didn’t feel like having to do to many manual steps, such as typing in the painfully long URLs for merging, so I decided to wrap the whole process into a nice little Bash script:

cat upgrade-mediawiki.sh
#!/bin/bash
 
svn_repo_url=file:///var/svn/wiki.omega-research.org/vendor/mediawiki/
 
merge=1 [ $# -gt 2 ] # Process extra options
    "$1"
        --no-merge)
            merge=0
            ;;
   
   
 [ $# -ne 2 ]; then
    "Usage: $0 [--no-merge] version version"
    1
 
last_version=$1
new_version=$2
 
tmp_dir=`mktemp -d` $tmp_dir
 version $*;
    "Downloading and extracting MediaWiki version $version..."
 
    [ -z `svn ls $svn_repo_url|grep $version` ];
        branch= $version |sed -e 's/\.[0-9]\+$//'`
        download_file="mediawiki-$version.tar.gz"
        download_url="http://download.wikimedia.org/mediawiki/$branch/$download_file"
 
        wget $download_url || { "Downloading $download_file failed">&2; 1; }
        tar --extract --ungzip --transform 's/^mediawiki-//' --file $download_file
        svn_load_dirs.pl $svn_repo_url -t $version current $version
   
 - [ $merge == '1' ];
    svn merge "$svn_repo_url$last_version" "$svn_repo_url$new_version" .
 0

The script only downloads and imports each specified version if that version doesn’t already exist in /vendor/mediawiki/. Also, because it has a --no-merge option, you can download all the old versions of MediaWiki that you’ve ever used, so that you can go back and compare old versions of your installation with the factory version. Of course, this is only useful if you were already tracking your installation in svn at the time, and even then not really. 😉

Anyway, the important thing is that you can use the script to download the version you’re running now and the version you want to upgrade too. I wanted to make a big jump, from 1.11.1 to 1.15.0 (I had put of the upgrade for a long time, because I first wanted to learn more about vendor branches):

Of course, as always, you need to check if the patch went well. My own results were a vivid demonstration of the unreliability of my previous method:

svn st|grep '^C'

C      languages/messages/MessagesKrj.php
C      languages/messages/MessagesWar.php
C      languages/messages/MessagesSe.php
C      languages/messages/MessagesFrc.php

Luckily, these were all files I was sure I hadn’t modified:

i `svn st|grep '^C'|sed -e 's/^C //'`; mv $i.merge-right.r32 $i; svn resolved $i;

Resolved conflicted state of 'languages/messages/MessagesKrj.php'
Resolved conflicted state of 'languages/messages/MessagesWar.php'
Resolved conflicted state of 'languages/messages/MessagesSe.php'
Resolved conflicted state of 'languages/messages/MessagesFrc.php'

Apparently, a previous upgrade hadn’t turned my MediaWiki installation exactly into 1.11.1.

TheSixtyOne, a better social music site

July 20, 2009 / Rowan Rodrik

If you liked Last.fm, you’ll love TheSixtyOne. Discovering music is turned into a reputation game. Much like Last.fm, you can explore similar artists, genres, tags, moods and users. But, by doing so and hearting songs, you can earn reputation points and extra hearts. As your points increase, you level up, which gives you more authority. There’s an endless amount of quests to earn points faster and to make you discover all the facets of the application.

Just try it out. It’s fun and—unlike Last.fm these days–it’s free. You can choose when and how to help these starving artists! 😀

Setting up Rsync daemon

July 15, 2009 / halfgaar

What can you do when you want to backup your entire machine to a remote location but only have non-root shell access? You use rsyncd.

On the server you need to configure a module in /etc/rsyncd.conf:

[module_name]
uid=0
gid=0
path=/path
transfer logging=no
read only=no

Then you need to rsync using a destination like user@host::module_name/

To make it work, you first need to set up a tunnel using SSH, because the rsync port (873) is not open in the firewall, most likely. So, you do this:

ssh -N -L 873:localhost:873 user@host

I wrote a script to automate this all. Perhaps I will follow-up on this post sometime and post it.

Blogging: quantity over quality

July 10, 2009 / Rowan Rodrik

They say that blogging is a long way down from essaying. Articles were already better than essays, and books even more so. So they say. A lot is said these days. I say: the more, the merrier. I say: let us say! Let us say whatever we want. And let people read whatever they want, now that they can.

I try to blog as much as I can, not because I think that each post is a piece of poetry that needs to be enjoyed by as many as possible, but because each post represents a train of thought that I need to get rid of. The sooner, the better. Noise is not a problem. The Internet is filled with noise, but nevertheless we only read what we want to read and we still find what we need. We filter. If you don’t, good luck surviving the digital age.

There is a convergence movement towards media integration. Eventually, this will lead to the collapse of all these media, meaning that the need for distinct, specific media will be less and less. This is good. We need to be truly press-less before the original promise of the press can be fulfilled: information liberation.

What I mean by this is that, when we publish, we don’t need to make a clear distinction between the weighty and the weightless. After all, God Google defines the weight of our publications not by means of what we say about it, but by means of incoming link frequency and the relative weight of these links.

When I say something worthwhile, I first have to convince others that I did, but before I can do that, I have to convince them to come look. Then, after visiting me and having been convinced by my great content, they might decide to link to me. Only as soon as that happens, do I gain weight. (According to Technorati, the weight of this blog is pathetic.)

Not every post I write resembles even an essay. Most postst don’t even come close. That’s because blog posts don’t need a point. I’ve made this point before, but I can’t repeat it often enough, because every time that I’m thinking “Shit, what did I do then and what was I thinking?” I’m reminded of it. I’ll stop reiterating this no sooner than I stop forgetting to write these things down.

When a blog post does attempt to make a point, there are even stronger arguments for quantity over quality, because publishing—really publishing—something makes it possible for others to respond, be this in comments or in their own blog. There’s something concrete for them to cite and to reference. This process gives me valuable feedback and I can simply republish my thoughts (which have matured not only through time, but also through argument) in a new post. Voila! And maybe the old post is so shitty that nobody is interested… Who cares?! Obviously, they don’t! So go on and do something else with the time that you’ve freed by not having to polish a turd.

Remember that blog posts are not naturally organized into some sort of reading hierarchy. There’s no reason that the same subject can’t be treated twice. Just hit that “Publish” button, get it out of your head and get on with your life!

Editing Wikipedia/MediaWiki articles with VIM

July 10, 2009 / Rowan Rodrik

Wikipedia has a number of tips for people who like to edit Wikipedia articles with VIM.

Of course, it all starts with opening an article in VIM. How I do this depends on my mood: I can open it by firing GVIM using Vimperator or I can mount the wiki with WikipediaFS. I prefer the latter.

Spell-checking

One tool that’s very convenient for editing large sections of prose is VIM’s built-in spell-checker. Example:

set textwidth=0
set spell spelllang=nl

Wrapping, matching and folding

There’s some more useful stuff that I copied to “ftplugin/mediawiki.vim” in “$HOME/.vim/“:

" Wikipedia articles often only have line-breaks at the end of each paragraph,
" a situation Vim by default doesn't handle as other text editors.
setlocal wrap linebreak
setlocal textwidth=0

" No auto-wrap at all
setlocal formatoptions-=tc formatoptions+=l
if v:version >= 602 | setlocal formatoptions-=a | endif

" Make navigation more amenable to the long wrapping lines.
noremap  k gk
noremap  j gj
noremap   gk
noremap   gj
noremap  0 g0
noremap  ^ g^
noremap  $ g$
noremap D dg$
noremap C cg$
noremap A g$a
inoremap   gk
inoremap   gj

" utf-8 should be set if not already done globally
setlocal fileencoding=utf-8
setlocal matchpairs+=<:>

" Treat lists, indented text and tables as comment lines and continue with the
" same formatting in the next line (i.e. insert the comment leader) when hitting
"  or using "o".
setlocal comments=n:#,n:*,n:\:,s:{\|,m:\|,ex:\|}
setlocal formatoptions+=roq

" match HTML tags (taken directly from $VIM/ftplugin/html.vim)
if exists("loaded_matchit")
    let b:match_ignorecase=0
    let b:match_skip = 's:Comment'
    let b:match_words = '<:>,' .
    \ '<\@<=[ou]l\>[^>]*\%(>\|$\):<\@<=li\>:<\@<=/[ou]l>,' .
    \ '<\@<=dl\>[^>]*\%(>\|$\):<\@<=d[td]\>:<\@<=/dl>,' .
    \ '<\@<=\([^/][^ \t>]*\)[^>]*\%(>\|$\):<\@<=/\1>'
endif

" Other useful mappings
" Insert a matching = automatically while starting a new header.
inoremap   = =(getline('.')==''\|\|getline('.')=~'^=\+$')?"==\Left>":"="

" Enable folding based on ==sections==
:set foldexpr=getline(v:lnum)=~'^\\(=\\+\\)[^=]\\+\\1\\(\\s*\\)\\=\\s*$'?\">\".(len(matchstr(getline(v:lnum),'^=\\+'))-1):\"=\"
:set fdm=expr

To have this automatically executed when opening files ending in .mw:

"autocmd BufRead,BufNewFile *.mw setfiletype mediawiki" > $HOME/.vim/ftdetect/mediawiki.vim

Syntax highlighting

Additionally, if you want to have syntax highlighting for MediaWiki articles, you can save this syntax file to $HOME/.vim/syntax/mediawiki.vim.

winepath

June 28, 2009 / Rowan Rodrik

I have all these little scripts in my $HOME/bin directory to ease the execution of Windows programs. For instance, I don’t like to type the full path to Excel Viewer every time when I need to view an Excel sheet in Linux:

#!/bin/sh
"/home/bigsmoke/.wine/drive_c/Program Files/Microsoft Office/OFFICE11/XLVIEW.EXE" "$*"

This works pretty well, except when trying to view a document that’s not in the current directory:

xlview "/tmp/Some Excel sheet sent to me by someone.xls"

To make that work, you need to use a handy utility that comes with Wine, winepath. With winepath, I can modify the script to work (the example is for wordview):

#!/bin/bash
path=`winepath --windows "$*"`
"/home/bigsmoke/.wine/drive_c/Program Files/Microsoft Office/OFFICE11/WORDVIEW.EXE" "$path"

This, together with Linux’s binfmt_misc makes executing Windows programs on Linux a breeze.

Native execution of Windows programs on Linux

June 28, 2009 / Rowan Rodrik

When I installed this laptop last year, I was pleasantly surprised by the performance of vanilla Wine (and Gentoo’s default Wine configuration). At that time, my only memory of Wine was that getting anything useful to work took work and that your best bet to get anything working was to install CrossOver Office. Things change and in the meantime Wine has even come to version 1.0.

I want to write down what I had to do to be able to execute Windows executables as if they are Linux native. This can be done thanks to Linux’ support for misc. binaries.

First, you have to enable CONFIG_BINFMT_MISC in your kernel configuration. If it was configured as a module, run /sbin/modprobe binfmt_misc.

Then you have to register the appropriate binary formats:

':windows:M::MZ::/usr/bin/wine:' > /proc/sys/fs/binfmt_misc/register

That’s it. Of course, the .exe files need to have their executable bit set:

chmod u+x program.exe

Done.

Xerox 7232 communication failure

June 22, 2009 / halfgaar

The Xerox 7232 color laser printer we’re leasing at work has a very vague problem causing communication failures all the time. When you boot it, it works fine. After a while it starts getting slow. And if you wait even longer, it doesn’t work at all anymore. This happens within a few minutes.

The problem seems to occur faster when you print from certain machines on the network. However, given enough time, none of them work anymore. The machines are a mix of Windows 2000 and Windows XP machines and there is no correlation between the OS and the degree of communication difficultly.

The problem has nothing to do with this existing Microsoft knowledge base item, as I will show with network traffic logs.

When the printer just starts and works fine, the communication is as follows: the client sends a [SYN] packet, the printer responds with [SYN, ACK] to which the client sends another [ACK]. At this point, the connection is established and the client starts sending data packages. After two packages the printer sends an [ACK] back. As the Microsoft’s knowledge base article states, TCP sends [ACK]’s every two packets, so this is normal. This repeats a number of times until the printer starts printing.

The following TCP/IP analysis shows it clearly (192.168.1.205 is the client, 192.168.1.103 is the printer. Don’t pay attention to the source port network name):

When the communication problems begin, the difference is that after the first two data packets are sent, the printer doesn’t send the expected [ACK] packet. The client keeps sending retransmissions until the printer eventually sends a [RST] (reset):

The printer obviously doesn’t adhere to TCP/IP standards. Xerox technical support likes to claim that the problem is in our network, because the fault doesn’t (easily) happen when they connect a laptop to the printer with a crosslink cable, but this evidence clearly shows the printer is at fault. Even if it starts doing this because of some invalid data packet our client PC’s might send, it’s still faulty behavior. To such an extent even, that if any server or client machine would have this problem, it would be classified as a security risk, because it is then susceptible to denial of service attacks.

Installing grub on RAID1

June 21, 2009 / halfgaar

When you have your kernel image and grub boot stages on a Linux software RAID1 partition, installing grub needs a bit of trickery.

It determines on what hardware device those files are located by looking at /etc/mtab. It will find /dev/md/0 for / or /boot (or whatever) where the files are located. That device has no BIOS device, because it is managed by the Linux kernel. Therefore, when you try to install grub, it says: “/dev/md/0 does not have any corresponding BIOS drive.“. But because the underlying partitions in a RAID1 have the same data as the virtual device, we can trick grub.

All we have to do is edit /etc/mtab and replace the /dev/md/0 with /dev/sda1 (or whatever). You can then run grub-install hd0. Of course, make a backup of mtab first so you can revert it after having installed grub.

I find it kind of weird that grub doesn’t determine that the device in question is a RAID1 and that it can simply use the underlying device; the internet is filled with people having the same problem. If anybody knows a more elegant solution than this, I’m all ears.

Based on the comments, this is what you need to install it from the grub prompt:

# grub
grub> root (hd0,0)
grub> setup(hd0)
grub> root (hd1,0)
grub> setup (hd1)

Fixing “expected NAV packet but none found” error on some downloaded DVD’s

June 19, 2009 / halfgaar

With certain DVD’s, Xine fails when seeking in the movie saying “Expected NAV packet but none found”. It seems that I have finally found out how to fix this and it is not even difficult. If the theory is correct, it only happens with DVD’s obtained as an ISO image.

What you need to do, is mount the image over the loopback, copy the data files to another location and playing those copied files should work. You can then burn them to a video DVD if you want to, of course.

I have no idea why this works or what causes the error, but I’m glad I finally figured it out.

Also, some of these DVD’s don’t even play on normal players (so it’s not just Xine). This should fix that as well, but I have yet to test that.

BigSmoke