Author: Rowan Rodrik (Page 17 of 27)

Rowan is mainly a writer. This blog here is a dumping ground for miscellaneous stuff that he just needs to get out of his head. He is way more passionate about the subjects he writes about on Sapiens Habitat: the connections between humans, each other, and to nature, including their human nature.

If you are dreaming of a holiday in the forests of Drenthe (the Netherlands), look no further than “De Schuilplaats”: a beautiful vacation home, around which Rowan maintains a magnificent ecological garden and a private heather field, brimming with biological diversity.

FlashMQ is a business that offers managed MQTT hosting and other services that Rowan co-founded with Jeroen and Wiebe.

XKB compose sequences

September 7, 2009 / Rowan Rodrik

I just ended a post about inserting special characters in VIM with the remark that I should find out how to insert special punctuation marks using just XKB, so I set out to find out how to add these to the list of existing compose options for XKB. Turns out that I should have simply taken another look at the configuration file for Compose mode (/usr/X11R6/share/X11/locale/en_US.UTF-8/Compose on my system).

Now I learned how to easily type a bunch of characters that I find myself using so often that I know their Unicode numbers by heart. This going to save my a ton of time and annoyance, and there’s no longer any need to remember the VIM digraphs either, although these were equally easy to remember.

Char.	XML	HTML	XKB compose
—	`—`	`—`	`Compose - - -`
–	`–`	`–`	`Compose - - .`
…	`…`	`…`	`Compose . .`
‘	`‘`	`‘`	`Compose < '`
’	`’`	`’`	`Compose > '`
“	`“`	`“`	`Compose < "`
”	`”`	`”`	`Compose > "`

VIM tips for editing prose

September 7, 2009 / Rowan Rodrik

I was transcribing a draft for a manuscript. Using VIM, of course. But, I found my VIM skills to be lacking somewhat, enough to become sufficiently annoyed to investigate the holes.

Word wrapping

The first thing that I wanted to learn to remember was how to control word wrapping and, especially, how to rewrap text.

I had noticed already that on my current machine, VIM enables word wrapping by default for .txt files. I liked this, except I had forgotten long ago (or never properly remembered) how to rewrap lines. This can be done with gq. gq operates on the current selection or on the argument (a number of words/characters/sentences/paragraphs/etc).

Soon, I decided to turn my .txt into a simple TeX file (to be able to add annotations using TeX comments). This disabled the word wrapping, so I had to find the setting to control this. There's actually two settings:

wrapmargin defines how close the text may approach the right edge of the VIM window before it starts wrapping,
whereas textwidth tells VIM to start wrapping when a fixed number of characters is approached.

Because VIM doesn't do wrapping by default for .tex files, I added the following modeline to the bottom of my draft:

% vim: set textwidth=80 spelllang=nl:

Note that I find 80 characters way too small for most programming tasks, but very convenient as a width for reading prose from a screen. On occasion, I've even used width: 80ex; in the CSS of a website.

Sentences and paragraphs

A few movement commands that I've never used enough to remember well are {/} and (/). } and { are used to move a paragraph forward or backward, respectively;) and ( are used to move a whole sentence forward or backward. This is particularly useful while editing prose.

To quickly select a paragraph, for example, you can easily move to the beginning of the paragraph using {, press v to start a selection and go to the end of the paragraph with } (or type 2} to also select the next).

If you want to delete a sentence, go to the start of the sentence (using either ( or )) and type d). It’s as easy as that.

If deleting the sentence fucked up the formatting of your paragraph, reformat by going to the beginning of the paragraph and typing gq}.

Proper punctuation and other special characters

TeX offers a method to construct special characters using plain ASCII source files. In the past, in my inability to properly configure everything for UTF-8, I’ve often made use of this. In TeX, \'e will be turned into é, \"i into ï, etc. This can be convenient, but it’s much more convenient to have an environment that’s properly configured for UTF-8. To enter special characters on my US keyboard layout (standard in The Netherlands), I’ve added compose:ralt to my XKB options. Using this option I can press Right Alt followed by a punctuation character, followed by a character to combine it with.

Clearly, constructing special characters on the level of X holds many advantages over having to do this differently for each and every application. This way I can also type in ë in this HTML <textarea> instead of having to type ë. (In HTML, it’s actually better to use a numeric character reference, such as ë instead of ë, because that doesn’t require the loading of the DTD, but that’s another rant altogether.)

If you don’t have an accommodating XKB configuration, it’s still possible to enter the characters directly at the VIM level. In VIM, :help digraph (see also the on-line HTML version) tells you everything about it. In short, use Control+K followed by a punctuation character, followed by a character to compose special characters in a way similar to X.

What’s very nice about VIM’s default setup is that it allows you to also easily create proper opening and closing single and double quotes. In TeX these are traditionally done using combinations of back-ticks (`) and apostrophes ('). TeX’s default behavior can be problematic, a good reason to switch to Unicode.

char.	VIM digraph	TeX
`“`	`Ctrl+K, ", 6`	``
`”`	`Ctrl+K, ", 9`	`''`
`‘`	`Ctrl+K, ', 6`	`
`’`	`Ctrl+K, ', 9`	`'`

Something else that becomes very easy with VIM’s digraphs is entering proper punctuation characters, such as em/en dashes. These are done by following Ctrl+K with a hyphen and a capital N or M. In TeX these could already be done by simply entering two or three hyphens, but if you prefer it that way, you’re probably better of with the UniCycle plugin for VIM, which I personally don’t dig. Anyway, you’re running out of excuses to let -- appear in your production documents.

char.	VIM digraph	TeX
–	`Ctrl+K, -, N`	`--`
—	`Ctrl+K, -, M`	`---`

I have to admit that I’ve waited an awful long time before finding this out. I’m ashamed to tell you that I’ve often gone to Alan Wood’s Unicode resources to look up a character and copy/paste it into an application. 😳 Now, at least I don’t have to further embarrass myself when I’m using VIM.

What remains is to configure XKB in such a way that I don’t need to use VIM digraphs for punctuation. Then I will no longer need to use character references for punctuation at times like these, when I’m typing HTML/XML outside of VIM (or, worse, using copy/paste from VIM into this <textarea>, which I just did :oops:). Let’s see if I can get XKB to compose these using the same combinations as VIM. That is where I’ll continue my quest next time.

PHP fgetcsv() behavior on empty lines

September 5, 2009 / Rowan Rodrik

The PHP documentation for fgetcsv() states that A blank line in a CSV file will be returned as an array comprising a single null field, and will not be treated as an error. Here’s a quick demonstration of this behavior.

fgetcsv.php:

<?php
 
while ($fields = fgetcsv(STDIN, 0, ';'))
  print_r($fields);
 
exit(0);

Execute the script and feed it some CSV with empty lines:

php -q fgetcsv.php
"Veld 1";"Veld 2";"Veld 3";;"Veld 5"
 
"Field 1";;"Field 3";"Field 4";
;;;;
;"Campo 2";;;"Campo 5"

After pressing Ctrl+D, I’m presented with the following output:

Array
(
    [0] => Veld 1
    [1] => Veld 2
    [2] => Veld 3
    [3] => 
    [4] => Veld 5
)
Array
(
    [0] => 
)
Array
(
    [0] => Field 1
    [1] => 
    [2] => Field 3
    [3] => Field 4
    [4] => 
)
Array
(
    [0] => 
    [1] => 
    [2] => 
    [3] => 
    [4] => 
)
Array
(
    [0] => 
    [1] => Campo 2
    [2] => 
    [3] => 
    [4] => Campo 5
)
Array
(
    [0] => 
)

This behaviour on empty lines is a little bit annoying if you want to test if the line is empty():

$a = array(null);
print_r($a);
 
if ( empty($a) )
  echo '$a is empty';
else
  echo '$a is not empty';
 
echo "\n";

This code will print:

Array
(
    [0] => 
)
$a is not empty

Hence, the following function:

/**
 * This function tests if the given array (as returned by fgetcsv())
 * is the result of an empty line in the CSV file.
 *
 * It does not work for lines that contain only delimiters.
 * From the POV of this function, these are simply records with
 * many empty fields.
 */
function fgetcsv_empty_line($row_array) {
  return ( !isset($row_array[1]) and empty($row_array[0]) );
}

Now, if I change the call to empty() in my test to a call to fgetcsv_empty_line():

$a is empty

Tracking MediaWiki in a Subversion vendor branch

July 20, 2009 / Rowan Rodrik

Vendor branches are the proper™ way of merging upstream changes in your web application installations. In Subversion, managing vendor branches isn’t so easy as it is in Git. Still, vendor branches make it much easier to track upstream.

From before I first deployed the Omega Research Wiki, I already used svn to track changes to my MediaWiki installation. However, for upgrading from one upstream release to the next, I used diff and patch. This isn’t the most reliable of methods as is exemplified by doing a diff comparing a fresh MediaWiki download with the actual files in my repo (which are supposed to belong to the same version).

What’s basically a shortcoming of svn is that I can’t just say:

svn merge http://svn.wikimedia.org/svnroot/mediawiki/tags/REL1_15_0/phase3/ \
http://svn.wikimedia.org/svnroot/mediawiki/tags/REL1_15_1/phase3/ .

This would have been incredibly helpful, because now I’m keeping a vendor branch not because of local modifications to upstream, but just to be able to merge cleanly.

In Subversion, maintaining a vendor branch by hand is quite some work, because you need to do a checkout first before you can import each version. (My working copy is normally a checkout of /trunk, not of /vendor/mediawiki.) Luckily, Subversion is distributed with a handy Perl script, svn_load_dirs.pl, which can do most of the heavy lifting.

Still, I didn’t feel like having to do to many manual steps, such as typing in the painfully long URLs for merging, so I decided to wrap the whole process into a nice little Bash script:

cat upgrade-mediawiki.sh
#!/bin/bash
 
svn_repo_url=file:///var/svn/wiki.omega-research.org/vendor/mediawiki/
 
merge=1 [ $# -gt 2 ] # Process extra options
    "$1"
        --no-merge)
            merge=0
            ;;
   
   
 [ $# -ne 2 ]; then
    "Usage: $0 [--no-merge] version version"
    1
 
last_version=$1
new_version=$2
 
tmp_dir=`mktemp -d` $tmp_dir
 version $*;
    "Downloading and extracting MediaWiki version $version..."
 
    [ -z `svn ls $svn_repo_url|grep $version` ];
        branch= $version |sed -e 's/\.[0-9]\+$//'`
        download_file="mediawiki-$version.tar.gz"
        download_url="http://download.wikimedia.org/mediawiki/$branch/$download_file"
 
        wget $download_url || { "Downloading $download_file failed">&2; 1; }
        tar --extract --ungzip --transform 's/^mediawiki-//' --file $download_file
        svn_load_dirs.pl $svn_repo_url -t $version current $version
   
 - [ $merge == '1' ];
    svn merge "$svn_repo_url$last_version" "$svn_repo_url$new_version" .
 0

The script only downloads and imports each specified version if that version doesn’t already exist in /vendor/mediawiki/. Also, because it has a --no-merge option, you can download all the old versions of MediaWiki that you’ve ever used, so that you can go back and compare old versions of your installation with the factory version. Of course, this is only useful if you were already tracking your installation in svn at the time, and even then not really. 😉

Anyway, the important thing is that you can use the script to download the version you’re running now and the version you want to upgrade too. I wanted to make a big jump, from 1.11.1 to 1.15.0 (I had put of the upgrade for a long time, because I first wanted to learn more about vendor branches):

Of course, as always, you need to check if the patch went well. My own results were a vivid demonstration of the unreliability of my previous method:

svn st|grep '^C'

C      languages/messages/MessagesKrj.php
C      languages/messages/MessagesWar.php
C      languages/messages/MessagesSe.php
C      languages/messages/MessagesFrc.php

Luckily, these were all files I was sure I hadn’t modified:

i `svn st|grep '^C'|sed -e 's/^C //'`; mv $i.merge-right.r32 $i; svn resolved $i;

Resolved conflicted state of 'languages/messages/MessagesKrj.php'
Resolved conflicted state of 'languages/messages/MessagesWar.php'
Resolved conflicted state of 'languages/messages/MessagesSe.php'
Resolved conflicted state of 'languages/messages/MessagesFrc.php'

Apparently, a previous upgrade hadn’t turned my MediaWiki installation exactly into 1.11.1.

TheSixtyOne, a better social music site

July 20, 2009 / Rowan Rodrik

If you liked Last.fm, you’ll love TheSixtyOne. Discovering music is turned into a reputation game. Much like Last.fm, you can explore similar artists, genres, tags, moods and users. But, by doing so and hearting songs, you can earn reputation points and extra hearts. As your points increase, you level up, which gives you more authority. There’s an endless amount of quests to earn points faster and to make you discover all the facets of the application.

Just try it out. It’s fun and—unlike Last.fm these days–it’s free. You can choose when and how to help these starving artists! 😀

Blogging: quantity over quality

July 10, 2009 / Rowan Rodrik

They say that blogging is a long way down from essaying. Articles were already better than essays, and books even more so. So they say. A lot is said these days. I say: the more, the merrier. I say: let us say! Let us say whatever we want. And let people read whatever they want, now that they can.

I try to blog as much as I can, not because I think that each post is a piece of poetry that needs to be enjoyed by as many as possible, but because each post represents a train of thought that I need to get rid of. The sooner, the better. Noise is not a problem. The Internet is filled with noise, but nevertheless we only read what we want to read and we still find what we need. We filter. If you don’t, good luck surviving the digital age.

There is a convergence movement towards media integration. Eventually, this will lead to the collapse of all these media, meaning that the need for distinct, specific media will be less and less. This is good. We need to be truly press-less before the original promise of the press can be fulfilled: information liberation.

What I mean by this is that, when we publish, we don’t need to make a clear distinction between the weighty and the weightless. After all, God Google defines the weight of our publications not by means of what we say about it, but by means of incoming link frequency and the relative weight of these links.

When I say something worthwhile, I first have to convince others that I did, but before I can do that, I have to convince them to come look. Then, after visiting me and having been convinced by my great content, they might decide to link to me. Only as soon as that happens, do I gain weight. (According to Technorati, the weight of this blog is pathetic.)

Not every post I write resembles even an essay. Most postst don’t even come close. That’s because blog posts don’t need a point. I’ve made this point before, but I can’t repeat it often enough, because every time that I’m thinking “Shit, what did I do then and what was I thinking?” I’m reminded of it. I’ll stop reiterating this no sooner than I stop forgetting to write these things down.

When a blog post does attempt to make a point, there are even stronger arguments for quantity over quality, because publishing—really publishing—something makes it possible for others to respond, be this in comments or in their own blog. There’s something concrete for them to cite and to reference. This process gives me valuable feedback and I can simply republish my thoughts (which have matured not only through time, but also through argument) in a new post. Voila! And maybe the old post is so shitty that nobody is interested… Who cares?! Obviously, they don’t! So go on and do something else with the time that you’ve freed by not having to polish a turd.

Remember that blog posts are not naturally organized into some sort of reading hierarchy. There’s no reason that the same subject can’t be treated twice. Just hit that “Publish” button, get it out of your head and get on with your life!

Editing Wikipedia/MediaWiki articles with VIM

July 10, 2009 / Rowan Rodrik

Wikipedia has a number of tips for people who like to edit Wikipedia articles with VIM.

Of course, it all starts with opening an article in VIM. How I do this depends on my mood: I can open it by firing GVIM using Vimperator or I can mount the wiki with WikipediaFS. I prefer the latter.

Spell-checking

One tool that’s very convenient for editing large sections of prose is VIM’s built-in spell-checker. Example:

set textwidth=0
set spell spelllang=nl

Wrapping, matching and folding

There’s some more useful stuff that I copied to “ftplugin/mediawiki.vim” in “$HOME/.vim/“:

" Wikipedia articles often only have line-breaks at the end of each paragraph,
" a situation Vim by default doesn't handle as other text editors.
setlocal wrap linebreak
setlocal textwidth=0

" No auto-wrap at all
setlocal formatoptions-=tc formatoptions+=l
if v:version >= 602 | setlocal formatoptions-=a | endif

" Make navigation more amenable to the long wrapping lines.
noremap  k gk
noremap  j gj
noremap   gk
noremap   gj
noremap  0 g0
noremap  ^ g^
noremap  $ g$
noremap D dg$
noremap C cg$
noremap A g$a
inoremap   gk
inoremap   gj

" utf-8 should be set if not already done globally
setlocal fileencoding=utf-8
setlocal matchpairs+=<:>

" Treat lists, indented text and tables as comment lines and continue with the
" same formatting in the next line (i.e. insert the comment leader) when hitting
"  or using "o".
setlocal comments=n:#,n:*,n:\:,s:{\|,m:\|,ex:\|}
setlocal formatoptions+=roq

" match HTML tags (taken directly from $VIM/ftplugin/html.vim)
if exists("loaded_matchit")
    let b:match_ignorecase=0
    let b:match_skip = 's:Comment'
    let b:match_words = '<:>,' .
    \ '<\@<=[ou]l\>[^>]*\%(>\|$\):<\@<=li\>:<\@<=/[ou]l>,' .
    \ '<\@<=dl\>[^>]*\%(>\|$\):<\@<=d[td]\>:<\@<=/dl>,' .
    \ '<\@<=\([^/][^ \t>]*\)[^>]*\%(>\|$\):<\@<=/\1>'
endif

" Other useful mappings
" Insert a matching = automatically while starting a new header.
inoremap   = =(getline('.')==''\|\|getline('.')=~'^=\+$')?"==\Left>":"="

" Enable folding based on ==sections==
:set foldexpr=getline(v:lnum)=~'^\\(=\\+\\)[^=]\\+\\1\\(\\s*\\)\\=\\s*$'?\">\".(len(matchstr(getline(v:lnum),'^=\\+'))-1):\"=\"
:set fdm=expr

To have this automatically executed when opening files ending in .mw:

"autocmd BufRead,BufNewFile *.mw setfiletype mediawiki" > $HOME/.vim/ftdetect/mediawiki.vim

Syntax highlighting

Additionally, if you want to have syntax highlighting for MediaWiki articles, you can save this syntax file to $HOME/.vim/syntax/mediawiki.vim.

winepath

June 28, 2009 / Rowan Rodrik

I have all these little scripts in my $HOME/bin directory to ease the execution of Windows programs. For instance, I don’t like to type the full path to Excel Viewer every time when I need to view an Excel sheet in Linux:

#!/bin/sh
"/home/bigsmoke/.wine/drive_c/Program Files/Microsoft Office/OFFICE11/XLVIEW.EXE" "$*"

This works pretty well, except when trying to view a document that’s not in the current directory:

xlview "/tmp/Some Excel sheet sent to me by someone.xls"

To make that work, you need to use a handy utility that comes with Wine, winepath. With winepath, I can modify the script to work (the example is for wordview):

#!/bin/bash
path=`winepath --windows "$*"`
"/home/bigsmoke/.wine/drive_c/Program Files/Microsoft Office/OFFICE11/WORDVIEW.EXE" "$path"

This, together with Linux’s binfmt_misc makes executing Windows programs on Linux a breeze.

Native execution of Windows programs on Linux

June 28, 2009 / Rowan Rodrik

When I installed this laptop last year, I was pleasantly surprised by the performance of vanilla Wine (and Gentoo’s default Wine configuration). At that time, my only memory of Wine was that getting anything useful to work took work and that your best bet to get anything working was to install CrossOver Office. Things change and in the meantime Wine has even come to version 1.0.

I want to write down what I had to do to be able to execute Windows executables as if they are Linux native. This can be done thanks to Linux’ support for misc. binaries.

First, you have to enable CONFIG_BINFMT_MISC in your kernel configuration. If it was configured as a module, run /sbin/modprobe binfmt_misc.

Then you have to register the appropriate binary formats:

':windows:M::MZ::/usr/bin/wine:' > /proc/sys/fs/binfmt_misc/register

That’s it. Of course, the .exe files need to have their executable bit set:

chmod u+x program.exe

Done.

Gentoo update: WiFi drivers (iwl4965)

June 11, 2009 / Rowan Rodrik

When I got the Wireless working on this laptop (a Lenovo T61), I was using Linux kernel 2.6.25. I remember that it was quite a pain, because the iwlwifi stuff had only just stabilized enough to replace the (now deprecated) ipw stuff. One of the things I didn’t get working at the time was Monitor mode.

Now, I’m upgrading to kernel 2.6.29, and there have been some changes to the kernel configuration for the iwl4965 driver. CONFIG_IWL4965 is now a suboption of CONFIG_IWLAGN. The wireless didn’t work on the first boot however. eth0 didn’t work either. But, when I changed the wired driver from “Intel PRO/1000 Gigabit Ethernet” (CONFIG_E1000) to the same for PCI-express (CONFIG_E1000E), it did work on the next boot and the wireless magically started working too. 😕

I was surprised because I don’t see what the wired and the wireless driver have got to do with each other. Also, before, I’ve always successfully used the non-PCI-express driver for my wired Ethernet interface and I’m so old fashioned that I can’t imagine either way that my hardware uses something so modern as PCI-express. 😛

Anyway, it works now. I’ve gone from kernel 2.6.25 to 2.6.29 and everything seems to be working. I still have to find out how to get Monitor mode working though.

One final thing I noticed is that upgrading net-wireless/iwl4965-ucode created an extra entry in my world profile: net-wireless/iwl4965-ucode:0. I wonder what that is for.