Smokes your problems, coughs fresh air.

Tag: CLI (Page 1 of 2)

Readline shortcuts: Ctrl-s and XOFF

And another contribution to my list of Readline keyboard shortcuts. This time it’s a note by Joăo Matos, pointing me to problems with the Ctrl-s shortcut. (It acts as a flow control character that pauses transmission on some terminals.)

He adds that:

It’s easy to disable special treatment of Ctrl-s by the terminal with “stty -ixon”.

Remember also that, if you accidentally hit Ctrl-s, it’s easy to undo the effects with Ctrl-q.

Thanks Joăo!

Converting a Subversion repository to Git

I just used the instructions in this article by John Albin to archive an old svn project on my private machine.

A shell summary (see the John’s article for details):

svn log -q | awk -F '|' '/^r/ {sub("^ ", "", $2); sub(" $", "", $2); print $2" = "$2" <"$2">"}' | sort -u > authors-transform.txt
 
vim authors-transform.txt
# Make changes
 
git svn clone [SVN repo URL] --no-metadata -A authors-transform.txt --stdlayout ~/temp
 ~/temp
git svn show-ignore > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'
 
git init --bare ~/new-bare.git ~/new-bare.git
git symbolic-ref HEAD refs/heads/trunk
 ~/temp
git remote add bare ~/new-bare.git
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
 ~
rm -rf ~/temp
 ~/new-bare.git
git branch -m trunk master
 ~/new-bare.git
git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 | ref
  git tag "$ref" "refs/heads/tags/$ref";
  git branch -D "tags/$ref";

John has also put all this into a number of scripts published on GitHub.

Effective CLI habits

Just an example of some effective CLI magic that I copy/pasted into a draft aboutexactly a year ago. Can you see what’s happening? I’m moving some selected files into a subdirectory.

$ ls *png
boucoule-17jaar-met-steen.png         evening_cloud.png  small-map-molenweg.png  tile11.png
boucoule-2001-2002-face5-400x300.png  hardwood-logo.png  step-01.png             tile9a.png
$ ls *png|while read f; do echo $f; done
boucoule-17jaar-met-steen.png
boucoule-2001-2002-face5-400x300.png
evening_cloud.png
hardwood-logo.png
small-map-molenweg.png
step-01.png
tile11.png
tile9a.png
$ ls *png|while read f; do svn mv $f index; done
A         index/boucoule-17jaar-met-steen.png
D         boucoule-17jaar-met-steen.png
A         index/boucoule-2001-2002-face5-400x300.png
D         boucoule-2001-2002-face5-400x300.png
A         index/evening_cloud.png
D         evening_cloud.png
A         index/hardwood-logo.png
D         hardwood-logo.png
A         index/small-map-molenweg.png
D         small-map-molenweg.png
A         index/step-01.png
D         step-01.png
A         index/tile11.png
D         tile11.png
A         index/tile9a.png
D         tile9a.png

Bonus points if you notice that I could have moved the JPEGs and PNGs in one command instead of doing the same thing for the second time for the JPEGs as below. (I probably forgot that I also had some JPEGs lying around, or there must have been some other lame excuse.)

$ ls *jpg
bruggetje-225x300.jpg  favicon.jpg  purple-rowan.jpg        rowan-2007.jpg                rowan-wilderness.jpg
bruggetje.jpg          hekje.jpg    rowan-2007-448x300.jpg  rowan-wilderness-400x300.jpg
$ ls *jpg|grep -v favi
bruggetje-225x300.jpg
bruggetje.jpg
hekje.jpg
purple-rowan.jpg
rowan-2007-448x300.jpg
rowan-2007.jpg
rowan-wilderness-400x300.jpg
rowan-wilderness.jpg
$ ls *jpg|grep -v favi|while read f; do svn mv $f index; done
A         index/bruggetje-225x300.jpg
D         bruggetje-225x300.jpg
A         index/bruggetje.jpg
D         bruggetje.jpg
A         index/hekje.jpg
D         hekje.jpg
A         index/purple-rowan.jpg
D         purple-rowan.jpg
A         index/rowan-2007-448x300.jpg
D         rowan-2007-448x300.jpg
A         index/rowan-2007.jpg
D         rowan-2007.jpg
A         index/rowan-wilderness-400x300.jpg
D         rowan-wilderness-400x300.jpg
A         index/rowan-wilderness.jpg
D         rowan-wilderness.jpg

Posting to WordPress via the command-line

In February I was interested in posting to WordPress from the command-line, a possibility that I enjoyed when I wanted to apply some CLI-magic to some of my MediaWiki installations in the past.

I came across a great Perl module (WordPress::CLI) by Leo Charre. It depends on WordPress::XMLRPC

Another approach is to use the appfs FUSE filesystem, which uses WordPress’ support for the Atom Publishing Protocol. There’s another FUSE filesystem, BlogFS. This one depends on WordPress’ XML-RPC instead of its Atom interface.

Monitor the progress of Unix commands with Pipe Viewer (pv)

I just stumbled across the following post while trying to find out how to copy text from VIM using XSel without losing the selected text. It introduces Pipe Viewer, a Unix utility which is a kind of cat with a progress bar.

I emerged it (it’s in Gentoo (Debian too)) and it works very simple, but allows you to do cooler, more complicated things.

# pv emerge.log |gzip >emerge.log.gz
1.24MB 0:00:00 [1.76MB/s] [================>] 100%  
$ pv -cN source access_log | gzip | pv -cN gzip > access_log.gz
   source: 28.7MB 0:00:00 [32.2MB/s] [=====>] 100%            
     gzip: 2.27MB 0:00:00 [2.54MB/s] [ <=>  ]

The first example is easy enough to understand when you mentally substitute pv with cat. The second example is much cooler. It uses the -N flag to make named groups and the -c flag to make sure that the output for these groups doesn’t get garbled.

Read Peteris Krumins’ article for more cool uses of Pipe Viewer.

XSel, for command-line operations on X selections

Since I first learned that Windowmaker installs two command-line tools, wxcopy and wxpaste, to play around with X selections, I have wanted to be able to make and use X selections from my Bash shell. wxcopy and wxpaste never did what I expected them to do, so I gave up until recently I learned about all the different X selections.

By default, wxcopy and wxpaste operate on the CUT_BUFFER[n] selections. These are deprecated. That’s why I could never make it work, because modern applications use only CLIPBOARD and SELECTION. So, wxcopy is pretty useless (unless its used to copy something to paste with wxpaste). With this knowledge wxcopy does seem useful thanks to its -selection [selection-name] flag, but this doesn’t seem to work; I only get the contents of CUT_BUFFER. This is not how the feature is advertised:

-selection [selection-name]
The data will be copied from the named selection. If cutting from the selection fails, the cutbuffer will be used. The default value for the selection name is PRIMARY.

Enter XSel

Fortunately, there’s XSel by Conrad Parker, a program which made him passionately hate the ICCCM.

XSel does exactly what it advertises. I’m actually surprised that I never heard of it before. It’s available in Gentoo, Debian and Ubuntu, so it’s a breeze to install.

Among its features are: --append, --follow, --clear, --delete (very weird, but logical if you understand X IPC), --primary, --secondary, --clipboard, --keep, and --exchange. Read the man page for more. It’s an excellent read.

One of the places where I’m going to use this tool is when copy-pasting to and from VIM. I really like how this compares to using :insert or :r!cat</dev/tty and then using the pointer to paste (or (Shift+)Insert with my custom XTerm config). Now, to paste something in VIM, I can simply type:

:r!xsel

I use the following to copy any amount of text from VIM. This works much better than fooling around with the mouse:

:'>,'> !tee >(xsel -i)

The '>,'> range is entered automatically if you press : while in visual (selection) mode. You could enter any range there, or even % to select the whole file. To copy to the CLIPBOARD instead of the PRIMARY, use xsel -i -b in the above example.

If someone know of a way to make VIM pipe something to a program without replacing the given range with that program’s output, I could simplify this…

Replacing the full contents of a Subversion working (sub)dir

The annoyances that I suffered earlier today during the upgrade of a WordPress plugin made me turn to my favorite text-editor to create a simple script, svn-replace-dir:

#!/bin/bash
 
usage() {
    cat <<"EOF"
$0 [--dry-run] <svn_dir> <replacement_dir>
 
This script replaces the contents of <svn_dir> with the contents of <replacement_dir>,
where <replacement_dir> is not an svn directory.
 
Copyleft 2010, Rowan Rodrik van der Molen <rowan@bigsmoke.us>
EOF
}
 
fatal_error() {
    message=$1
 
    -e "\e[1;31m$message\e[0m"
    1
}
 
usage_error() {
    error="Wrong usage."
 
    [ -n "$1" ];
        error=$1
   
 
    -e "\e[1;31m$error\e[0m"
    1
}
 
run_command() {
    -e "\e[1;34m$1\e[0m"
 
    [ $dry_run == 1 ] || $1
}
 
dry_run=0 [ $1 == '--dry-run' ];
  dry_run=1
 
 
 
[ $# == 2 ] || usage_error "Wrong number of arguments."
 
svn_dir= "$1"|sed -e 's#/$##'`
replacement_dir= "$2"|sed -e 's#/$##'`
begin_path=$PWD
 
#if [ "${svn_dir:0:1}" != "/" ]; then svn_dir="$PWD/$svn_dir"; fi
#if [ "${replacement_dir:0:1}" != "/" ]; then replacement_dir="$PWD/$replacement_dir"; fi
 
[ -d "$svn_dir" ] || usage_error "$svn_dir is not a directory."
[ -d "$replacement_dir" ] || usage_error "$replacement_dir is not a directory."
 
 
# Create all subdirectories in $svn_dir that do not yet exist
$replacement_dir
find . -mindepth 1  d -print | sed -e 's#^./##' | d;
    $begin_path/$svn_dir
    # Doesn't the destination directory already exist?
    [ ! -d "$d" ];
        run_command "svn mkdir '$d'"
   
 
# Copy all files from $replacement_dir to $svn_dir
$begin_path/$replacement_dir
find .  f -print | sed -e 's#^./##' | f;
    $begin_path
    run_command "cp '$replacement_dir/$f' '$svn_dir/$f'" # FIXME: Quoting problem
 
# Remove all files that do no longer exist in $replacement dir
$begin_path/$svn_dir
find .  f -print | grep -v '.svn' | f;
    [ ! -f "$begin_path/$replacement_dir/$f" ];
        run_command "svn rm '$f'"
   
 
# Remove all subdirs that do no longer exist in $replacement dir
$begin_path/$svn_dir
find . -mindepth 1  d -print | grep -v '.svn' | d;
    [ ! -d "$begin_path/$replacement_dir/$d" ];
        run_command "svn rm '$d'"
   
 0

Using the script is simple:

svn-replace-dir simple-tags new-simple-tags|less -R

It replaces all the contents of the first directory (simple-tags in the example) with those of the second directory and it deletes everything that is no longer present in the second dir. In the process, it does all the necessary calls to svn mkdir, svn rm and (in the next version) svn add.

diff tells me that the script has done its work correctly:

diff -x .svn -ruN simple-tags new-simple-tags
# Emptiness is bliss :-) 

This is another one of these occasions when Git would have made life so much easier. Luckily, at least there’s GitHub to host this script as a Gist. Check there if you want to fetch the newest version of this script.

Dumping a bunch of MP3s as WAV files

I just used the following command for converting a directory with a bunch of MP3s to WAV. Does someone know a command that is shorter? I find mine a bit convoluted to say the least.

mkdir wav
ls *mp3|while i;  mpg123 --stdout "$i" > wav/ $i|sed -r -e 's/^([0-9]+) .*$/\1/'`.wav;

The directory looks like this:

$ ls -1
01 - Meant to Be.mp3
02 - Reflections.mp3
03 - Semester Days.mp3
04 - Friend.mp3
05 - True Gemini.mp3
06 - Down the Road.mp3
07 - Tulip Trees.mp3
08 - Not Alone.mp3
09 - Woods of Chaos.mp3
10 - Twilight.mp3
[en] Readme - www.jamendo.com .txt
[es] Lee me - www.jamendo.com .txt
[fr] Lisez moi - www.jamendo.com .txt
[it] Readme - www.jamendo.com .txt
License.txt
Rob Costlow - Solo Piano - Woods of Chaos.1.0.jpg

Tracking WordPress in a Subversion vendor branch

Two months prior to writing a script to upgrade MediaWiki installations using Subversion vendor branches, I wrote something similar for WordPress. It’s a little bit more limited and should really incorporate some of the improvements made for the MediaWiki version, but it worked fine so far:

cat upgrade-wordpress.sh 
#!/bin/bash
 
svn_repo_url=file:///var/svn/blog.omega-research.org/vendor/wordpress/
last_version=$1
new_version=$2
 
tmp_dir=`mktemp -d` $tmp_dir
 version $*;
    "Downloading and extracting WordPress version $version..."
 
    [ -z `svn ls $svn_repo_url|grep $version` ];
        branch= $version |sed -e 's/\.[0-9]\+$//'`
        archive_file="wordpress-$version.tar.gz"
        md5_file="wordpress-$version.md5"
 
        wget "http://wordpress.org/$md5_file"
        wget "http://wordpress.org/$archive_file"
 
        [ `md5sum $archive_file |cut -f 1 -d ' '` != `cat $md5_file` ];
            "MD5 sum did not match!" >&2
            1
       
 
        tar --extract --ungzip --transform "s/^wordpress/$version/" --file $archive_file
        svn_load_dirs.pl $svn_repo_url -t $version current $version
   
 -
svn merge "$svn_repo_url$last_version" "$svn_repo_url$new_version" .

I’m actually planning to make both scripts a little bit more generic (in the sense that svn_repo_url becomes an external param) and to track future changes to them using GitHub’s Gist. (How ironic is that, tracking an script for Subversion using Git?)

« Older posts

© 2024 BigSmoke

Theme by Anders NorenUp ↑