Smokes your problems, coughs fresh air.

Tag: Debian (Page 1 of 3)

Creating a Xen bridge with bonding on Debian

Like this, but then with bonding:

# The loopback network interface
auto lo
iface lo inet loopback
auto bond0
# The primary network interface
iface bond0 inet manual
  slaves eth0 eth1
  bond-mode active-backup
  bond-miimon 100
  bond-downdelay 200
  bond-updelay 200
auto xenbr0
iface xenbr0 inet static
  bridge_ports bond0
  bridge_fd 0

I needed it because I had have a dedicated server that was setup with ethernet bonding that I wanted to install Xen on. Normally, you can let xen setup the network with “network-script network-bridge” in /etc/xen/xend-config.xsp, but that doesn’t work when you have bonding. And, it’s actually better to do it with Debian’s netconfig anyway.

Creating a drbd for an existing Xen domain

I needed some VMs to be available on a backup node, which I accomplished with the distributed remote block device, or DRBD. My host machine is Debian 6.

This post replaced an older one I made.

First install drbd:

aptitude -P install drbd8-utils

Then make some config files. First adjust /etc/drbd.d/global.conf (I only had to uncomment the notify rules):

global {
        usage-count yes;
        # minor-count dialog-refresh disable-ip-verification
common {
        protocol C;
        handlers {
                pri-on-incon-degr "/usr/lib/drbd/; /usr/lib/drbd/; echo b > /proc/sysrq-trigger ; reboot -f";
                pri-lost-after-sb "/usr/lib/drbd/; /usr/lib/drbd/; echo b > /proc/sysrq-trigger ; reboot -f";
                local-io-error "/usr/lib/drbd/; /usr/lib/drbd/; echo o > /proc/sysrq-trigger ; halt -f";
                # fence-peer "/usr/lib/drbd/";
                split-brain "/usr/lib/drbd/ root";
                out-of-sync "/usr/lib/drbd/ root";
                # before-resync-target "/usr/lib/drbd/ -p 15 -- -c 16k";
                # after-resync-target /usr/lib/drbd/;
        startup {
                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb;
                # The timeout value when the last known state of the other side was available.
                wfc-timeout 0;
                # Timeout value when the last known state was disconnected.
                degr-wfc-timeout 180;
        disk {
                # on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
                # no-disk-drain no-md-flushes max-bio-bvecs   
        net {
                # snd‐buf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
                # max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
                # after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
        syncer {
                # rate after al-extents use-rle cpu-mask verify-alg csums-alg

Then I made a resource for my existing logical volume:

resource r0
  meta-disk internal;
  device /dev/drbd1;
    # The timeout value when the last known state of the other side was available.
    wfc-timeout 0;
    # Timeout value when the last known state was disconnected.
    degr-wfc-timeout 180;
    # This is recommended only for low-bandwidth lines, to only send those
    # blocks which really have changed.
    #csums-alg md5;
    # Set to about half your net speed
    rate 8M;
    # It seems that this option moved to the 'net' section in drbd 8.4.
    verify-alg md5;
    # The manpage says this is recommended only in pre-production (because of its performance), to determine
    # if your LAN card has a TCP checksum offloading bug. 
    #data-integrity-alg md5;
    # Detach causes the device to work over-the-network-only after the
    # underlying disk fails. Detach is not default for historical reasons, but is
    # recommended by the docs.
    # However, the Debian defaults in drbd.conf suggest the machine will reboot in that event...
    on-io-error detach;
    # LVM doesn't support barriers, so disabling it. It will revert to flush. Check wo: in /proc/drbd. If you don't disable it, you get IO errors.
  on top
    disk /dev/universe/lvtest;
  on bottom
    disk /dev/universe/lvtest;

Copy all config files to the slave machine (and write an rsync-script for it…).

I learned that Linux 3.1 now has write barriers enabled by default for ext3 (they already were for ext4). This causes bugs and IO errors with xen-blkfront, so that needs to be disabled:

# grep barrier /etc/fstab
/dev/xvda2 / ext3 barrier=0 0 1

I’ll see about finding out if there are bug reports and file them if necessary.

The drbd data is going to be written on the actual LV, so on the primary node, we need to make space (you can also grow the LV):

e2fsck -f /dev/universe/lvtest
resize2fs /dev/universe/lvtest 500M # or however big that's a tad smaller than the actual LV.
drbdadm create-md r0
drbdadm up r0

On the secondary node, make the device as well:

drbdadm create-md r0
drbdadm up r0

Then we can start syncing and re-grow it. On the primary:

drbdadm -- --overwrite-data-of-peer primary r0 # the -- is necessary because of weird option handling by drbdadm.
resize2fs /dev/drbd1

The logical volume has been converted from ext3 to drbd:

# mount /dev/universe/lvtest /mnt/temp
mount: unknown filesystem type 'drbd'

Then, it is recommended you create /etc/modprobe.d/drbd.conf with:

options drbd disable_sendpage=1

I don’t know what it does, but it’s recommended by the DRBD devices docs when you put Xen domains on DRBD devices.

In Xen, you can configure the disk device of a VM like this (actually, I learned that this doesn’t work with pygrub):

disk = [ 'drbd:resource,xvda,w' ]

Drbd has installed the necessary scripts in /etc/xen/scripts to support this. Xen will now automatically promote a drbd device to primary when you start a VM.

Bewarned: because of that, don’t put the VM in the /etc/xen/auto dir on the fallback node, otherwise whichever machine is faster will start the VM, preventing the other machine from starting it (because you can’t have two primaries).

Then, I noticed that Debian arranges it’s boot process erroneously, starting xemdomains before drbd. I comment on an old bug.

You can fix it by adding xendomains to the following lines in /etc/init.d/drbd:

# X-Start-Before: heartbeat corosync xendomains
# X-Stop-After:   heartbeat corosync xendomains 

Mdadm (software RAID) schedules monthly checks of your array. You can do that for DRBD too). You do that on the primary node with a cronjob in /etc/cron.d/:

42 0 * * 0    root    /sbin/drbdadm verify all

One last thing: the docs state that when you perform a verify and it detects an out-of-sync device, all you have to do is disconnect and connect. That didn’t work for me. Instead, I ran the following on the secondary node (the one I had destroyed with dd) to initiate a resync:

drbdadm invalidate r0

Fixing mailscanner insecure dependancy

Mailscanner cut out on me, without errors in the log. It was only after turning on debug (which prevents backgrounding and it then only processes one batch) that it showed me.

I needed to change the first line of Mailscanner in /usr/sbin/Mailscanner:

#!/usr/bin/perl -I/usr/share/MailScanner/ -U 


Of course, this is far from optimal and it bugs me that Debian didn’t fix this yet, because it’s an old issue.

Executing system commands from PHP with SUID executable.

If you want to execute system commands from something like PHP, you need a SUID executable which you can call from your PHP scripts. This is such a script. It could be extended to support parameters for the commands you want to execute, but that would be an enormous security risk, because then anybody can execute any command. If you need something as flexible as that, you need to think about adding some kind of security restrictions, like a list of allowed commands.

When writing this, it occurred to me how unnecessary this all is. I will explain below. First, I will describe the old way.

Here is the c source code, as written for our backup script,

// Wrapper for the shell script, to be able to run it as root when
// started from a webserver, for example. Set the resulting executable to SUID
// root.
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <error.h>
int main(int argc, char *argv[], char *envp[])
    int set_uid_result;
    int effective_user_id;
    int execute_script_error;
    char* script;
    effective_user_id = geteuid();
    // Set real and effective user ID
    set_uid_result = setreuid(effective_user_id, effective_user_id);
    if (set_uid_result != 0)
        printf("Failed to set user id\n");
        return 1;
    script = "/usr/local/sbin/";
    // This does not return on success.
    execve(script, argv, envp);
    execute_script_error = errno;
    // Show a fancy error message.
    error(execute_script_error, execute_script_error, script);
    // Shouldn't be necceary, but you never know.
    return 1;

To compile:

gcc -o bsbackup bsbackup.c

You can then run this inside PHP:

// The 2>&1 makes all error messages appear on stdout, for easy capturing.
passthru('/usr/local/sbin/bsbackup usb_backup 2>&1');

As I said, when writing this, it all became very clear to me that it is quite useless. One can also install sudo, run visudo and put this in (assuming your webserver runs as www-data, like on (Debian and Ubuntu):

www-data ALL = NOPASSWD: /usr/local/sbin/

Then in PHP, just run this:

passthru('sudo /usr/local/sbin/ usb_backup 2>&1');

I haven’t tested whether specifying the parameters after the script in the passthru actually works, but I think so. If not, you can just write a wrapper script around the command you’re going to execute.

See what you like best 🙂

Allowing apache to set Nagios cmd file

On debian, to prevent:

Error: Could not stat() command file ‘/var/lib/nagios3/rw/nagios.cmd’!


/etc/init.d/nagios3 stop
dpkg-statoverride --update --add nagios www-data 2710 /var/lib/nagios3/rw
dpkg-statoverride --update --add nagios nagios 751 /var/lib/nagios3
/etc/init.d/nagios3 start


Disabling exim port 25 listening when zimbra is installed

Edit: This doesn’t work anymore I don’t think, because more modern versions check for SMTP conflicts and don’t allow this.

When you’re installing zimbra in an Ubuntu or Debian machine, it seems it installs the MTA in such a way that command line tools like mail and such don’t work. But when you install exim, it conflicts with the postfix in Zimbra.

To fix it, you can install exim4, but configure this line in /etc/default/exim4:



grml seems like an interesting Debian-based Linux Live CD. It seems interesting because “[it] includes a collection of GNU/Linux software especially for system administrators and users of texttools.”

Making a Debian Squeeze machine into a Xen virtual machine host

My attempts to get Xen working right on Debian stable (Lenny) were not really successful. Xen has had some interesting developments and the version in Lenny is just too old. Plus, the debian bootstrap scripts used to create images don’t support Ubuntu Maverick… Squeeze (testing) has the newest Xen and deboostrap, so that’s cool. I used the AMD64 architecture.

First install Xen:

aptitude -P install xen-hypervisor-4.0-amd64 linux-image-xen-amd64 xen-tools

Debian Sqeeuze uses Grub 2, and the defaults are wrong for Xen. The Xen hypervisor (and not just a Xen-ready kernel!) should be the first entry, so do this:

mv -i /etc/grub.d/10_linux /etc/grub.d/50_linux

Then, disable the OS prober, so that you don’t get entries for each virtual machine you install on a LVM.

"" >> /etc/default/grub "# Disable OS prober to prevent virtual machines on logical volumes from appearing in the boot menu." >> /etc/default/grub "GRUB_DISABLE_OS_PROBER=true" >> /etc/default/grub

Per default (on Debian anyway), Xen tries to save-state the VM’s upon shutdown. I’ve had some problems with that, so I set these params in /etc/default/xendomains to make sure they get shut down normally:


In /etc/xen/xend-config.sxp I enabled the network bridge (change an existing or commented-out line for this). I also set some other useful params (for me):

(network-script network-bridge)
(dom0-min-mem 196)
(dom0-cpus 0)
(vnc-listen '')
(vncpasswd '')
(Also, don’t forget to disable ballooning and setting a max memory.)

Then I edited /etc/xen-tools/xen-tools.conf. This config contains default values the xen-create-image script will use. Most important are:

# Virtual machine disks are created as logical volumes in volume group universe (LVM storage is much faster than file)
lvm = universe
install-method = debootstrap
size   = 50Gb      # Disk image size.
memory = 512Mb    # Memory size
swap   = 2Gb    # Swap size
fs     = ext3     # use the EXT3 filesystem for the disk image.
dist   = `xt-guess-suite-and-mirror --suite` # Default distribution to install.
gateway    = x.x.x.x
netmask    =
# When creating an image, interactively setup root password
passwd = 1
# I think this option was this per default, but it doesn't hurt to mention.
mirror = `xt-guess-suite-and-mirror --mirror`
mirror_maverick =
# Ext3 had some weird settings per default, like noatime. I don't want that, so I changed it to 'defaults'.
ext3_options     = defaults
ext2_options     = defaults
xfs_options      = defaults
reiserfs_options = defaults
btrfs_options    = defaults
# let xen-create-image use pygrub, so that the grub from the VM is used, which means you no longer need to store kernels outside the VM's. Keeps this very flexible.
# scsi is specified because when creating maverick for instance, the xvda disk that is used can't be accessed. 
# The scsi flag causes names like sda to be used.
# scsi=1 # no longer needed 

I created the following script to easily let me make VM’s:

# Script to easily create VM's. Hardy, maverick and Lenny are tested
if [ -z "$hostname" -o -z "$ip" -o -z "$dist" ]; then
  echo "No dist, hostname or ip specified"
  echo ""
  echo "Usage: $0 dist hostname ip"
  exit 1
if [ "$dist" == "hardy" ]; then
  serial_device="--serial_device xvc0"
xen-create-image $serial_device --hostname $hostname --ip $ip --vcpus 2 --pygrub --dist $dist

At this point, you can reboot (well, you could earlier, but well…).

Usage of the script should be simple. When creating a VM named ‘machinex’, start it and attach console:

xm create -c /etc/xen/machinex.cfg

You can escape the console with ctrl-]. Place a symlink in /etc/xen/auto to start the VM on boot.

As a sidenote: when creating a lenny, the script installs a xen kernel in the VM. When installing maverick, it installs a normal kernel. Normals kernels since version 2.6.32 (I believe) support pv_ops, meaning they can run on hypervisors like Xen’s.

Configuring a Debian exim server for internet delivery

I wanted an Exim server which could send mail on its own without the use of another SMTP server.

First see this for some general explantion.

I used this config file:

# /etc/exim4/update-exim4.conf.conf
# Edit this file and /etc/mailname by hand and execute update-exim4.conf
# yourself or use 'dpkg-reconfigure exim4-config'
# Please note that this is _not_ a dpkg-conffile and that automatic changes
# to this file might happen. The code handling this will honor your local
# changes, so this is usually fine, but will break local schemes that mess
# around with multiple versions of the file.
# update-exim4.conf uses this file to determine variable values to generate
# exim configuration macros for the configuration file.
# Most settings found in here do have corresponding questions in the
# Debconf configuration, but not all of them.
# This is a Debian specific file

Also remember to put that in /etc/mailname.

« Older posts

© 2024 BigSmoke

Theme by Anders NorenUp ↑