Author: halfgaar (Page 8 of 26)

Halfgaar is Wiebe. Wiebe is a contributing author on this weblog. He also has a lot of stuff (such as long, in-depth articles) on his personal website.

Wiebe's day job is as a senior software developer and system administrator at YTEC.

In his free time, he built the free, open-source FlashMQ software. Together with Jeroen and Rowan, he is now building a managed MQTT hosting business around his open masterpiece.

Getting munin to run every 10 minutes

December 18, 2012 / halfgaar

Munin is kind of inefficient and on my P4 2Ghz, it running every 5 minutes is too often, and the munin processes keep dying because locks already exist. You can’t increase the munin-cron script to 10 minute intervals, because then rrdtool will generate gaps.

The munin-cron script is nothing but a wrapper for munin-graph, munin-html and munin-update. I made my own wrappers, which I then run with separate cronjobs:

# cat /usr/local/bin/munin-graph 
#!/bin/bash
 
# file copied from /usr/bin/munin-cron and adjusted.
 
# This used to test if the executables were installed.  But that is
# perfectly redundant and supresses errors that the admin should see.
 
#/usr/share/munin/munin-update $@ || exit 1
 
# The result of munin-limits is needed by munin-html but not by
# munin-graph.  So run it in the background now, it will be done
# before munin-graph.
 
# When runnin update at */5 and graph at */10, munin-update and munin-graph
# will be started at the same time, and this sleep it to prevent a
# race-condition on the update-running file.
sleep 5
 
while [ -f "/var/run/munin/update-running" ]; do
        sleep 1
done
 
/usr/share/munin/munin-limits $@ &
 
nice /usr/share/munin/munin-graph --cron $@ 2>&1 | fgrep -v "*** attempt to put segment in horiz list twice"
 
wait
 
nice /usr/share/munin/munin-html $@ || exit 1

# cat /usr/local/bin/munin-update-data
#!/bin/bash
 
# file copied from /usr/bin/munin-cron and adjusted.
 
# This used to test if the executables were installed.  But that is
# perfectly redundant and supresses errors that the admin should see.
 
runfile="/var/run/munin/update-running"
touch "$runfile"
 
/usr/share/munin/munin-update $@ || exit 1
 
rm "$runfile"
 
# The result of munin-limits is needed by munin-html but not by
# munin-graph.  So run it in the background now, it will be done
# before munin-graph.
 
#/usr/share/munin/munin-limits $@ &
 
#nice /usr/share/munin/munin-graph --cron $@ 2>&1 | fgrep -v "*** attempt to put segment in horiz list twice"
 
#wait
 
#nice /usr/share/munin/munin-html $@ || exit 1

# cat /etc/cron.d/munin
#
# cron-jobs for munin
#
 
MAILTO=root
 
#*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi
*/5 * * * *     munin if [ -x /usr/local/bin/munin-update-data ]; then /usr/local/bin/munin-update-data; fi
*/10 * * * *    munin if [ -x /usr/local/bin/munin-graph ]; then /usr/local/bin/munin-graph; fi
14 10 * * *     munin if [ -x /usr/share/munin/munin-limits ]; then /usr/share/munin/munin-limits --force --contact nagios --contact old-nagios; fi

Adding a clock in screen to avoid your ssh’s from being killed

October 31, 2012 / halfgaar

The world is filled with stupid routers, which kill all connections that have no activity for a while (even a very short while). I keep loosing my SSH sessions because of this. To fix it, I added a clock in my GNU screen bar:

hardstatus alwayslastline "%= %H | %l | [%c:%s]"

For the record, my entire .screenrc:

multiuser on
caption always "%{= kB}%-Lw%{=s kB}%50>%n%f* %t %{-}%+Lw%<"
vbell off
startup_message off
term linux
hardstatus alwayslastline "%= %H | %l | [%c:%s]"

Converting all tables in MySQL DB to InnoDB

October 2, 2012 / halfgaar

#!/bin/bash
 
exit 1
 
dbname="eorder"
 
echo 'SHOW TABLES;'  | mysql $dbname  | awk '!/^Tables_in_/ {print "ALTER TABLE `"$0"` ENGINE = InnoDB;"}'  | column -t 
echo 'SHOW TABLES;'  | mysql $dbname  | awk '!/^Tables_in_/ {print "ALTER TABLE `"$0"` ENGINE = InnoDB;"}'  | column -t | mysql $dbname

Translating Dutch programming error messages

August 9, 2012 / halfgaar

In .NET, you get Dutch errors when your Windows is in Dutch, which is hugely annoying. You can use this site to translate them, so that you can Google them.

My universal remote programming codes

August 6, 2012 / halfgaar

Whenever this remote’s battery is loose for a while, it forgets its programming. So, here it is:

For computer, use the TV setting. Press 1 and 3 for a few seconds, then when the light turns on, enter 0677 as code.

For my amp, you have to set the aux mode for a second TV. To do that, press 1 and 6 for a while. Then press 9, 9, 2. Then press TV, then aux. Next step is to enter the TV code. Press aux, then hold 1 and 3. Enter the following code: 0064.

Kart racing scores

July 25, 2012 / halfgaar

I want to keep track of my kart racing scores:

date	Track	Racer	fastest Time
2012-07-24	Long Beach	Wiebe	55.95
2012-07-24	Long Beach	Wiebe	56.22
2012-07-24	Long Beach	Wiebe	58.84

Apache mod_proxy configuration for The Pirate Bay

July 13, 2012 / halfgaar

I found several apache mod_proxy configs for setting up a proxy for The Pirate Bay, but none worked fully.

You need to enable/install:

mod_proxy
mod_rewrite
mod_headers
mod_proxy_http

<Virtualhost *:80>
        ServerName tpb.yourdomain.com
 
        # Plausible deniability, and respecting your fellow pirate's privacy.
        Loglevel emerg
        CustomLog /dev/null combined
        ErrorLog /dev/null
 
        <Proxy *>
          Order deny,allow
          Allow from all
        </Proxy>
 
        # Just to fix a few links...
        RewriteEngine On
        RewriteRule \/static\.thepiratebay\.se\/(.*)$ /static/$1 [R=302,L]
 
        ProxyRequests off
 
        # Cookies are imporant to be able to disable the annoying double-row mode.
        # The . before the domain is required, but I don't know why :)
        ProxyPassReverseCookieDomain .thepiratebay.se tpb.yourdomain.com
 
        ProxyPass / http://thepiratebay.se/
        ProxyPass /static/ http://static.thepiratebay.se/
        ProxyPass /torrents/ http://torrents.thepiratebay.se/
        ProxyHTMLURLMap http://thepiratebay.se /
        ProxyHTMLURLMap http://([a-z]*).thepiratebay.se /$1 R
 
        ProxyHTMLEnable On
 
        <Location /static/>
          ProxyPassReverse /
          SetOutputFilter proxy-html
          ProxyHTMLURLMap / /static/
          RequestHeader unset Accept-Encoding
        </Location>
 
        <Location /torrents/>
          ProxyPassReverse /
          SetOutputFilter proxy-html
          ProxyHTMLURLMap / /torrents/
          RequestHeader unset Accept-Encoding
        </Location>
 
</Virtualhost>

Listing MySQL table sizes

July 12, 2012 / halfgaar

This query lists the sizes of all tables in MySQL:

SELECT 
  TABLE_SCHEMA, 
  TABLE_NAME, 
  CONCAT(ROUND(data_length / ( 1024 * 1024 ), 2), 'MB') DATA, 
  CONCAT(ROUND(data_free  / ( 1024 * 1024 ), 2), 'MB') FREE 
from information_schema.TABLES 
where TABLE_SCHEMA NOT IN ('information_schema','mysql', 'performance_schema');

This query lists the database sizes:

SELECT 
  TABLE_SCHEMA, 
  TABLE_NAME, 
  CONCAT(ROUND(sum(data_length) / ( 1024 * 1024 ), 2), 'MB') DATA 
from information_schema.TABLES 
where TABLE_SCHEMA NOT IN ('information_schema','mysql', 'performance_schema') 
group by TABLE_SCHEMA;

Generating an SSL CSR and key

May 10, 2012 / halfgaar

To generate an SSL certificate signing request (CSR) with key you can do this:

openssl req -nodes -newkey rsa:2048 -keyout bla.key -out bla.csr

This syntax does not force you to supply a password, which is convenient.

If you generate a CSR for startcom, you don’t have to fill in any fields; only the public key from the CSR is used. For other vendors, the common name is important; the domain name must be entered there.

Creating a drbd for an existing Xen domain

May 1, 2012 / halfgaar

I needed some VMs to be available on a backup node, which I accomplished with the distributed remote block device, or DRBD. My host machine is Debian 6.

This post replaced an older one I made.

First install drbd:

aptitude -P install drbd8-utils

Then make some config files. First adjust /etc/drbd.d/global.conf (I only had to uncomment the notify rules):

global {
        usage-count yes;
        # minor-count dialog-refresh disable-ip-verification
}
 
common {
        protocol C;
 
        handlers {
                pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
        }
 
        startup {
                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb;
 
                # The timeout value when the last known state of the other side was available.
                wfc-timeout 0;
 
                # Timeout value when the last known state was disconnected.
                degr-wfc-timeout 180;
        }
 
        disk {
                # on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
                # no-disk-drain no-md-flushes max-bio-bvecs   
        }
 
        net {
                # snd‐buf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
                # max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
                # after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
        }
 
        syncer {
                # rate after al-extents use-rle cpu-mask verify-alg csums-alg
        }
}

Then I made a resource for my existing logical volume:

resource r0
{
  meta-disk internal;
  device /dev/drbd1;
 
  startup
  {
    # The timeout value when the last known state of the other side was available.
    wfc-timeout 0;
 
    # Timeout value when the last known state was disconnected.
    degr-wfc-timeout 180;
  }
 
  syncer
  {
    # This is recommended only for low-bandwidth lines, to only send those
    # blocks which really have changed.
    #csums-alg md5;
 
    # Set to about half your net speed
    rate 8M;
 
    # It seems that this option moved to the 'net' section in drbd 8.4.
    verify-alg md5;
  }
 
  net
  {
    # The manpage says this is recommended only in pre-production (because of its performance), to determine
    # if your LAN card has a TCP checksum offloading bug. 
    #data-integrity-alg md5;
  }
 
  disk
  {
    # Detach causes the device to work over-the-network-only after the
    # underlying disk fails. Detach is not default for historical reasons, but is
    # recommended by the docs.
    # However, the Debian defaults in drbd.conf suggest the machine will reboot in that event...
    on-io-error detach;
 
    # LVM doesn't support barriers, so disabling it. It will revert to flush. Check wo: in /proc/drbd. If you don't disable it, you get IO errors.
    no-disk-barrier;
  }
 
  on top
  {
    disk /dev/universe/lvtest;
    address 192.168.2.6:7789;
  }
 
  on bottom
  {
    disk /dev/universe/lvtest;
    address 192.168.2.7:7790;
  }
}

Copy all config files to the slave machine (and write an rsync-script for it…).

I learned that Linux 3.1 now has write barriers enabled by default for ext3 (they already were for ext4). This causes bugs and IO errors with xen-blkfront, so that needs to be disabled:

# grep barrier /etc/fstab
/dev/xvda2 / ext3 barrier=0 0 1

I’ll see about finding out if there are bug reports and file them if necessary.

The drbd data is going to be written on the actual LV, so on the primary node, we need to make space (you can also grow the LV):

e2fsck -f /dev/universe/lvtest
resize2fs /dev/universe/lvtest 500M # or however big that's a tad smaller than the actual LV.
drbdadm create-md r0
drbdadm up r0

On the secondary node, make the device as well:

drbdadm create-md r0
drbdadm up r0

Then we can start syncing and re-grow it. On the primary:

drbdadm -- --overwrite-data-of-peer primary r0 # the -- is necessary because of weird option handling by drbdadm.
resize2fs /dev/drbd1

The logical volume has been converted from ext3 to drbd:

# mount /dev/universe/lvtest /mnt/temp
mount: unknown filesystem type 'drbd'

Then, it is recommended you create /etc/modprobe.d/drbd.conf with:

options drbd disable_sendpage=1

I don’t know what it does, but it’s recommended by the DRBD devices docs when you put Xen domains on DRBD devices.

In Xen, you can configure the disk device of a VM like this (actually, I learned that this doesn’t work with pygrub):

disk = [ 'drbd:resource,xvda,w' ]

Drbd has installed the necessary scripts in /etc/xen/scripts to support this. Xen will now automatically promote a drbd device to primary when you start a VM.

Bewarned: because of that, don’t put the VM in the /etc/xen/auto dir on the fallback node, otherwise whichever machine is faster will start the VM, preventing the other machine from starting it (because you can’t have two primaries).

Then, I noticed that Debian arranges it’s boot process erroneously, starting xemdomains before drbd. I comment on an old bug.

You can fix it by adding xendomains to the following lines in /etc/init.d/drbd:

# X-Start-Before: heartbeat corosync xendomains
# X-Stop-After:   heartbeat corosync xendomains

Mdadm (software RAID) schedules monthly checks of your array. You can do that for DRBD too). You do that on the primary node with a cronjob in /etc/cron.d/:

42 0 * * 0    root    /sbin/drbdadm verify all

One last thing: the docs state that when you perform a verify and it detects an out-of-sync device, all you have to do is disconnect and connect. That didn’t work for me. Instead, I ran the following on the secondary node (the one I had destroyed with dd) to initiate a resync:

drbdadm invalidate r0