Create a command in /etc/nagios3/commands.cfg:

# This command needs this in /etc/sudoers on the target:
# nagios ALL = NOPASSWD: /usr/local/sbin/check_3ware.sh
define command {
       command_name     check_3ware
       command_line     /usr/lib/nagios/plugins/check_by_ssh -H $HOSTADDRESS$ -i /etc/nagios3/id_rsa -l nagios -t 25 -C 'sudo check_3ware.sh'
}

Run visudo and add this line:

nagios ALL = NOPASSWD: /usr/local/sbin/check_3ware.sh

Then install the script from here. Last time I did that I needed to fix bugs in it, so beware. I submitted a patch, which will be accepted I guess.

Then go download the tw_cli tool.

Then create a hostgroup for your 3ware hosts:

define hostgroup {
        hostgroup_name  3ware-machines
        alias           3Ware machines
        members         boxen
}

Then a service:

define service {
        hostgroup_name                  3ware-machines
        service_description             3Ware status
        check_command                   check_3ware
        use                             generic-service
        notification_interval           0
}

That should be it.

First add some commands to commands.cfg:

define command{
        command_name    check_remote_disk
        command_line    /usr/lib/nagios/plugins/check_by_ssh -p $ARG1$ -l nagios -t 30 -o StrictHostKeyChecking=no -i /etc/nagios3/id_rsa -H $HOSTADDRESS$ -C '/usr/lib/nagios/plugins/check_disk -w $ARG2$ -c $ARG3$ -p $ARG4$'
}
 
define command{
        command_name    check_remote_load
        command_line    /usr/lib/nagios/plugins/check_by_ssh -p $ARG1$ -l nagios -t 30 -o StrictHostKeyChecking=no -i /etc/nagios3/id_rsa -H $HOSTADDRESS$ -C '/usr/lib/nagios/plugins/check_load -w $ARG2$ -c $ARG3$'
}
 
define command{
        command_name    check_remote_swap
        command_line    /usr/lib/nagios/plugins/check_by_ssh -p $ARG1$ -l nagios -t 30 -o StrictHostKeyChecking=no -i /etc/nagios3/id_rsa -H $HOSTADDRESS$ -C '/usr/lib/nagios/plugins/check_swap -w $ARG2$ -c $ARG3$'
}

The file referenced by -i is an SSH secret key. You can create this key by running ssh-keygen and giving the right path. You can’t store anything in the nagios home dir, because that is in /var/run, which is cleared after a reboot. So, you can’t use the default key file.

The -t 30 is necessary because sometimes there is network lag, causing the service to be reported as failure. The default of 10 is not enough…

Next you need to create a user nagios on the target machine and add the .pub file of the key to the authorized_keys. Creating the user should be done like:

useradd --system --shell /bin/bash nagios

Then install the nagios plugins on the target host:

aptitude -P install nagios-plugins-basic

The nagios host needs to be able to log in with user nagios. On Debian, the user that runs nagios (also called ‘nagios’) doesn’t have a shell by default. So, give it a shell.

Then you can create a hostgroup (for example). You can do:

define hostgroup {
        hostgroup_name  nagios-enabled
        alias           Nagios enabled
        members         host1, host2
}

Then create services:

define service {
        hostgroup_name                  nagios-enabled
        service_description             Root partition space
        check_command                   check_remote_disk!22!20%!10%!/
        use                             generic-service
        notification_interval           0
}
 
define service {
        hostgroup_name                  nagios-enabled
        service_description             Swap space
        check_command                   check_remote_swap!22!50%!30%
        use                             generic-service
        notification_interval           0
}
 
define service {
        hostgroup_name                  nagios-enabled
        service_description             Load
        check_command                   check_remote_load!22!5.0,4.0,3.0!10.0,6.0,4.0
        use                             generic-service
        notification_interval           0
}

This will check load, swap and root space on all your standard nagios enabled hosts. Next you can define custom services:

define service {
        host_name                       piet
        service_description             Some partition
        check_command                   check_remote_disk!22!40%!30%!/mnt/dinklefat
        use                             generic-service
        notification_interval           0
}

On the server you need to configure a module in /etc/rsyncd.conf:

[module_name]
uid=0
gid=0
path=/path
transfer logging=no
read only=no

Then you need to rsync using a destination like user@host::module_name/

To make it work, you first need to set up a tunnel using SSH, because the rsync port (873) is not open in the firewall, most likely. So, you do this:

ssh -N -L 873:localhost:873 user@host

I wrote a script to automate this all. Perhaps I will follow-up on this post sometime and post it.

The answer (courtesy of Google and Yacin Nadji) is that Ctrl+a will target the outer screen. Each a that you add after that will go down one nesting level.

Not that I don’t still find controlling nested screen confusing, but now at least I don’t feel helpless and stuck whenever it happens.

Extra tips

1. Visible captions make it easier:

   

   GNU Screen within Screen with captions

   (If you don’t know how to configure Screen with captions, I’ve blogged about his previously.)

2. Debian Administration, a very high-quality site has an article about GNU Screen.

First, disable the user’s account by running:

usermod --lock [user]

This way, only public key authentication is allowed. Then, when adding the user’s key to the ~/.ssh/authorized_keys file, prefix it with this:

command="/usr/local/bin/svnserve -t",no-port-forwarding,no-pty,no-agent-forwarding,no-X11-forwarding

I used our wrapper script in /usr/local/bin as the command, because it sets an umask of 002 before actually running svnserve. This is necessary when using svn+ssh access.

The source for this trick explains it in more detail.

GNU Screen rehash

If you don’t know GNU screen yet and you ever find yourself using the command-line for an extended period of time, learn it. Now. I’ll name just a few advantages:

1. You can manage multiple “windows” conveniently even if you don’t have a tabbed terminal emulator, or even when you’re not within a graphic environment such as X.
2. You can “detach” and “reattach” your Screen sessions and continue exactly where you left. This is very useful if you do your work on a remote server, through SSH, for example. Never by stumped by instable connections again!
3. But, the feature which is most useful for pair programming is the ability to attach to the same session from multiple terminals.

Starting GNU Screen is very easy. Just type screen at your shell prompt (or screen -S SESSION_NAME if you want your session to have an easy-to-remember name).

Press CTRL+A followed by d to detach. Exit your terminal. Start a new terminal, type screen -r and be amazed that you have your session back. screen -r can take as an argument the name or PID of the screen, which is useful if you have more than one screen running. To get a list of current screen sessions, type screen -ls.

Inviting your observer

The first thing you have to do is to add the following command to your .screenrc file:

multiuser on

If you don’t want to enable multiuser by default, you can also choose to type the command from within Screen each time that you need it. This is done by pressing Ctrl+A, followed by : and the command.

Myself, I prefer to have the command in my .screenrc. You need to admit users explicitly anyway. Admitting full access to a user is done by typing the :acladd USERNAME command (after pressing Ctrl+A). Then the given user can join this session by starting screen with screen -x SESSION_OWNER/ where SESSION_OWNER is the driver.

Get out of my driver’s seat! (Dealing with annoying observers)

The :acladd USERNAME command will give the observer full read-write access. Maybe, if you have to deal with an observer who insists on taking the driver seat, you want to limit his or her access to read-only. This can be done just as easily: press Ctrl+A; then type :aclchg USERNAME -w "#".

Make your terminals match

Using a shared screen, it can be kind of annoying if your terminal sizes don’t match. As an observer, I fix this by asking the driver to tell me the values of the $ROWS and $COLS environment variables. If then, for example $COLS=110 and $ROWS=40, I start my xterm with this in mind: xterm -geometry 110x40

Have fun with Screen!

I’ve only touched upon some of the things you can do with screen. The manual page contains much more information—perhaps a bit too much even.

One of the things I also like to do with a shared screen session is remote system administration. If I want to perform delicate tasks as root, I find it kind of comforting if someone can stop me in time, before I do anything stupid. Besides, if you’re both root, you don’t even have to set permissions. So, it’s easy to.

Once connected, Laurelin quickly installed the Azureus BitTorrent client to be able to download some things (only legal things licensed under a liberal license, of course). Correct usage of the BitTorrent protocol implies opening up a port or two for incoming connections. After all, BitTorrent is all about sharing.

So, she needed help configuring port forwarding on the ADSL modem / NAT router (a microcom AD 2636) in Greece. She asked for this help two days ago while Wiebe was looking at my screen, because we where doing an extreme programming session on a database schema using VNC and Skype. We interrupted our work hoping that we could easily solve her problem.

I had been looking at screenshots and manuals of her router and her modem’s administrative interface earlier when they couldn’t get the connection to work. After making sure that the problem was not a configuration or software problem and identifying that there was probably a problem between their modem and their telephone exchange, they later got their problem fixed. So, now I had to dig up the manual again. But, this time I could only find some screencaps which accompanied a useless tutorial in Greek. This exemplifies why I think it is useful to keep a blog.

So, Wiebe and I did our best to guide her through a GUI we couldn’t see and eventually she succeeded in defining a few forwards. However, they didn’t do their job. According to nmap they were still filtered. She then installed RealVNC server for us and added port 5900 to the list of forwarded ports. Wiebe tried to connect using the kvnc client, but no response. By that time, we had fooled around quite a bit and after mucking about a little more, we decided to postpone the rest and return to our database design. We decided for next time to let her use PuTTY to put a hole through the NAT router.

Today was the next time. She installed PuTTY. I gave her an account on the Debian server in our local LAN (I am behind a Windows terminal myself). Now she had to set up a tunnel and connect to that account:

At this point, I still couldn’t access the tunnel between her box and the Debian machine from my Windows terminal:

$ netstat -l -n|grep 5900
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:5900          0.0.0.0:*               LISTEN

Since PuTTY was configured to let remote ports accept connections from other hosts, the problem had to be in the OpenSSH configuration on the Debian box. And indeed it was. I had to set the following option:

GatewayPorts yes

After a restart of the OpenSSH daemon, she opened a tunnel which I could access from my own machine:

$ netstat -l -n|grep 5900
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:5900            0.0.0.0:*               LISTEN

I could now control her desktop through the RealVNC viewer. Next time, I’ll need to actually do something with this connection.