A recent example of the usefulness of Vagrant is the new packaging and testing work undertaken in the Sensu project. The project set out to build a new set of native OS packages with a goal of making Sensu easy to deploy on a variety of platforms and without a lot of the friction that sometimes accompanies Ruby apps. As part of the packaging effort we needed a simple mechanism to build native packages on the relevant platforms, ie: .deb's on debian and .rpm on redhat/centos.

We ended up using a combination of Vagrant and some homegrown tools such as Bunchr.

You can see the work in these 2 repos:

• http://github.com/sensu/sensu-build
• http://github.com/joemiller/sensu-tests

Both codebases contain a para-vagrant.sh script that is used in place of the normal vagrant up to kick off parallel provisioning tasks. sensu-tests is the more interesting example as it runs a set of rspec tests against Sensu across 14 VM's and this will likely grow to encompass other OS's in the future. The tests are executed as Vagrant provisioners (a combo of Chef and shell to call rspec).

The simplest way to use multi-VM's with Vagrant is the typical vagrant up. However, this will boot and run the provisioning tasks sequentially on each VM. With 14 VM's to test, this process can take a long time.

Can we speed this up? Yes. In fact we were able to reduce the time taken to run the sensu-build tasks from about 33 minutes to 12 minutes, and reduced sensu-tasks from almost 90 minutes to 15!

Here was the first attempt at a parallelization script:

#!/bin/sh
 
# concurrency is hard, let's have a beer
 
MAX_PROCS=4
 
parallel_provision() {
    while read box; do
        echo "Provisioning '$box'. Output will be in: $box.out.txt" 1>&2
        echo $box
    done | xargs -P $MAX_PROCS -I"BOXNAME" \
        sh -c 'vagrant provision BOXNAME >BOXNAME.out.txt 2>&1 || echo "Error Occurred: BOXNAME"'
}
 
## -- main -- ##
 
# start boxes sequentially to avoid vbox explosions
vagrant up --no-provision
 
# but run provision tasks in parallel
cat <<EOF | parallel_provision
centos_5_64
centos_5_32
ubuntu_1004_32
ubuntu_1004_64
EOF

There are 2 steps to the process:

• Sequentially boot each VM listed in the Vagrantfile, but without running provisioners (vagrant up --no-provision). The reason we do the bootups sequentially is to avoid potential kernel panics that VirtualBox is prone to do, at least on OSX.

• Feed a list of VM's into xargs to execute $MAX_PROCS vagrant provision $BOXNAME processes in parallel. xargs will manage the concurrency for us.

The final script we used with the sensu-tests repo is a little different than the above example, which can be viewed here: para-vagrant.sh. This script reduced the time it takes to run the full test suite from over 90 minutes to 15 minutes. The differences in this final version of the script:

• uses GNU parallel instead of xargs for better handling/grouping of the output from the vagrant provision processes.

• reads the list of VM's from a JSON file (the Vagrantfile also uses this JSON file). To add/remove new VM's, you only need to edit one file now.

Future?

It should be possible to make the process go even faster by parallelizing the bootup phase, but I have not tried this yet because I'm worried about VirtualBox stability.

Also, I'm sure there's a more polished script/utility that can be written to parallelize any multi-VM Vagrantfile.

Alternatives?

As @vvuksan correctly points out, doing such short tasks with Vagrant is probably always going to be a little slow due to the overhead in spinning up VM's, so a possible alternative maybe LXC. There's already a Vagrant-like tool for LXC called toft. LXC should work fine for workflows that include only various Linux distributions, but won't help if you need to test other OS's like FreeBSD or Solaris.

For example, consider you have a standard set of handlers that you assign to most of your checks — pagerduty, irc, campfire. Now, suppose you want to add the GELF (graylog2) handler to all of your monitors as well. If each of your checks is defined as such:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["pagerduty", "irc", "campfire"]
    }
  }
}

… you would need to modify every check's "handlers" attribute to include your new "gelf" handler. If you have a lot of checks this can be a little bit of a burden.

Handler sets save a lot of time in this situation. If you instead define your checks like this:

{
  "checks": {
    "all_disk_check": {
      "notification": "Diskspace Too Low",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins  check_disk -w 25% -c 15% /",
      "subscribers": [ "all" ],
      "interval": 60,
      "handlers": ["default"]
    }
  }
}

And then define your handlers on your Sensu server like the following. Notice we have converted the 'default' handler to a set.

"handlers": {
      "default": {
        "type": "set",
        "handlers": ["pagerduty", "irc", "campfire", "gelf"]
      },
      "pagerduty": {
        "type": "pipe", 
        "command": "/etc/sensu/handlers/pagerduty"
      },
      "irc": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/irc"
      },
      "campfire": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/campfire"
      },
      "gelf": {
        "type": "pipe",
        "command": "/etc/sensu/handlers/gelf.rb"
      }   
    },

You only need to modify the "default" handler set to add the new "gelf" handler and no need to change any of your check definitions.

It's a simple feature, but it can save a bunch of time. You could also setup a handler set specifically for metrics and another set for notifications. In the case of metrics, you could easily ship metrics to multiple systems – Graphite, Librato, Cube, etc. And adding a new system would be as simple as creating the handler and adding it to the handler set. Happy Sensu'ing.

Available on github and rubygems.org: https://github.com/joemiller/amqpcat

Installation is easy:

sudo gem install amqpcat --no-rdoc --no-ri

1:1 Messaging demonstration

Publish a message:

$ echo "hello there" | amqpcat amqp://guest:guest@rabbitmq -t direct -n test.direct --publisher

Receive the message:

$ amqpcat amqp://guest:guest@rabbitmq -t direct -n test.direct --consumer
hello there

Pretty simple. There are various other options, run with -h to see them all. You can easily load-balance amongst multiple consumers by starting multiple consumers with the same url and options. See the test_direct.sh example for a demo of this.

1:N / PubSub Messaging

Start 2 (or more) consumers:

$ amqpcat amqp://rabbitmq -t fanout -n test.fanout -s "Consumer-1: " --consumer &
$ amqpcat amqp://rabbitmq -t fanout -n test.fanout -s "Consumer-2: " --consumer &

Publish a message:

$ echo "hello there" | amqpcat amqp://guest:guest@rabbitmq -t fanout -n test.fanout --publisher

You should see both consumers receive the message, ie:

Consumer-1: hello there
Consumer-2: hello there

See test_fanout.sh for a similar example.

SSL support

SSL support (server verification, and client certs) has been written into the tool but I have not been able to test it to a working state yet.

Updated October 16, 2012: Removed “passive”:”true” from the graphite amqp handler definition in Sensu. This is too brittle. Sensu will fail to start unless graphite has started first and created the exchange on the RabbitMQ server. By matching the “durable”:”true” setting that graphite expects, then we can start either service in any order.

Updated October 16, 2012: Updated graphite amqp handler definition to use the new “mutator”‘s in Sensu 0.9.7. See the for details on backwards-incompatible changes as Sensu moves towards a 1.0.0 release.

It's been pretty exciting to see the number of folks getting involved with Sensu lately, as judging by the increased activity on the #sensu channel on Freenode. One of the most common questions is how to integrate Sensu and Graphite. In this article I'll cover two approaches for pushing metrics from Sensu to Graphite.

Remember: think of Sensu as the "monitoring router". While we are going to show how to push metrics to Graphite, it is just as easy to push metrics to any other system – Librato, Cube, OpenTSDB, etc. In fact, it would not be difficult at all to push metrics to multiple graphing backends in a fanout manner.

Install vmstat-metrics plugin

For this example, we're going to use the vmstat-metrics plugin which can be found in the sensu-community-plugins repository on github. The plugins in this repo are meant to be cherry-picked, so let's just grab the one we're interested in:

cd /etc/sensu/plugins/
sudo wget https://raw.github.com/sonian/sensu-community-plugins/master/plugins/system/vmstat-metrics.rb
sudo chmod +x vmstat-metrics.rb

Most, but not all, of the plugins in the sensu-community-plugins repository use helper classes from the sensu-plugin, so we'll need to install that gem on the nodes that will be running the vmstat-metrics plugin.

sudo gem install sensu-plugin --no-rdoc --no-ri

Open up vmstat-metrics.rb and we see that it inherits a lot of plumbing from the Sensu::Plugin::Metric::CLI::Graphite class. Keep this in mind when writing your own metrics-gathering plugins, it can save you some time.

Let's run the plugin manually so we can see the output.

./vmstat-metrics.rb 
stats.swap.in   0   1328153991
stats.swap.out  0   1328153991
stats.memory.active 122160  1328153991
stats.memory.swap_used  8   1328153991
stats.memory.free   48556   1328153991
stats.memory.inactive   73704   1328153991
stats.cpu.waiting   1   1328153991
stats.cpu.idle  95  1328153991
stats.cpu.system    4   1328153991
...

We will want to customize the path before sending this data to Graphite, including adding a hostname to differentiate it from other hosts. This can be done with the –scheme switch:

./vmstats-metrics.rb --scheme stats.`hostname -s`
stats.host01.swap.in    0   1328155423
stats.host01.swap.out   0   1328155423
stats.host01.memory.active  122512  1328155423
stats.host01.memory.swap_used   8   1328155423
stats.host01.memory.free    43856   1328155423
stats.host01.memory.inactive    75120   1328155423
stats.host01.cpu.waiting    1   1328155423
stats.host01.cpu.idle   95  1328155423
stats.host01.cpu.system 4   1328155423
...

If you're familiar with Graphite, this should look familiar. This data fits Graphite's "stat.name value timestamp" format and we can be feed it directly into Graphite via a couple different methods.

Create a check .json

Let's push out a check definition to our nodes running sensu-client and sensu-server nodes. Don't forget to install the plugin as well as the sensu-plugin gem.

File: /etc/sensu/conf.d/metrics_vmstat.json:

{
  "checks": {
    "vmstat_metrics": {
      "type": "metric",
      "handlers": ["graphite"], 
      "command": "/etc/sensu/plugins/vmstat-metrics.rb --scheme stats.:::name:::",
      "interval": 60,
          "subscribers": [ "webservers" ]
    }
  }  
}

There are two new items in this check definition that we haven't covered yet:

• The first is a new attribute: "type": "metric". This is critical. It tells the sensu-server to send the output of every invocation of this check to the specified handlers. Normally, only checks that return a non-zero exit status indicating a failed check will be passed onto handlers. For metrics, however, we always want to send the output to the handlers.

• The second is the use of a "custom variable" in the command: :::name:::. Sensu will replace this at execution time with the name attribute from the client section of the Sensu config. There's a lot of other stuff we can do with custom variables which will be covered in future blogs.

Next, we need to create a handler to do something with this data. Since we're using Graphite in this example, we'll explore two methods for getting this data into graphite: direct TCP and AMQP.

Method 1 – Direct TCP (via netcat) handler

The first method will be to simply send this data to Graphite via TCP socket using netcat. This is a very simple approach and should work on most boxes (with netcat installed) and doesn't require configuring Graphite to use AMQP.

Fetch graphite_tcp.rb from the sensu-community-plugins repo and copy to /etc/sensu/handlers.

Next create /etc/sensu/conf.d/graphite_tcp.json config file:

{
  "graphite": {
    "server":"graphite.example.com",
    "port":"2003"
  }
}

Create /etc/sensu/conf.d/handler_graphite.json:

{
  "handlers": {
    "graphite": {
      "type": "pipe",
      "command": "/etc/sensu/handlers/graphite_tcp.rb"
    }
  }
}

Restart sensu-server and you should start seeing vmstat metrics show up in Graphite.

There is a downside to this approach, however, and that is scalability. For each metric that is received, sensu-server will fork and execute this handler. With many nodes, many checks generating metrics, and a low interval, this could quickly add up to dozens or hundreds of short-lived processes being forked on the sensu-server.

Method 2 – integrated AMQP handler

Remember that I keep calling Sensu "the monitoring router"? Because Sensu and Graphite both speak the AMQP messaging protocol, we can configure sensu-server to take the check output directly off of a rabbitmq queue and copy it to a new queue in a format for Graphite. This approach should be very fast and very scalable.

Configure Graphite's carbon-cache for AMQP:

/opt/graphite/conf/carbon.conf:

ENABLE_AMQP = True
# AMQP_VERBOSE = True
AMQP_HOST = sensu.example.com
AMQP_PORT = 5672
AMQP_VHOST = /sensu
AMQP_USER = sensu
AMQP_PASSWORD = mypass
AMQP_EXCHANGE = metrics
AMQP_METRIC_NAME_IN_BODY = True

An important part of this config is AMQP_METRIC_NAME_IN_BODY = True which means Graphite will determine the metric name from the contents of the message rather than via the routing key. See here for more info.

Next, we configure a handler on the sensu-server that will send metrics to the queue Graphite is listening on:

Edit /etc/sensu/conf.d/handler_graphite.json:

{
  "handlers": {
    "graphite": {
      "type": "amqp",
      "exchange": {
        "type": "topic",
        "name": "metrics",
        "durable": "true"
      },
      "mutator": "only_check_output"
    }
  }
}

A few things to note here:

• "type": "amqp" : We're telling sensu-server that this handler will re-route the check output to an AMQP exchange.
• "durable": "true" : Graphite (carbon-cache) creates the exchange as a durable exchange. We need to match this setting otherwise we will get an error from Rabbit and the connection will fail.
• "mutator": "only_check_output" : This instructs sensu-server to only send the raw output (from stdout) returned by the check to the AMQP exchange. By default, sensu-server would send the entire JSON doc with other metadata, but Graphite wouldn't know what to do with that data.

Debug tips:

For debugging the Graphite side of things, it can be helpful to enable AMQP_VERBOSE = True in carbon.conf. With this enabled, the following will be visible in listener.log when a metric is successfully retrieved from rabbitmq by Graphite:

02/02/2012 10:11:11 :: Message received: Method(name=deliver, id=60) ('graphite_consumer', 8, False, 'metrics', '') content = <Content instance: body='test.sensu-server.swap.in\t0\t1328195471\ntest.sensu-server.swap.out\t0\t1328195471\ntest.sensu-server.memory.active\t126856\t1328195471\ntest.sensu-server.memory.swap_used\t8\t1328195471\ntest.sensu-server.memory.free\t52528\t1328195471\ntest.sensu-server.memory.inactive\t57952\t1328195471\ntest.sensu-server.cpu.waiting\t1\t1328195471\ntest.sensu-server.cpu.idle\t95\t1328195471\ntest.sensu-server.cpu.system\t4\t1328195471\ntest.sensu-server.cpu.user\t0\t1328195471\ntest.sensu-server.system.interrupts_per_second\t89\t1328195471\ntest.sensu-server.system.context_switches_per_second\t476\t1328195471\ntest.sensu-server.io.received\t8\t1328195471\ntest.sensu-server.io.sent\t17\t1328195471\ntest.sensu-server.procs.waiting\t4\t1328195471\ntest.sensu-server.procs.uninterruptible\t0\t1328195471\n', children=[], properties={'priority': 0, 'content type': 'application/octet-stream', 'delivery mode': 1}>
02/02/2012 10:11:11 :: Metric posted: test.sensu-server.swap.in 0 1328195471
02/02/2012 10:11:11 :: Metric posted: test.sensu-server.swap.out 0 1328195471
02/02/2012 10:11:11 :: Metric posted: test.sensu-server.memory.active 126856 1328195471
...

Install the check_http nagios plugin

Following on the previous article, we’ll be doing this demo based off of CentOS 5 but it should not be difficult to find a build of check_http for your distribution.

First, Make sure you have the EPEL yum repo installed on your machine. Next, let’s install the nagios-plugin-http package onto our Sensu client node(s):

sudo yum -y install nagios-plugins-http

Let’s see where the check_http binary was installed (note the arch specific path):

$ rpm -ql nagios-plugins-http
/usr/lib64/nagios/plugins/check_http

Write the Sensu check definition

Ok, now we are ready to create a check definition for Sensu. We will create this file on the nodes running sensu-client as well as sensu-server (pro tip: this part is easier if you’re using a CM tool like Chef or Puppet.)

/etc/sensu/conf.d/check_google.json:

{
  "checks": {
    "check_google": {
      "notification": "Google HTTP failed",
      "command": "PATH=$PATH:/usr/lib64/nagios/plugins:/usr/lib/nagios/plugins check_http google.com -R 'search'",
      "subscribers": [ "webservers" ],
      "interval": 60,
      "handlers": ["default", "pagerduty"]
    }
  }
}

Let’s look at each part of this check definition:

• notification: This can be thought of as a “friendly message”. It’s most useful with handlers like Twitter or Pagerduty. For example, this will be what you hear when the creepy Pagerduty.com computer voice wakes you up at 3 AM. The full output from the check is also available to handlers in the output attribute.
• command: This is where we specify the check_http command we want to run and relevant options. Notice that we are adding both /usr/lib64/... and /usr/lib/... to the PATH. This is not required, but it is a nice way to make this check work on both i386 and x86_64 platforms for the EPEL-supplied version of check_http.
• subscribers: This is where we specify which sensu-client nodes should run this check. Recall that we defined "subscriptions": "webservers" in our /etc/sensu/conf.d/client.json in the previous article.
• interval: How often this check should be executed.
• handlers: The handlers that should receive the output from this plugin. The handlers will be executed on the sensu-server node.

Turn a profit

After we’ve created the json file on the clients and servers, we need to restart the sensu-client and sensu-server services so they pickup the new .json file.

In a couple minutes, you should see the following in /var/log/sensu/sensu-server.log:

I, [2012-01-23T19:14:22.218301 #7397]  INFO -- : [publisher] -- publishing check request -- check_google -- webservers {"message":"[publisher] -- publishing check request -- check_google -- webservers","level":"info","timestamp":"2012-01-23T19:14:22.   %6N-0700"}

And on the client, in /var/log/sensu/sensu-client.log:

I, [2012-01-23T19:15:22.224437 #18787]  INFO -- : [subscribe] -- received check request -- check_google {"message":"[subscribe] -- received check request -- check_google","level":"info","timestamp":"2012-01-23T19:15:22.   %6N-0700"}
I, [2012-01-23T19:15:22.348884 #18787]  INFO -- : [result] -- publishing check result -- check_google -- 0 -- HTTP OK: HTTP/1.0 200 OK - 11294 bytes in 0.099 second response time |time=0.099463s;;;0.000000 size=11294B;;;0

There we go. We have implemented a Sensu check using a Nagios plugin. There are quite a few Nagios plugins out there and we should be able to use most (all?) of them with Sensu. Go forth and profit.

For questions on using Sensu, stop by the #sensu channel on Freenode.

(5/15/2012) NOTE: This guide has been superseded by the official ‘Install Guide‘ doc on the Sensu wiki. The new process utilizes the simpler Omnibus-style Sensu packages and covers installation on Debian/Ubuntu platforms as well. Please use this guide instead of the instructions below.

I’m excited about Sensu, a new open source monitoring framework, and I’d like to help others get started with it as well. So, after observing the frequent questions from new visitors to #sensu on Freenode I thought perhaps the best way to do that is to write a blog article to help folks get started. If you still have questions after reading this, feel free to come by #sensu on Freenode.

In this article I will provide a brief overview of Sensu with some background, walk through a client and server install, and then I will show you how to add a check and a handler. This should lay the groundwork for future articles with more examples on how to get the most value out of Sensu in your infrastructure.

Before we start, I owe a huge thanks to @jeremy_carroll for the many hours of work he put into building RPM’s for Sensu. His work on packaging will undoubtedly save many folks quite a bit of time.

What is Sensu?

Sensu is the creation of @portertech and his colleagues at sonian.com. They have graciously open-sourced the project and made it available to all of us searching for a modern monitoring platform (or anyone searching for an alternative to Nagios.)

Sensu is often described as the “monitoring router”. Put another way, Sensu connects the output from “check” scripts run across many nodes with “handler” scripts run on Sensu servers. Messages are passed via RabbitMQ. Checks are used, for example, to determine if Apache is up or down. Checks can also be used to collect metrics such as MySQL statistics. The output of checks is routed to one or more handlers. Handlers determine what to do with the results of checks. Handlers currently exist for sending alerts to Pagerduty, IRC, Twitter, etc. Handlers can also feed metrics into Graphite, Librato, etc. Writing checks and handlers is quite simple and can be done in any language.

Key details:

• Ruby 1.8.7+ (EventMachine, Sinatra, AMQP), RabbitMQ, Redis
• Excellent test coverage with continuous integration (travis-ci)
• Messaging oriented architecture. Messages are JSON objects.
• Ability to re-use existing Nagios plugins
• Plugins and handlers (think notifications) can be written in any language
• Supports sending metrics into various backends (Graphite, Librato, etc)
• Designed with modern configuration management systems such as Chef or Puppet in mind
• Designed for cloud environments
• Lightweight, less than 1200 lines of code

Components

Sensu is made up of several small components. I won’t go into too much detail about each of them here, as their purpose will become obvious when we get started, so here a few short descriptions:

sensu-server

The server initiates checks on clients, receives the output of the checks feeds them to handlers. (As of version 0.9.2, clients can also execute checks that the server doesn’t know about and the server will still process their results, more on these ‘standalone checks’ in a future article.)

Sensu-server relies on a Redis instance to keep persistent data. It also relies heavily (as do most sensu components) on access to rabbitmq for passing data between itself and sensu-client nodes.

sensu-client

Run this on all of your systems that you want to monitor. Sensu-client will execute checks scripts (think check_http, check_load, etc) and return the results from these checks to sensu-server via rabbitmq.

sensu-api

A REST API that provides access to various pieces of data maintained on the sensu-server in Redis. You will typically run this on the same server as your sensu-server or Redis instance. It is mostly used by internal sensu components at this time.

sensu-dashboard

Web dashboard providing an overview of the current state of your Sensu infrastructure and the ability to perform actions, such as temporarily silencing alerts.

Installing

As you start to explore Sensu you will find that it was built from the start to be used in conjunction with a CM tool such as Chef or Puppet. However, for the purposes of this article I will walk through a simple manual install.

This article covers installation of Sensu via RPM on CentOS-5 and CentOS-6. Debian/ubuntu and derivatives are not covered in this guide, but many of the same steps will apply. At this time there are no .deb packages for the Sensu components so you will have to install Sensu from gem (ie: gem install sensu sensu-dashboard). Hopefully soon we will have native .deb packages for the Sensu components.

One thing to note when installing from gem: you may get an error when starting sensu-server or sensu-client if you do not have a “client” section defined in your config and if you don’t have at least one “check”.

You will probably want to use Sensu with Chef or Puppet soon after you get bootstrapped (of course you’re already using a modern CM tool in your infrastructure anyway, right?) There are good Chef and Puppet recipes in the github repos that can help you get going fairly quickly. There are also a few community members working on improving these pieces so should get even better over time.

Additionally, the original dev platform for Sensu was Ubuntu but work has been done to help make it a little more CentOS/RHEL-friendly. I’m going to use CentOS 5 and 6 in this article just because I’m more familiar with this platform than the debian/ubuntu family. In any case, it shouldn’t matter too much because the purpose of this article is to show you Sensu.

We will use 2 nodes, one will be our server and the other will be a simple client, with the following bits on each:

Server:

• rabbitmq
• redis
• sensu-server / sensu-client / sensu-api / sensu-dashboard

Client:

• sensu-client

Install a Sensu server node

Install rabbitmq

We will base our approach on the rabbit install guide from here: http://www.rabbitmq.com/install-rpm.html

(CentOS 5 only) We need to install both the EPEL-5 and epel-erlang yum repos. The EPEL-5 yum repo contains the older R12B version of Erlang which would work fine with rabbit except we wouldn’t have access to SSL nor the web management plugins. Thus, we’ll be installing a newer Erlang from the epel-erlang repo which provides R14B for cent5.

sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
sudo wget -O /etc/yum.repos.d/epel-erlang.repo http://repos.fedorapeople.org/repos/peter/erlang/epel-erlang.repo

(CentOS 6 only) Install the EPEL-6 yum repo which contains Erlang R14B:

sudo rpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Install Erlang:

sudo yum install erlang

Install RabbitMQ:

sudo rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo rpm -Uvh http://www.rabbitmq.com/releases/rabbitmq-server/v2.7.1/rabbitmq-server-2.7.1-1.noarch.rpm

We need to make some SSL certs for our rabbitmq server and the sensu clients. I put a simple script up on github to help with this. You’ll want to change a few things in the openssl.cnf to for your organization if you use this in production. The script will generate a few files that we’ll need throughout the guide, so keep them nearby.

git clone git://github.com/joemiller/joemiller.me-intro-to-sensu.git
cd joemiller.me-intro-to-sensu/
./ssl_certs.sh clean
./ssl_certs.sh generate

Configure RabbitMQ to use these SSL certs

mkdir /etc/rabbitmq/ssl
cp server_key.pem /etc/rabbitmq/ssl/
cp server_cert.pem /etc/rabbitmq/ssl/
cp testca/cacert.pem /etc/rabbitmq/ssl/

Create /etc/rabbitmq/rabbitmq.config:

[
  {rabbit, [
    {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                   {certfile,"/etc/rabbitmq/ssl/server_cert.pem"},
                   {keyfile,"/etc/rabbitmq/ssl/server_key.pem"},
                   {verify,verify_peer},
                   {fail_if_no_peer_cert,true}]}
  ]}
].

Install the RabbitMQ webUI management console:

rabbitmq-plugins enable rabbitmq_management

Set RabbitMQ to start on boot and start it up immediately:

sudo /sbin/chkconfig rabbitmq-server on
sudo /etc/init.d/rabbitmq-server start

Verify operation with the RabbitMQ Web UI: Username is “guest”, password is “guest” – http://<SENSU-SERVER>:55672. Protocol amqp should be bound to port 5672 and amqp/ssl on port 5671.

Finally, let’s create a /sensu vhost and a sensu user/password on our rabbit:

rabbitmqctl add_vhost /sensu
rabbitmqctl add_user sensu mypass
rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

Install redis

At this point we already have the EPEL repo installed on our server so we will install EPEL’s version of Redis. For Cent5 this will be a fairly old redis v2.0, and for Cent6 it will be v2.2. Both should work fine with Sensu.

sudo yum install redis
sudo /sbin/chkconfig redis on
sudo /etc/init.d/redis start

Install ruby 1.8.7

(CentOS 5 only) Sensu needs ruby 1.8.7+ but CentOS-5 ships with older Ruby 1.8.5. We will use the ruby 1.8.7 rpm’s from Opscode’s Chef. See this page on the Chef wiki for additional details.

sudo wget -O /etc/yum.repos.d/aegisco.repo http://rpm.aegisco.com/aegisco/el5/aegisco.repo

(CentOS 6) CentOS 6 ships with ruby 1.8.7 so we don’t need any external repos.

Install Ruby packages;

sudo yum install ruby ruby-ri ruby-rdoc ruby-shadow rubygems curl openssl-devel

Install Sensu components.

Now we are ready to install Sensu. We will install two rpms: rubygem-sensu and rubygem-sensu-dashboard. This will install four components: sensu-server, sensu-client, sensu-api, sensu-dashboard. Sensu servers use all of these and clients only use sensu-client.

We are going to use @jeremy_carroll‘s recently created yum repo to install Sensu via rpm. This repo contains sensu rpm’s for both CentOS 5 and 6. It’s awesome that Jeremy has taken the time to set this up and I hope we can use this as a basis for automating the building of rpms and debs for all releases of Sensu. Since this repo was setup pretty much as this blog was being written, it’s possible that this repo will move to a different location in the future.

(CentOS 5 only)

sudo rpm -Uvh http://yum.carrollops.com/el/5/sensu-release-1.noarch.rpm

(CentOS 6 only)

sudo rpm -Uvh http://yum.carrollops.com/el/6/sensu-release-1.noarch.rpm

We need to ignore the rubygem rpm’s that come from EPEL because they will conflict with the sensu rpm’s. Edit your /etc/yum.repos.d/epel.repo file and add the following line to the [epel] section.

exclude=rubygem-rack,rubygem-sinatra

Install and enable sensu service components:

sudo yum install rubygem-sensu rubygem-sensu-dashboard
sudo chkconfig sensu-server on
sudo chkconfig sensu-api on
sudo chkconfig sensu-client on
sudo chkconfig sensu-dashboard on

Copy the SSL client key + cert that we created earlier into /etc/sensu/ssl

cp client_key.pem client_cert.pem  /etc/sensu/ssl/

Next we need to configure sensu by editing /etc/sensu/config.json. For now we will create just enough config to start sensu. Later we will add checks and handlers. Note (for later use) that Sensu will also read json config snippets out of the /etc/sensu/conf.d directory so you can piece together a config easily using your CM tool.

{
  "rabbitmq": {
    "ssl": {
      "private_key_file": "/etc/sensu/ssl/client_key.pem",
      "cert_chain_file": "/etc/sensu/ssl/client_cert.pem"
    },
    "port": 5671,
    "host": "localhost",
    "user": "sensu",
    "password": "mypass",
    "vhost": "/sensu"
  },
  "redis": {
    "host": "localhost",
    "port": 6379
  },
  "api": {
    "host": "localhost",
    "port": 4567
  },
  "dashboard": {
    "host": "localhost",
    "port": 8080,
    "user": "admin",
    "password": "secret"
  }
}

Configure /etc/sensu/conf.d/client.json for the current node:

{
  "client": {
    "name": "sensu-server.dom.tld",
    "address": "10.0.0.1",
    "subscriptions": [ "test" ]
  }
}

(CentOS 5 x86_64 only) With the current set of rpm’s we need to add a path to the GEM_PATH in order to find a couple of the rubygems we installed. Run the following:

echo "export GEM_PATH=\$GEM_PATH:/usr/lib64/ruby/gems/1.8" > /etc/profile.d/gem_path.sh
. /etc/profile.d/gem_path.sh

Now let’s try to start the Sensu components:

sudo /etc/init.d/sensu-server start
sudo /etc/init.d/sensu-api start
sudo /etc/init.d/sensu-client start    
sudo /etc/init.d/sensu-dashboard start

If all goes well, the 4 processes mentioned above will be running and the dashboard will be accessible on http://<SENSU SERVER>:8080. Log files are available in /var/log/sensu in case anything is wrong.

sensu    14249  0.0  3.4  92924 17648 ?        S    02:56   0:00 ruby /usr/bin/sensu-server ...
sensu    14404  0.0  4.1 102172 20884 ?        S    03:05   0:00 ruby /usr/bin/sensu-api ...
sensu    14425  0.0  3.7 104860 19292 ?        Sl   03:06   0:00 ruby /usr/bin/sensu-client ...
sensu    14553  0.4  7.0 140544 35932 ?        Sl   03:07   0:00 ruby /usr/bin/sensu-dashboard ...

If you see an openssl error like the one below on CentOS-5, it’s likely because you’re on a x86_64 box but some ruby-libs or ruby-devel 1.8.5 rpm’s from the base repo were accidentally installed, remove them.

Starting sensu-client: /usr/lib/ruby/1.8/openssl/cipher.rb:22: Cipher is not a module (TypeError)
    from /usr/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:36:in `gem_original_require'

Installing a sensu client node

Installing Sensu on a client node is similar to installing the server. We will need to install ruby, the Sensu rpm, and then configure Sensu. There are only a few small differences which are detailed below.

Install ruby 1.8.7

Follow the same steps from the Install ruby 1.8.7 section we used to build our server.

Install sensu-client

Follow the steps from Install Sensu components section we used to build our server, with the following differences:

• Only install rubygem-sensu and skip rubygem-sensu-dashboard
• You will only need the rabbitmq section in /etc/sensu/config.json file. Make sure you point it to your rabbit server.
• Only enable and start the sensu-client service, ie: chkconfig sensu-client on and /etc/init.d/sensu-client start.

The client will log to /var/log/sensu/sensu-client.log.

Add a check

Now that we’ve installed a Sensu server and a client, let’s create a simple check so we can begin to see how the pieces fit together. We’re going to write a check to determine if crond is running. We’ll be using the check-procs.rb script from the sensu-community-plugins repo.

Most of the plugins in the sensu-community-plugins repo rely on the helper classes from the sensu-plugins gem, so let’s install that first. You may need to install gcc in order to build the json gem dependency. Note that we are installing this from a gem because there is not an rpm available yet.

sudo gem install sensu-plugin --no-rdoc --no-ri

Next, we’re going to grab the check-procs.rb script directly from github and install it into /etc/sensu/plugins. You don’t have to install checks into this directory, but it’s convenient.

wget -O /etc/sensu/plugins/check-procs.rb https://raw.github.com/sonian/sensu-community-plugins/master/plugins/processes/check-procs.rb
chmod 755 /etc/sensu/plugins/check-procs.rb

Let’s create a new json file to hold our check definition in /etc/sensu/conf.d/check_cron.json. Put this file on both the Sensu server and client.

(NOTE: as of sensu 0.9.2 ‘standalone’ checks were added which only need to be configured on the client-side. We will cover standalone checks in future articles.)

{
  "checks": {
    "cron_check": {
      "handler": "default",
      "command": "/etc/sensu/plugins/check-procs.rb -p crond -C 1 ",
      "interval": 60,
      "subscribers": [ "webservers" ]
    }
  }
}

Next, we need to tell our client node to listen to subscribe to the webservers queue. The Sensu server will publish a request every 60 seconds on the webservers queue and any client registered to this queue will execute checks that have been registered to this queue. Edit the /etc/sensu/conf.d/client.json file on the client:

{
  "client": {
    "name": "sensu-client.domain.tld",
    "address": "127.0.0.1",
    "subscriptions": [ "test", "webservers" ]
  }
}

Finally, restart sensu on the client and server nodes.

After a few minutes we should see the following in the /var/log/sensu/sensu-client.log on the client:

I, [2012-01-18T21:17:07.561000 #12984]  INFO -- : [subscribe] -- received check request -- cron_check {"message":"[subscribe] -- received check request -- cron_check","level":"info","timestamp":"2012-01-18T21:17:07.   %6N-0700"}

And on the server we should see the following in /var/log/sensu/sensu-server.log:

I, [2012-01-18T21:18:07.559666 #30970]  INFO -- : [publisher] -- publishing check request -- cron_check -- webservers {"message":"[publisher] -- publishing check request -- cron_check -- webservers","level":"info","timestamp":"2012-01-18T21:18:07.   %6N-0700"}
I, [2012-01-18T21:25:07.745012 #30970]  INFO -- : [result] -- received result -- sensu-client.domain.tld -- cron_check -- 0 -- CheckProcs OK: Found 1 matching processes; cmd /crond/

Next, let’s see if we can raise an alert.

/etc/init.d/crond stop

After about a minute we should see an alert on the sensu-dashboard: http://<SERVER IP>:8080

Add a handler

Now that we have created our first check we are ready to hook it up to a handler. Out of the box Sensu ships with a ‘default’ handler which does nothing more than parse the JSON its fed via STDIN and spits back to STDOUT.

There is a growing list of handlers available in the sensu-community-plugins repo, including Pagerduty, IRC, Campfire, etc.

Let’s create a simple handler that simply sends the raw check output to ourselves via email.

The most common handler type is “pipe” which tells Sensu to shell out and run the specified command. We’ll cover more handler types in the future. On the server nodes, we will define our ‘email’ handler in /etc/sensu/conf.d/handler_email.json.

{
  "handlers": {
    "email": {
      "type": "pipe",
      "command": "mail -s 'sensu alert' your@address"
    }
  }
}

On the sensu-server and sensu-client nodes we’ll also need to update our check definition and connect it to the new handler, edit the /etc/sensu/conf.d/check_cron.json files and modify the “handlers” attribute:

{
  "checks": {
    "cron_check": {
      "handlers": ["default", "email"],
 ...

Restart sensu-client and sensu-server on the nodes and then stop the crond daemon again. In a few minutes we should get an email from sensu with the subject “sensu alert” and a bag full of JSON data.

This isn’t the most useful handler but it illustrates the concepts of checks and handlers and how they work together. At this point we now have a working sensu-client and sensu-server to start experimenting further. In the future we’ll cover more examples of checks, handlers, metrics, etc.

If you have further questions please visit #sensu on IRC Freenode.

Update (3/2/2012): Olivier Le Cam (@olecam) posted some Debian-specific instructions that accompany this process which also include instructions on building .deb packages for Sensu.

More Sensu articles: sensu

For example, in this diagram we have plotted the load average from a group of servers. Blue vertical lines mark points in time when puppet modified a resource on a host in the group. We can see that immediately following a puppet change the load spiked on one of the servers.

Code available on github:

• https://github.com/joemiller/puppet-graphite_event

As would be expected there are statsd clients in many languages. But, there are also many implementations of the statsd server. This is nice because each organization can pick the one that best fits them. For example, a python shop might prefer to deploy a python based statsd instead of Etsy’s original node.js implementation. Also, there are some statsd implementations that diverge from the original design and provide additional features.

I could not find a single resource that listed all of the different implementations, so I figured I would try to start one here.

• Etsy’s statsd: node.js. The Original
• petef-statsd: ruby. Supports AMQP.
• statsd_rb: ruby.
• quasor/statsd: ruby. can send data to graphite or mongoDB
• py-statsd: python (including python client code).
• zbx-statsd: python, based on py-statsd. Sends data to Zabbix instead of graphite.
• statsd.scala: scala. Sends data to Ganglia instead of Graphite. Different messaging protocol, uses JSON.
• txStatsD: python + twisted, from the folks @ Canonical
• statsd-librato: node.js. Fork of etsy’s statsd for sending data to Librato instead of graphite from the folks @ Engine Yard.
• estatsd: erlang. From the folks @ Opscode
• metricsd: scala. Should be drop-in compatible with etsy’s statsd, but with support for additional metric types (eg: meter, gauge, histogram)
• statsd-c: C. compatible with original etsy statsd
• statsd (librato): node.js.  Librato’s officially maintained fork of statsd based on the changes from Engine Yard. Supports multiple graphing services including Librato Metrics
• bucky: python. A unique spin on statsd that supports collecting data from statsd clients, collectd, and metricsd, with output to graphite. The ability to translate collectd plugin names to be more graphite-friendly is very compelling.
• clj-statsd-svr: Clojure.
• statsite: C. Statsite is designed to be both highly performant, and very flexible, using libev to be extremely fast.

Deprecated:

• statsite: python. Replaced by a new implementation in C, see above.

Please leave a comment if you have an implementation that should be listed here. Feedback on any of the above implementations would be helpful too.

Then, today I noticed that Apple now includes a new prefPane in Xcode 4.1 and Lion called “Network Link Conditioner” that simplifies all of this, and even includes a few profiles to get you started (eg: “Wifi, Average case”, “3G, Lossy Network”.) Pretty cool feature. Especially useful for iOS developers. Screenshot below.

• Install: find and run /Developer/Applications/Utilities/Network Link Conditioner/Network Link Conditioner.prefPane

It’s not available on Maven Central or via Grape at this time, which will be a future learning exercise. In the meantime you can download a pre-built .jar or the source code from github:

• https://github.com/joemiller/groovy-statsdclient

For more background on Statsd, check out this blog article from Etsy:  Measure Anything, Measure Everything