Using Sensu for System monitoring, incident management and alerting

So it's roughly 3 weeks ago that I wrote that I was doing some research on system monitoring and incident management system for the infrastructure I manage. Monitoring is a big part of DevOps which am particularly enthusiastic about.

Among the several options I came across, I zeroed down on Sensu. I previously developed a custom monitoring system for our organization, but it wasn't that scalable and I was not willing to put resources in maintaining the system.

Sensu on the other hand is a fully blown monitoring platform which borrows its design and implementation inspiration from decade-only industry old-timers such as Nagios.

I have never really used Nagios in production, so for me getting on Sensu which by way works with some Nagios proven monitoring plugins was a no-brainer.
Sensu is also so flexible, I could reuse some of the python and bash scripts from my old custom monitoring system.

sensu web interface uchiwa

Sensu is written in Ruby, but it by no means restricts you to write your own custom monitoring plugins in Ruby although doing so enables you to take advantage of the system's internal libraries. I know a bit of Ruby, so at least I was able to write an SMS alert plugin for alerting me of critical incidents.

Installing Sensu

Installing Sensu might be a bit challenging. It has a modular design, so it works with different independent components which you have to install separately. So let's look at them;

  • RabbitMQ as a message transport bus
  • Redis server for preserving state
  • Sensu server which schedules monitoring checks
  • Sensu client/agent which executes checks and reports them to the server
  • Sensu API which provides an interface for third party apps to interact with Sensu
  • Uchiwa which provides a web interface for all checks, clients and other functionality.

It's important that you understand what each of these component do and how they interact with each other. Here's a gif from https://sensuapp.org that shows how the pieces work together.
sensu-gig

Installing Sensu on monitoring server
To install Sensu on Ubuntu 14.04 or 16.04 LTS server, you have to install rabbitmq, redis and sensu, uchiwa package. You can do that from the repos which I find the easiest. For your specific distro instructions, checkout the Sensu documentation for details.

On the monitoring server, the most important configuration file is /etc/sensu/config.json which defines how to connect to rabbitmq, redis and the sensu api.

Also if you wish to access the web interface called Uchiwa, you have to add your configures to /etc/sensu/uchiwa.json file.

Optionally, if your sensu monitoring server also has some services you wish to monitor, then you have define it as a client with the /etc/sensu/conf.d/client.json.

Finally you can optionally install nagios monitoring plugins which have most the common monitoring checks you can use out of the box.

All configuration settings are written in json format.

I wrote a bash script that nicely installs everything including the config files below;

#!/bin/bash
#works for installing sensu server on ubuntu 14.04/16.04 LTS 
#installs rabbitmq, redis, sensu(server, client, api) and Uchiwa
printf "installing rabbitmq and redis server \n\n"  
apt-get update  
apt-get install rabbitmq-server redis-server -y  
printf "enabling rabbitmq_management plugin \n\n"  
service rabbitmq-server restart  
printf "adding rabbitmq virtual host username: sensu pwd: secret \n\n"  
rabbitmqctl add_vhost /sensu  
rabbitmqctl add_user sensu secret  
rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"

#get the gpg key
printf "now adding sensu repo \n\n"  
wget -q https://sensu.global.ssl.fastly.net/apt/pubkey.gpg -O- | sudo apt-key add -  
CODENAME=$(lsb_release -cd | grep Codename | cut -d ":" -f2)  
echo "deb     https://sensu.global.ssl.fastly.net/apt $CODENAME main" | sudo tee /etc/apt/sources.list.d/sensu.list

#update repo and install sensu
apt-get update  
printf "installing sensu serve, client and api \n\n"  
apt-get install sensu -y  
printf "adding config.json file with rabbitmq, redis and api settings \n\n"  
cat <<EOT > /etc/sensu/config.json  
{
    "rabbitmq": {
      "host": "localhost",
      "vhost": "/sensu",
      "user": "sensu",
      "password": "secret"
    },
    "redis": {
      "host": "localhost"
    },
    "api": {
      "host": "localhost",
      "port": 4567
    }
}
EOT  
printf "adding client \n\n"  
ip=$(ip route get 1 | awk '{print $NF;exit}')  
hostname=$(hostname -f)  
cat <<EOT > /etc/sensu/conf.d/client.json  
{
        "client": {
                "name": "$hostname",
                "address": "$ip"
        }
}
EOT

printf "starting sensu server, cliend and api"

if [[ $CODENAME == "xenial" ]]; then  
    service sensu-server start
    service sensu-client start
    service sensu-api start
else  
    /etc/init.d/sensu-server start
    /etc/init.d/sensu-client start
    /etc/init.d/sensu-api start 
fi


#install uchiwa web interface
apt-get -y install uchiwa  
cat <<EOT > /etc/sensu/uchiwa.json  
{
  "sensu": [
    {
      "name": "SensuServer",
      "host": "localhost",
      "port": 4567,
      "ssl": false,
      "path": "",
      "user": "sensu",
      "pass": "secret",
      "timeout": 5
    }
  ],
  "uchiwa": {
    "host": "0.0.0.0",
    "port": 7000,
    "user": "sensu",
    "password": "secret",
    "interval": 5
  }
}

EOT  
printf "starting Uchiwa web interface on port 7000"  
#loging to http://localhost:7000 with user sensu and password secret

if [[ $CODENAME == "xenial" ]]; then  
    service uchiwa start
else  
    /etc/init.d/uchiwa start
fi  
#install nagios plugins 
printf "installing nagios plugins \n\n"  
apt-get install nagios-plugins  

Installing Sensu on client server
To add clients we wish to monitor, we simply need to install Sensu as well as Nagios plugins which do most of the common checks out the box. Then we need to define our client in the /etc/sensu/conf.d/client.json file. We shall only need to start sensu-client on the client.

In the settings, we need to tell the client how to connect to RabbitMQ server so it can send monitoring check results there where the server will pick them up. We do this in the /etc/sensu/conf.d/rabbitmq.json as well as the /etc/sensu/conf.d/transport.json which specifies transport as RabbitMQ.

Basically you need at least 3 config files minus the checks;

# ls
client.json  rabbitmq.json  transport.json  

Here's a bash script I wrote that does everything for you for Ubuntu servers.

#!/bin/bash
#this script is for installing sensu on ubuntu server 
#get the gpg key
printf "now adding sensu repo \n\n"  
wget -q https://sensu.global.ssl.fastly.net/apt/pubkey.gpg -O- | sudo apt-key add -  
CODENAME=$(lsb_release -cd | grep Codename | cut -d ":" -f2)  
echo "deb     https://sensu.global.ssl.fastly.net/apt $CODENAME main" | sudo tee /etc/apt/sources.list.d/sensu.list

#update repo and install sensu
apt-get update  
printf "installing sensu server, client and api \n\n"  
apt-get install sensu -y  
sleep 2  
printf "adding client \n\n"  
ip=$(ip route get 1 | awk '{print $NF;exit}')  
hostname=$(hostname -f)  
cat <<EOT > /etc/sensu/conf.d/client.json  
{
        "client": {
                "name": "$hostname",
                "address": "$ip"
        }
}
EOT

printf "adding rabbitmq.json file \n\n"  
cat <<EOT > /etc/sensu/conf.d/rabbitmq.json  
{
    "rabbitmq": {
      "host": "my-sensu-server.com",
      "port": 5672,
      "vhost": "/sensu",
      "user": "sensu",
      "password": "secrete"
    }
}
EOT

printf "adding transport.json file  \n\n"  
cat <<EOT > /etc/sensu/conf.d/transport.json  
{
  "transport": {
    "name": "rabbitmq",
    "reconnect_on_error": true
  }
}
EOT

printf "starting sensu client"  
if [[ $CODENAME == "xenial" ]]; then  
    service sensu-client start
else  
    /etc/init.d/sensu-client start
fi

apt-get install -y nagios-plugins  
Sensu Checks

Checks are the reasons I went for Sensu. It allows you to monitor services, applications, collect and analyze metrics. You can literally monitor anything.

Checks are commands or scripts that output data to STDOUT or STDERR and produce an exit status code to indicate a state.

The common exit status codes used are 0 for OK, 1 for WARNING, 2 for CRITICAL, and 3 or greater to indicate UNKNOWN or CUSTOM.

You can write your own or use Nagios plugins or even Sensu checks. I prefer to use Nagio and my own.

You can define your checks in /etc/sensu/conf.d directory. Let's say you wish to monitor http services and mysql server are running, add your checks in json format and save your file. As you can see, I am using nagios plugins which we installed earlier.

Notice "standalone" is set to true which means that the client is the one that will manage and schedule monitoring checks. Remember that checks can be standalone like in this case or standard. If standard, then checks are scheduled by the server and not the client. I prefer the former.
//my-checks.json

{
        "checks": {
                "http": {
                        "command": "/usr/lib/nagios/plugins/check_http -I localhost",
                        "standalone": true,
                        "interval": 60
                },
                "mysql": {
                        "command": "/usr/lib/nagios/plugins/check_mysql -u sensu -p ''",
                        "standalone": true,
                        "interval": 60
                }
       }
}

Sensu Handlers
According to the Sensu documentation, Sensu event handlers are for taking action on events (produced by check results), such as sending an email alert, creating or resolving a PagerDuty incident, or storing metrics in Graphite. There are several types of handlers: pipe, tcp, udp, transport, and set. You can read more here

For my case, am interested in mail and sms handlers. These notify admins once an incident happens.

sensu-incident-resolved

For mail handlers, you can install sensu mailer handler as a ruby gem like so
/opt/sensu/embedded/bin/gem install sensu-plugins-mailer. Notice am using the sensu embedded ruby interpreter instead of the system binary to achieve this. The scripts are installed in the /opt/sensu/embedded/bin/.

The mailer handler enables you to send emails through your own mail server or external smtp providers such as Mailgun or Amazon SES.

You can specify your handler in /etc/sensu/conf.d directory.
//my-handlers.json

{
    "mailer" :  {
        "admin_gui" : "http://localhost:7000",
        "mail_from": "sensu@davidokwii.com",
        "mail_to": "admin@example.com",
        "delivery_method": "smtp",
        "smtp_address": "mail.davidokwii.com",
        "smtp_port": "25",
        "smtp_domain": "davidokwii.com",
        "smtp_enable_starttls_auto": "true",
        "smtp_username" : "sensu@davidokwii.com",
        "smtp_password" : "secret"
    },
    "handlers": {
        "mailer": {
            "type": "pipe",
            "command": "/opt/sensu/embedded/bin/handler-mailer.rb"
        },
        "sms": {
            "type": "pipe",
            "command": "/etc/sensu/conf.d/sms_handler.rb"
        }
    }
}

Specify contacts to alert in the contacts key;
//contacts.json

{
    "contacts": {
    "david": {
        "email": {
        "to": "sensu@davidokwii.com"
        },
        "phone": "0791123456"
    }
  }
}

To add handlers to your monitoring checks, simply add the handlers key. You can specify the handlers as a list/array like so;

{
        "checks": {
                "disk_var": {
                        "command": "/usr/lib/nagios/plugins/check_disk -w 40% -c 20% /var",
                        "standalone": true,
                        "interval": 60,
                        "handlers": ["mailer","sms"],
                        "contacts": ["david", "david_gmail"],
                        "occurrences": 5,
                        "refresh": 1
                }
     }
}

So this is it so far. Over the coming weeks, I'll be adding my own custom checks especially from my previous custom monitoring system as well as writing my own handlers as I dig deep into sensu.

Image: Pixabay.com

David Okwii

David Okwii is a Systems Engineer who currently works with Africa's Talking, a pan-African company serving millions of API queries for SMS/USSD/Voice, Airtime and Mobile Payments across 6 countries.

Kampala Uganda http://www.davidokwii.com

Subscribe to David Okwii dev blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!