Monit with Slack notifications

Published on October 03, 2014

Monit

Monit is a small Open Source utility for managing and monitoring Unix systems. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.

Slack

Slack is a platform for team communication: everything in one place, instantly searchable, available wherever you go.

Monit with Slack Notifications

On a normal monit setup, monit sends out emails when an alert is triggered. Since email is getting a bit too ’90s, it’s way more intuitive to receive alerts via push notifications on your phone or chat client. Slack has many service integrations and also a Incoming Webhooks integration. This way you can receive those Monit alerts in a specific chatroom. Enable the mobile Slack application on you phone, enable push notifications on that specific channel and you’re all set.

Requirements

Make sure you have:

Monit installed (on a Unix system)
A Slack account where you can enable integrations
Slack mobile applications installed (to receive push notifications)

Create an Incoming Webhook integration

Login to your Slack account and add a new Incoming Webhook integrations. This way you’ll get an unique webhook URL like this: https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX

Notification shell script

Instead of triggering email alerts we will trigger a shell script. This shell script does a simpel http POST onto the webhook with our alert. I’ve created this in /etc/monit/slack_notifications.sh.

Execute the following commands to make the script executable:

$ sudo touch /etc/monit/slack_notifications.sh
$ sudo chmod +x /etc/monit/slack_notifications.sh

#!/bin/sh
/usr/bin/curl \
    -X POST \
    -s \
    --data-urlencode "payload={ \
        \"channel\": \"#monit\", \
        \"username\": \"Monit\", \
        \"pretext\": \"servername | $MONIT_DATE\", \
        \"color\": \"danger\", \
        \"icon_emoji\": \":ghost:\", \
        \"text\": \"$MONIT_SERVICE - $MONIT_DESCRIPTION\" \
    }" \
    https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX

Don’t forget to replace https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX with your unique webhook URL and replace #monit with the channel of your choice.

Swap emails for Slack notifications

Since there is no native solution to set this notification as a default notification for all alerts, you should set this on all your monit service configurations files. For example in /etc/monit/conf.d/nginx :

check process nginx with pidfile /var/run/nginx.pid
  start program = "/usr/sbin/service nginx start"
  stop  program = "/usr/sbin/service nginx stop"
  if failed host 127.0.0.1 port 80 then restart
  if changed pid then exec "/etc/monit/slack_notifications.sh"
  if failed host 127.0.0.1 port 80 then exec "/etc/monit/slack_notifications.sh" else if succeeded then exec "/etc/monit/slack_notifications.sh"
  if cpu is greater than 40% for 2 cycles then exec "/etc/monit/slack_notifications.sh" else if succeeded then exec "/etc/monit/slack_notifications.sh"
  if cpu > 60% for 5 cycles then restart

Don’t forget to add the else if succeeded then part, or you’ll not receive notifications when thing are fine again.

Restart your Monit, restart Nginx and you’ll see a PID changed message.

A native solution

At first I tried a native solution. There is a specific Monit branch with Slack Notifications included.

$ sudo apt-get install monit install libssl-dev bison flex checkinstall
$ sudo apt-get remove monit
$ cd /tmp
$ git clone git@bitbucket.org:pierreaubert/monit.git
$ cd monit/
$ git checkout feature/slack
$ ./bootstrap && ./configure --without-pam --enable-optimized  --prefix=/usr
$ sudo checkinstall

All you need to do is add the following line at the bottom of /etc/monit/monitrc:

set slack https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX

But this isn’t very customizable.

Note

Big up to the developer of Monit and this pull-request.

Thanks for reading

Feel free to leave a comment if you have remarks or like this post

Joeri Verdeyen