Monit with Slack notifications
Monit
Monit is a small Open Source utility for managing and monitoring Unix systems. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations.
Slack
Slack is a platform for team communication: everything in one place, instantly searchable, available wherever you go.
Monit with Slack Notifications
On a normal monit setup, monit sends out emails when an alert is triggered. Since email is getting a bit too ’90s, it’s way more intuitive to receive alerts via push notifications on your phone or chat client. Slack has many service integrations and also a Incoming Webhooks integration. This way you can receive those Monit alerts in a specific chatroom. Enable the mobile Slack application on you phone, enable push notifications on that specific channel and you’re all set.
Requirements
Make sure you have:
- Monit installed (on a Unix system)
- A Slack account where you can enable integrations
- Slack mobile applications installed (to receive push notifications)
Create an Incoming Webhook integration
Login to your Slack account and add a new Incoming Webhook integrations.
This way you’ll get an unique webhook URL like this: https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX
Notification shell script
Instead of triggering email alerts we will trigger a shell script. This shell script does a simpel http POST onto the webhook with our alert.
I’ve created this in /etc/monit/slack_notifications.sh
.
Execute the following commands to make the script executable:
$ sudo touch /etc/monit/slack_notifications.sh
$ sudo chmod +x /etc/monit/slack_notifications.sh
#!/bin/sh
/usr/bin/curl \
-X POST \
-s \
--data-urlencode "payload={ \
\"channel\": \"#monit\", \
\"username\": \"Monit\", \
\"pretext\": \"servername | $MONIT_DATE\", \
\"color\": \"danger\", \
\"icon_emoji\": \":ghost:\", \
\"text\": \"$MONIT_SERVICE - $MONIT_DESCRIPTION\" \
}" \
https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX
Don’t forget to replace https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX
with your unique webhook URL and replace #monit
with the channel of your choice.
Swap emails for Slack notifications
Since there is no native solution to set this notification as a default notification for all alerts, you should set this on all your monit service configurations files. For example in /etc/monit/conf.d/nginx
:
check process nginx with pidfile /var/run/nginx.pid
start program = "/usr/sbin/service nginx start"
stop program = "/usr/sbin/service nginx stop"
if failed host 127.0.0.1 port 80 then restart
if changed pid then exec "/etc/monit/slack_notifications.sh"
if failed host 127.0.0.1 port 80 then exec "/etc/monit/slack_notifications.sh" else if succeeded then exec "/etc/monit/slack_notifications.sh"
if cpu is greater than 40% for 2 cycles then exec "/etc/monit/slack_notifications.sh" else if succeeded then exec "/etc/monit/slack_notifications.sh"
if cpu > 60% for 5 cycles then restart
Don’t forget to add the else if succeeded then
part, or you’ll not receive notifications when thing are fine again.
Restart your Monit, restart Nginx and you’ll see a PID changed message.
A native solution
At first I tried a native solution. There is a specific Monit branch with Slack Notifications included.
$ sudo apt-get install monit install libssl-dev bison flex checkinstall
$ sudo apt-get remove monit
$ cd /tmp
$ git clone git@bitbucket.org:pierreaubert/monit.git
$ cd monit/
$ git checkout feature/slack
$ ./bootstrap && ./configure --without-pam --enable-optimized --prefix=/usr
$ sudo checkinstall
All you need to do is add the following line at the bottom of /etc/monit/monitrc
:
set slack https://YYYYY.slack.com/services/hooks/incoming-webhook?token=XXXXXXXXXXXXXXXXX
But this isn’t very customizable.
Note
Big up to the developer of Monit and this pull-request.
Thanks for reading
Feel free to leave a comment if you have remarks or like this post