icinga2 notifications via Signal messenger

I recently finished implementing icinga2 to monitor our servers and services. It is a tad complicated but overall I like it very much.

Right around the same time, I also started using Signal on my Android device. I had never used Signal before since it used to require you to have Google Play services, but it is not the case anymore.

The idea of using Signal as a way to alert me of critical failures on my systems thus came to me. Most serious organisations use some sort of SMS gateway to alert employees of failures and these services are often expensive. I also do not have a "real" phone with a SIM card and only use my VoIP account to receive SMS when I have WiFi with the neat voip.ms sms app.

Contrarywise to an SMS gateway on a server, Signal is fairly easy to use, it is free and messages are end-to-end encrypted between devices. Configuration of a client is simple enough with the wonderful signal-cli project.

Configuring signal-cli

First of all, since Signal uses real phone numbers as usernames, you will need to get a phone number with some SMS support. You could also try one without SMS support and try to get confirmation codes via Signal's calling feature, but it's a hassle. Instead, I recommended using a good VoIP provider like voip.ms.

Once you have a phone number, download signal-cli on your server and simlink it to /usr/local/bin.

You will first need to register your number on Signal with the unix user you use for icinga2. Don't make the mistake I did of doing everything with your regular user and then wondering why it does not work:

$ sudo -u nagios signal-cli -u ICINGANUMBER register

Once that ran, check your SMS (or voicemail) and use the confirmation code this way:

$ sudo -u nagios signal-cli -u ICINGANUMBER verify YYY-ZZZ

signal-cli should now work. You can test this by sending a test message to your personnal number:

$ echo "test message - icinga2" | sudo -u nagios signal-cli -u ICINGANUMBER send MYNUMBER

Important notes

You will not be able to register a same phone number on multiple machines. Doing so will likely bork all other instances of signal-cli where you used that number. If you ever run into troubles with this (I know I did), you can always unregister your number (give it a few minutes after completing the form) and try again.

Also note that signal-cli can be a little slow compared to the mobile client and can take a few minutes to run a command.

Making icinga2 work with signal-cli

Now that you have signal-cli working, we need to make icinga2 use it to send notifications.

Notifications in icinga2 are sent using scripts. By default, there are two scripts in /etc/icinga2/scripts that are used to send email notifications.

You will need to add two files to the /scripts directory on the master. First add signal-host-notification.sh, the script we will use to alert you of a problem with a host:

#!/bin/sh
template=`cat <<TEMPLATE
$NOTIFICATIONTYPE - $HOSTDISPLAYNAME is $HOSTSTATE

Notification Type: $NOTIFICATIONTYPE

Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $HOSTSTATE

Date/Time: $LONGDATETIME

Additional Info: $HOSTOUTPUT

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
`

/usr/bin/printf "%b" "$template" | signal-cli -u $SIGNALNUMBER send $USERPHONE

Then add signal-service-notification.sh, the script used to alert you in case a service goes wrong:

#!/bin/sh
template=`cat <<TEMPLATE
$NOTIFICATIONTYPE - $HOSTDISPLAYNAME - $SERVICEDISPLAYNAME is $SERVICESTATE

Notification Type: $NOTIFICATIONTYPE

Service: $SERVICEDESC
Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $SERVICESTATE

Date/Time: $LONGDATETIME

Additional Info: $SERVICEOUTPUT

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
`

/usr/bin/printf "%b" "$template" | signal-cli -u $SIGNALNUMBER send $USERPHONE

Once that is done, you'll need to add some NotificationCommand objects to your commands.conf file. Normally, that file is in a global zone:

object NotificationCommand "signal-host-notification" {
  command = [ SysconfDir + "/icinga2/scripts/signal-host-notification.sh" ]

  env = {
    NOTIFICATIONTYPE = "$notification.type$"  
    HOSTALIAS = "$host.display_name$"
    HOSTADDRESS = "$address$"
    HOSTSTATE = "$host.state$"
    LONGDATETIME = "$icinga.long_date_time$"  
    HOSTOUTPUT = "$host.output$"
    NOTIFICATIONAUTHORNAME = "$notification.author$"
    NOTIFICATIONCOMMENT = "$notification.comment$"
    HOSTDISPLAYNAME = "$host.display_name$"   
    SIGNALNUMBER = "ICINGANUMBER"
    USERPHONE = "MYNUMBER"
  }
  timeout = 10m
}

object NotificationCommand "signal-service-notification" {
  command = [ SysconfDir + "/icinga2/scripts/signal-service-notification.sh" ]

  env = {
    NOTIFICATIONTYPE = "$notification.type$"
    SERVICEDESC = "$service.name$"
    HOSTALIAS = "$host.display_name$"
    HOSTADDRESS = "$address$"
    SERVICESTATE = "$service.state$"
    LONGDATETIME = "$icinga.long_date_time$"
    SERVICEOUTPUT = "$service.output$"
    NOTIFICATIONAUTHORNAME = "$notification.author$"
    NOTIFICATIONCOMMENT = "$notification.comment$"
    HOSTDISPLAYNAME = "$host.display_name$"
    SERVICEDISPLAYNAME = "$service.display_name$"
    SIGNALNUMBER = "ICINGANUMBER"
    USERPHONE = "MYNUMBER"
  }
  timeout = 10m
}

Note that for multiple users systems, you can set the USERPHONE variable dynamically based on schedules.

Finally, you need to apply the NotificationObject to certain hosts. Add this snippet to your notifications.conf file, normally also in your global zone:

apply Notification "signal-icingaadmin" to Host {
  import "signal-host-notification"

  user_groups = host.vars.notification.signal.groups
  users = host.vars.notification.signal.users

  assign where host.vars.notification.signal

  if (host.vars.notification_interval) {
    interval = host.vars.notification_interval
  }
}

apply Notification "signal-icingaadmin" to Service {
  import "signal-service-notification"

  user_groups = host.vars.notification.signal.groups
  users = host.vars.notification.signal.users

  assign where host.vars.notification.signal

  if (host.vars.notification_interval) {
    interval = host.vars.notification_interval
  }
}

On a host, you can now use signal by adding this to your hosts.conf file:

vars.notification["signal"] = {
    /* The UserGroup `icingaadmins` is defined in `users.conf`. */
    groups = [ "icingaadmins" ]
}