Skip to main content

Hipchat and Icinga

Hipchat Notify 2.0

Hipchat notification with API 2.0 to be used with ICINGA/Nagios

Table of Contents

Author

Audience

  • System Engineers and operation engineers

Introduction

Change the default mail notification of Icinga server to hipchat notification using ruby code. This will allow a single place of management of all the notification and alerts across organization. Let that be service,host or business level alerts all can be managed and monitored using hipchat and hubot will give certain advantage over traditional alerting system.
  1. Proactive and reactive alerting
  2. Managed monitoring
  3. Single place of all the alerts
  4. Better communication and collaboration
  5. Integration with multiple tools in CI cycle
    • Jenkins
    • Chef
    • Bitbucket/Github
    • Jira
    • Confluence …. Etc
As per plan once Elastalert is implemented Hipchat will support business and revenue alerts, and hopefully with event based proactive/reactive handling or issues.

Ruby Script

Need to create a Ruby script in order to make sure that we can send messages from a single server. Idea here is to be able to configure icinga server to send message to custom user groups and Hipchat account.
We are going to use a Ruby gem called hipchat which will support API version 2.0. In order for script to work we need to install 2 gems,
  1. Hipchat
  2. Trollop
# gem install hipchat

# gem install trollop
#!/usr/bin/env ruby

require 'hipchat'

require 'trollop'


#

# Provides a hipchat notifier with minimal requirements.

# Post the nofication to room

#

# Docs: http://wiki.opscode.com/display/chef/Exception+and+Report+Handlers

#

# Install - add the following to your client.rb:

# gem install hipchat \# Configure CLI entries

# gem install trollop \# configure commandline option parser

module HipChat

class NotifyRoomCli

def initialize(api_token, room_name, msg, options={})

defaults = { hipchat_options: {api_version: 'v2',server_url: 'https://api.hipchat.com'}, msg_options: {:notify => true}, excluded_envs: [], msg_prefix: ''}

options = defaults.merge(options)

@api_token = api_token

@room_name = room_name

@msg = msg

@hipchat_options = options[:hipchat_options]

@msg_options = options[:msg_options]

@msg_prefix = options[:msg_prefix]

@excluded_envs = options[:excluded_envs]

@to_user=options[:name]

case

when options[:alerttype].match(/warning/i)

@color = 'yellow'

when options[:alerttype].match(/critical/i)

@color = 'red'

when options[:alerttype].match(/info/i)

@color = 'green'

end if options[:alerttype]

end

def report

if @msg

@msg_options[:color]=(@color || 'yellow')

client = HipChat::Client.new(@api_token, @hipchat_options)

client[@room_name].send(@to_user, [@msg_prefix, @msg].join(' '), @msg_options)

end

end

end

end

begin

opts = Trollop::options do

opt :message, "Use monkey mode" ,:type => :string \# flag --monkey, default false

opt :name, "Monkey name", :type => :string \# string --name <s>, default nil

opt :apitoken, "HIPCHAT API Token 2.0", :type=> :string

opt :roomname, "Room id from Hipchat", :type=> :string

opt :alerttype, "warning/critical/info", :type => :string

end

[ :message, :apitoken, :roomname].each do |key|

Trollop::die "arguments required --\#{key}" unless opts[key]

end

hipchat=HipChat::NotifyRoomCli.new(opts[:apitoken],opts[:roomname],opts[:message],opts)

hipchat.report

rescue Errno::ENOENT => err

abort "hip_chat_cli: \#{err.message}"

end

Script used on server

following is a script which should which we need to configure to use this code with a icinga server

Service notification

root@hubot0:/etc/icinga2/scripts\# cat hipchat-service-notification.sh

#!/bin/sh

template=\`cat <<TEMPLATE

\*\*\*\*\* Icinga \*\*\*\*\*

Notification Type: \$NOTIFICATIONTYPE

Service: \$SERVICEDESC

Host: \$HOSTALIAS

Address: \$HOSTADDRESS

State: \$SERVICESTATE

Date/Time: \$LONGDATETIME

Additional Info: \$SERVICEOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

TEMPLATE

\`

\#/usr/bin/printf "%b" "\$template" | mail -s "\$NOTIFICATIONTYPE - \$HOSTDISPLAYNAME - \$SERVICEDISPLAYNAME is \$SERVICESTATE" \$USEREMAIL

dir="\$(readlink -f \$(dirname \$0))"

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "\$SERVICESTATE"

Host notification

#root@hubot0:/etc/icinga2/scripts\# cat mail-host-notification-hipchat.sh

#!/bin/sh

dir="\$(readlink -f \$(dirname \$0))"

template=\`cat <<EOF

HOST DOWN \$HOSTALIAS; Address: \$HOSTADDRESS; State: \$HOSTSTATE ; Date/Time: \$LONGDATETIME

Additional Info: \$HOSTOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

EOF

\`

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "warning"

root@hubot0:/etc/icinga2/scripts\#

Change in command.conf for Icinga server

This will change the Icinga server to support the hipchat adapter ,
root@hubot0:/etc/icinga2/scripts\# cat ../conf.d/commands.conf

/\* Command objects \*/

object NotificationCommand "mail-host-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/mail-host-notification-hipchat.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

HOSTSTATE = "\$host.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

HOSTOUTPUT = "\$host.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

object NotificationCommand "mail-service-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/hipchat-service-notification.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

SERVICEDESC = "\$service.name\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

SERVICESTATE = "\$service.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

SERVICEOUTPUT = "\$service.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

SERVICEDISPLAYNAME = "\$service.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

Example notification

Roadmap

  1. Add hipchat user for Icinga server
  2. Configure to talk to groups using token from user settings in
    hipchat
  3. Change notification files and update token+room_ids
All these steps are required to make sure hubot can take actions on events. For now hubot can not take actions if the user Hubot and notifying user is matched. It is required to prevent race conditions.

Labels:

Comments

Post a Comment

Popular posts

Change password of mysql users

Change MySQL user's password in bulk Sometimes we need to change the password of mysql users at bulk. Lets say your company has fired a group of peoples or dissolved a project. This requirement can be supplied as : We can achieve the same by changing the password field in mysql.user table. Password() is a predifined method in mysql which generates encrypted value of a certain string. host update mysql.user set password = PASSWORD("passwd") where host IN (<Comma separated list of HostName/IP>); user update mysql.user set password = PASSWORD("passwd") where user IN (<Comma separated list of ‘user’@’host’>);
Post a Comment
Read more

A few useful sql functions

Start mysql in Ubuntu without having root privilege:- If you want to use mysql in Ubutu you can use following command which will use a root level privilege   $ mysql -u root -p Enter password: Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 147 Server version: 5.1.49-1ubuntu8.1-log (Ubuntu) When it demands to enter the password fill it with 'root' and hopefully you'll get logged in .   Last_insert_id():- (with no argument) returns the first automatically generated value that was set for an AUTO_INCREMENT column by the most recently executed INSERT statement to affect such a column. For example, after inserting a row that generates an AUTO_INCREMENT value, you can get the value like this:   mysql> SELECT LAST_INSERT_ID(); Database():- Database() method returns the current selected database and you can use it in your communication and your queries. The syntax is :   mysql>select Database (); User():- It a...
Post a Comment
Read more

Allowing users to have ssh access

Allowing users to have ssh access Hi Readers, It is one of the tasks we need to complete in order to allow users to log-in into your server without compromising your security. We are going to accomplish the following tasks, 1.) Allow the users for given domain only 2.) Must allow access to a given domain 3.) Block access for a specific domain. These questions are asked in RedHat certification examination RHCE6 We are going to complete the above mention task using iptables To give proper example we are taking 192.168.20.0/255.255.255.0 as our domain and 192.168.21.0/255.255.255.0 as other domain. Assuming that your system is a fresh installation we can remove all rules previously applied. # iptables -F  The above mention command will flush all the previously applied rules. Insert a rule in your input chain by below mention command, # iptables -I INPUT -s <ip of your domain>/<subnet mask> -p <protocal tcp/udp> --dport <port> -...
Post a Comment
Read more

Add user in Sudoer file

The command sudo provides other users to run a command as a root. To use sudo user must have their level of privilege in /etc/sudoer file. In some cases this file is set for read only and others can edit that file and add their user name in Sudoer for this case such as Fedora 12 users need to provide this syntax as a root on their terminal. Syntax for adding a user in Sudoer file is: echo "username ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers And to gain the writing privilege you can use the following code in your terminal : chmod 755 /etc/sudoers This code will be enough to give permissions to the user and his user group. If you are an ubuntu user you can easily add your user in this file like: vi /etc/sudoers And you will get a text file. Afterward you need to press 'i' for edit and edit by adding your username just below "root ALL=(ALL) ALL" "username ALL=(ALL) ALL"
Post a Comment
Read more

Enter your email address: