Skip to main content

Hipchat and Icinga

Hipchat Notify 2.0

Hipchat notification with API 2.0 to be used with ICINGA/Nagios

Table of Contents

Author

Audience

  • System Engineers and operation engineers

Introduction

Change the default mail notification of Icinga server to hipchat notification using ruby code. This will allow a single place of management of all the notification and alerts across organization. Let that be service,host or business level alerts all can be managed and monitored using hipchat and hubot will give certain advantage over traditional alerting system.
  1. Proactive and reactive alerting
  2. Managed monitoring
  3. Single place of all the alerts
  4. Better communication and collaboration
  5. Integration with multiple tools in CI cycle
    • Jenkins
    • Chef
    • Bitbucket/Github
    • Jira
    • Confluence …. Etc
As per plan once Elastalert is implemented Hipchat will support business and revenue alerts, and hopefully with event based proactive/reactive handling or issues.

Ruby Script

Need to create a Ruby script in order to make sure that we can send messages from a single server. Idea here is to be able to configure icinga server to send message to custom user groups and Hipchat account.
We are going to use a Ruby gem called hipchat which will support API version 2.0. In order for script to work we need to install 2 gems,
  1. Hipchat
  2. Trollop
# gem install hipchat

# gem install trollop
#!/usr/bin/env ruby

require 'hipchat'

require 'trollop'


#

# Provides a hipchat notifier with minimal requirements.

# Post the nofication to room

#

# Docs: http://wiki.opscode.com/display/chef/Exception+and+Report+Handlers

#

# Install - add the following to your client.rb:

# gem install hipchat \# Configure CLI entries

# gem install trollop \# configure commandline option parser

module HipChat

class NotifyRoomCli

def initialize(api_token, room_name, msg, options={})

defaults = { hipchat_options: {api_version: 'v2',server_url: 'https://api.hipchat.com'}, msg_options: {:notify => true}, excluded_envs: [], msg_prefix: ''}

options = defaults.merge(options)

@api_token = api_token

@room_name = room_name

@msg = msg

@hipchat_options = options[:hipchat_options]

@msg_options = options[:msg_options]

@msg_prefix = options[:msg_prefix]

@excluded_envs = options[:excluded_envs]

@to_user=options[:name]

case

when options[:alerttype].match(/warning/i)

@color = 'yellow'

when options[:alerttype].match(/critical/i)

@color = 'red'

when options[:alerttype].match(/info/i)

@color = 'green'

end if options[:alerttype]

end

def report

if @msg

@msg_options[:color]=(@color || 'yellow')

client = HipChat::Client.new(@api_token, @hipchat_options)

client[@room_name].send(@to_user, [@msg_prefix, @msg].join(' '), @msg_options)

end

end

end

end

begin

opts = Trollop::options do

opt :message, "Use monkey mode" ,:type => :string \# flag --monkey, default false

opt :name, "Monkey name", :type => :string \# string --name <s>, default nil

opt :apitoken, "HIPCHAT API Token 2.0", :type=> :string

opt :roomname, "Room id from Hipchat", :type=> :string

opt :alerttype, "warning/critical/info", :type => :string

end

[ :message, :apitoken, :roomname].each do |key|

Trollop::die "arguments required --\#{key}" unless opts[key]

end

hipchat=HipChat::NotifyRoomCli.new(opts[:apitoken],opts[:roomname],opts[:message],opts)

hipchat.report

rescue Errno::ENOENT => err

abort "hip_chat_cli: \#{err.message}"

end

Script used on server

following is a script which should which we need to configure to use this code with a icinga server

Service notification

root@hubot0:/etc/icinga2/scripts\# cat hipchat-service-notification.sh

#!/bin/sh

template=\`cat <<TEMPLATE

\*\*\*\*\* Icinga \*\*\*\*\*

Notification Type: \$NOTIFICATIONTYPE

Service: \$SERVICEDESC

Host: \$HOSTALIAS

Address: \$HOSTADDRESS

State: \$SERVICESTATE

Date/Time: \$LONGDATETIME

Additional Info: \$SERVICEOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

TEMPLATE

\`

\#/usr/bin/printf "%b" "\$template" | mail -s "\$NOTIFICATIONTYPE - \$HOSTDISPLAYNAME - \$SERVICEDISPLAYNAME is \$SERVICESTATE" \$USEREMAIL

dir="\$(readlink -f \$(dirname \$0))"

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "\$SERVICESTATE"

Host notification

#root@hubot0:/etc/icinga2/scripts\# cat mail-host-notification-hipchat.sh

#!/bin/sh

dir="\$(readlink -f \$(dirname \$0))"

template=\`cat <<EOF

HOST DOWN \$HOSTALIAS; Address: \$HOSTADDRESS; State: \$HOSTSTATE ; Date/Time: \$LONGDATETIME

Additional Info: \$HOSTOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

EOF

\`

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "warning"

root@hubot0:/etc/icinga2/scripts\#

Change in command.conf for Icinga server

This will change the Icinga server to support the hipchat adapter ,
root@hubot0:/etc/icinga2/scripts\# cat ../conf.d/commands.conf

/\* Command objects \*/

object NotificationCommand "mail-host-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/mail-host-notification-hipchat.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

HOSTSTATE = "\$host.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

HOSTOUTPUT = "\$host.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

object NotificationCommand "mail-service-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/hipchat-service-notification.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

SERVICEDESC = "\$service.name\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

SERVICESTATE = "\$service.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

SERVICEOUTPUT = "\$service.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

SERVICEDISPLAYNAME = "\$service.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

Example notification

Roadmap

  1. Add hipchat user for Icinga server
  2. Configure to talk to groups using token from user settings in
    hipchat
  3. Change notification files and update token+room_ids
All these steps are required to make sure hubot can take actions on events. For now hubot can not take actions if the user Hubot and notifying user is matched. It is required to prevent race conditions.

Labels:

Comments

Post a Comment

Popular posts

A few useful sql functions

Start mysql in Ubuntu without having root privilege:- If you want to use mysql in Ubutu you can use following command which will use a root level privilege   $ mysql -u root -p Enter password: Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 147 Server version: 5.1.49-1ubuntu8.1-log (Ubuntu) When it demands to enter the password fill it with 'root' and hopefully you'll get logged in .   Last_insert_id():- (with no argument) returns the first automatically generated value that was set for an AUTO_INCREMENT column by the most recently executed INSERT statement to affect such a column. For example, after inserting a row that generates an AUTO_INCREMENT value, you can get the value like this:   mysql> SELECT LAST_INSERT_ID(); Database():- Database() method returns the current selected database and you can use it in your communication and your queries. The syntax is :   mysql>select Database (); User():- It a...
Post a Comment
Read more

Nodejs SSA learnings

nodejs-akamai-page.MD Overview In almost every blog when people talk about deploying something on K8S they use node mostly because setting up http server is not that easy. I wish that it could be this easy for our case, (which it wasn’t). Manly because the way we wanted the application to work, plans for SEO and multiple data pipelines for business including Amplitude for client and Pubsub for application and business metrics. Making all these calls from nodejs was easy for developers just 1 more promise. Everything went smoothly as long as functional testing was required. We were ready to launch and somebody from SRE/Devops team asked: “Have you done load testing?” . Everything was superb up to this point where load testing results are required to make service live. Load Testing We started to put load to our services using apache benchmark. Very soon we realised our application is not scaling as much as we expected. To our surprise application was able to handle only 3 requ...
Post a Comment
Read more

Istio multicluster, gotchas ....

istio.md Istio lets you connect, secure, control, and observe services. At a high level, Istio helps reduce the complexity of these deployments, and eases the strain on your development teams. It is a completely open source service mesh that layers transparently onto existing distributed applications. It is also a platform, including APIs that let it integrate into any logging platform, or telemetry or policy system. Istio’s diverse feature set lets you successfully, and efficiently, run a distributed microservice architecture, and provides a uniform way to secure, connect, and monitor microservices. In context of Vuclip istio allows us to reduce the code and environment configurations while keeping the similar or more feature sets at our disposal. Since istio is designed to bridge the gap for both development teams and SRE, it is essential to see and visualize that in practice. Istio will affect us in our ability to connect , secure(HTTPs TLS, mtls [Phase-2]), control(external comm...
Post a Comment
Read more

Managing & Configuring YUM [Centos 6.0]

Yum Configuration Hi Friends, we all have faced problems while configuring yum in RedHat, Fedora and Centos. Please find the below mention steps to use all the RPM packages come along with your distribution. Yum configuration will allow you to use them with proper dependency resolution. To configure yum you must have two packages, 1. yum 2. createrepo To obtain the same do the below mention just after you have mounted you disk. There could be two conditions a.) You have the distribution disk i.e. DVD. Please follow the below mention instructions to manage the same. Your system mush have the greater or equal space of DVD. # df -ah User the partition which has the space and create a directory. In my case /mnt has the space. # mount /dev/dvdwriter /mnt # cd /mnt/Packages/ # rpm -ivh yum-* --nodeps –force # rpm -ivh vsftps* -y # rpm -ivh createrepo* --nodeps --force # cp -rpvf /mnt/* /var/ftp/pub It will create the dump on /var/ftp/pub now we need to unmount the disk. # umount /mnt b.)...
Post a Comment
Read more

Using except command with bash

1.Use the interpreter for bash at first line . #!/bin/bash 2. Use variables as per requirement and pass it to except if needed. HOST="localhost" USER="chitti" PASS="123" CMD=$@ 3. Use expect script as required. XYZ=$(expect -c " spawn ssh $USER@$HOST expect \"password:\" send \"$PASS\r\" expect \"\\\\$\" send \"$CMD\r\" expect -re \"$USER.*\" send \"logout\" ") 4.Print the result of except using echo like echo "${XYZ}"
Post a Comment
Read more

Enter your email address: