Skip to main content

Hipchat and Icinga

Hipchat Notify 2.0

Hipchat notification with API 2.0 to be used with ICINGA/Nagios

Table of Contents

Author

Audience

  • System Engineers and operation engineers

Introduction

Change the default mail notification of Icinga server to hipchat notification using ruby code. This will allow a single place of management of all the notification and alerts across organization. Let that be service,host or business level alerts all can be managed and monitored using hipchat and hubot will give certain advantage over traditional alerting system.
  1. Proactive and reactive alerting
  2. Managed monitoring
  3. Single place of all the alerts
  4. Better communication and collaboration
  5. Integration with multiple tools in CI cycle
    • Jenkins
    • Chef
    • Bitbucket/Github
    • Jira
    • Confluence …. Etc
As per plan once Elastalert is implemented Hipchat will support business and revenue alerts, and hopefully with event based proactive/reactive handling or issues.

Ruby Script

Need to create a Ruby script in order to make sure that we can send messages from a single server. Idea here is to be able to configure icinga server to send message to custom user groups and Hipchat account.
We are going to use a Ruby gem called hipchat which will support API version 2.0. In order for script to work we need to install 2 gems,
  1. Hipchat
  2. Trollop
# gem install hipchat

# gem install trollop
#!/usr/bin/env ruby

require 'hipchat'

require 'trollop'


#

# Provides a hipchat notifier with minimal requirements.

# Post the nofication to room

#

# Docs: http://wiki.opscode.com/display/chef/Exception+and+Report+Handlers

#

# Install - add the following to your client.rb:

# gem install hipchat \# Configure CLI entries

# gem install trollop \# configure commandline option parser

module HipChat

class NotifyRoomCli

def initialize(api_token, room_name, msg, options={})

defaults = { hipchat_options: {api_version: 'v2',server_url: 'https://api.hipchat.com'}, msg_options: {:notify => true}, excluded_envs: [], msg_prefix: ''}

options = defaults.merge(options)

@api_token = api_token

@room_name = room_name

@msg = msg

@hipchat_options = options[:hipchat_options]

@msg_options = options[:msg_options]

@msg_prefix = options[:msg_prefix]

@excluded_envs = options[:excluded_envs]

@to_user=options[:name]

case

when options[:alerttype].match(/warning/i)

@color = 'yellow'

when options[:alerttype].match(/critical/i)

@color = 'red'

when options[:alerttype].match(/info/i)

@color = 'green'

end if options[:alerttype]

end

def report

if @msg

@msg_options[:color]=(@color || 'yellow')

client = HipChat::Client.new(@api_token, @hipchat_options)

client[@room_name].send(@to_user, [@msg_prefix, @msg].join(' '), @msg_options)

end

end

end

end

begin

opts = Trollop::options do

opt :message, "Use monkey mode" ,:type => :string \# flag --monkey, default false

opt :name, "Monkey name", :type => :string \# string --name <s>, default nil

opt :apitoken, "HIPCHAT API Token 2.0", :type=> :string

opt :roomname, "Room id from Hipchat", :type=> :string

opt :alerttype, "warning/critical/info", :type => :string

end

[ :message, :apitoken, :roomname].each do |key|

Trollop::die "arguments required --\#{key}" unless opts[key]

end

hipchat=HipChat::NotifyRoomCli.new(opts[:apitoken],opts[:roomname],opts[:message],opts)

hipchat.report

rescue Errno::ENOENT => err

abort "hip_chat_cli: \#{err.message}"

end

Script used on server

following is a script which should which we need to configure to use this code with a icinga server

Service notification

root@hubot0:/etc/icinga2/scripts\# cat hipchat-service-notification.sh

#!/bin/sh

template=\`cat <<TEMPLATE

\*\*\*\*\* Icinga \*\*\*\*\*

Notification Type: \$NOTIFICATIONTYPE

Service: \$SERVICEDESC

Host: \$HOSTALIAS

Address: \$HOSTADDRESS

State: \$SERVICESTATE

Date/Time: \$LONGDATETIME

Additional Info: \$SERVICEOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

TEMPLATE

\`

\#/usr/bin/printf "%b" "\$template" | mail -s "\$NOTIFICATIONTYPE - \$HOSTDISPLAYNAME - \$SERVICEDISPLAYNAME is \$SERVICESTATE" \$USEREMAIL

dir="\$(readlink -f \$(dirname \$0))"

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "\$SERVICESTATE"

Host notification

#root@hubot0:/etc/icinga2/scripts\# cat mail-host-notification-hipchat.sh

#!/bin/sh

dir="\$(readlink -f \$(dirname \$0))"

template=\`cat <<EOF

HOST DOWN \$HOSTALIAS; Address: \$HOSTADDRESS; State: \$HOSTSTATE ; Date/Time: \$LONGDATETIME

Additional Info: \$HOSTOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

EOF

\`

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "warning"

root@hubot0:/etc/icinga2/scripts\#

Change in command.conf for Icinga server

This will change the Icinga server to support the hipchat adapter ,
root@hubot0:/etc/icinga2/scripts\# cat ../conf.d/commands.conf

/\* Command objects \*/

object NotificationCommand "mail-host-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/mail-host-notification-hipchat.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

HOSTSTATE = "\$host.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

HOSTOUTPUT = "\$host.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

object NotificationCommand "mail-service-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/hipchat-service-notification.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

SERVICEDESC = "\$service.name\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

SERVICESTATE = "\$service.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

SERVICEOUTPUT = "\$service.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

SERVICEDISPLAYNAME = "\$service.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

Example notification

Roadmap

  1. Add hipchat user for Icinga server
  2. Configure to talk to groups using token from user settings in
    hipchat
  3. Change notification files and update token+room_ids
All these steps are required to make sure hubot can take actions on events. For now hubot can not take actions if the user Hubot and notifying user is matched. It is required to prevent race conditions.

Labels:

Comments

Post a Comment

Popular posts

Helm generic springboot templates

With the dramatically increasing demand for container orchestration specifically Kubernetes, demand to template K8S manifests(Json/Yaml) also came to light. To handle increasing manifests, new CRDs(Custom resource definition), etc… it became obvious that we need a package manager somewhat like yum, apt, etc… However, the nature of Kubernetes manifest is very different than what one used to have with Yum and Apt. These manifests required a lot of templates which is now supported by Helm, a tool written in GoLang with custom helm functions and pipelines. Neutral background on templating Templating has been a driver for configuration management for a long time. While it may seem trivial for users coming from Ansible, Chef, Puppet, Salt, etc…, it is not. Once one moves to Kubernetes, the very first realization is hard declarative approach that Kubernetes follows. It is difficult to make generic templating with declarative form since each application may have some unique feature and r...
Post a Comment
Read more

Istio multicluster, gotchas ....

istio.md Istio lets you connect, secure, control, and observe services. At a high level, Istio helps reduce the complexity of these deployments, and eases the strain on your development teams. It is a completely open source service mesh that layers transparently onto existing distributed applications. It is also a platform, including APIs that let it integrate into any logging platform, or telemetry or policy system. Istio’s diverse feature set lets you successfully, and efficiently, run a distributed microservice architecture, and provides a uniform way to secure, connect, and monitor microservices. In context of Vuclip istio allows us to reduce the code and environment configurations while keeping the similar or more feature sets at our disposal. Since istio is designed to bridge the gap for both development teams and SRE, it is essential to see and visualize that in practice. Istio will affect us in our ability to connect , secure(HTTPs TLS, mtls [Phase-2]), control(external comm...
Post a Comment
Read more

Sent mail from perl

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 sub Mail_Mailer { my $destination_directory = '/data/' ; my $from_address = "abc@gmail.com" ; my $to_address = "abc@gmail.com" ; my $subject = "SOFT DATA " ; my $body = "Dear Sir\nPlease find the complete set of data on sftp\n." ; my $cc = "test@gmail.com" ; opendir (DES, $destination_directory ); my @files = readdir (DES); close (DES); my @mail_sent_file = @file ; foreach my $mail_file_names ( @mail_sent_file ) { $body = $ {body} . "\n" . $mail_file_names . "\n" ; } $body = $ {body} . "\nRegards\nreportsadmin." ; my $mailer = Mail:: Mailer -> new ( "sendmail" ) or die ; $mailer -> open ( { From => $from_address , To => $t...
Post a Comment
Read more

Using Flash in your HTML

Edit your Publish settings for Flash SWFs and HTML to reflect how you want your Flash SWF to appear in your web page. Export your Flash movie as HTML. Locate your HTML file on your computer, right-click, and select "Open With". Choose either NotePad or another text editor. Copy the source code from the HTML file. Paste it into your web page's source code in the appropriate location where you want your SWF file to display. Edit the file path to reflect the location of the SWF file on your web server, and upload both your HTML and SWF file to the appropriate directories on your server. ( Note:  this also applies if you're using PHP, JSP, ASP, CGI, or other web page extensions.) Your code should look something like this: <OBJECT classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,0,0" WIDTH="320" HEIGHT="240" id="You...
Post a Comment
Read more

A few useful sql functions

Start mysql in Ubuntu without having root privilege:- If you want to use mysql in Ubutu you can use following command which will use a root level privilege   $ mysql -u root -p Enter password: Welcome to the MySQL monitor.  Commands end with ; or \g. Your MySQL connection id is 147 Server version: 5.1.49-1ubuntu8.1-log (Ubuntu) When it demands to enter the password fill it with 'root' and hopefully you'll get logged in .   Last_insert_id():- (with no argument) returns the first automatically generated value that was set for an AUTO_INCREMENT column by the most recently executed INSERT statement to affect such a column. For example, after inserting a row that generates an AUTO_INCREMENT value, you can get the value like this:   mysql> SELECT LAST_INSERT_ID(); Database():- Database() method returns the current selected database and you can use it in your communication and your queries. The syntax is :   mysql>select Database (); User():- It a...
Post a Comment
Read more

Enter your email address: