Skip to main content

Hipchat and Icinga

Hipchat Notify 2.0

Hipchat notification with API 2.0 to be used with ICINGA/Nagios

Table of Contents

Author

Audience

  • System Engineers and operation engineers

Introduction

Change the default mail notification of Icinga server to hipchat notification using ruby code. This will allow a single place of management of all the notification and alerts across organization. Let that be service,host or business level alerts all can be managed and monitored using hipchat and hubot will give certain advantage over traditional alerting system.
  1. Proactive and reactive alerting
  2. Managed monitoring
  3. Single place of all the alerts
  4. Better communication and collaboration
  5. Integration with multiple tools in CI cycle
    • Jenkins
    • Chef
    • Bitbucket/Github
    • Jira
    • Confluence …. Etc
As per plan once Elastalert is implemented Hipchat will support business and revenue alerts, and hopefully with event based proactive/reactive handling or issues.

Ruby Script

Need to create a Ruby script in order to make sure that we can send messages from a single server. Idea here is to be able to configure icinga server to send message to custom user groups and Hipchat account.
We are going to use a Ruby gem called hipchat which will support API version 2.0. In order for script to work we need to install 2 gems,
  1. Hipchat
  2. Trollop
# gem install hipchat

# gem install trollop
#!/usr/bin/env ruby

require 'hipchat'

require 'trollop'


#

# Provides a hipchat notifier with minimal requirements.

# Post the nofication to room

#

# Docs: http://wiki.opscode.com/display/chef/Exception+and+Report+Handlers

#

# Install - add the following to your client.rb:

# gem install hipchat \# Configure CLI entries

# gem install trollop \# configure commandline option parser

module HipChat

class NotifyRoomCli

def initialize(api_token, room_name, msg, options={})

defaults = { hipchat_options: {api_version: 'v2',server_url: 'https://api.hipchat.com'}, msg_options: {:notify => true}, excluded_envs: [], msg_prefix: ''}

options = defaults.merge(options)

@api_token = api_token

@room_name = room_name

@msg = msg

@hipchat_options = options[:hipchat_options]

@msg_options = options[:msg_options]

@msg_prefix = options[:msg_prefix]

@excluded_envs = options[:excluded_envs]

@to_user=options[:name]

case

when options[:alerttype].match(/warning/i)

@color = 'yellow'

when options[:alerttype].match(/critical/i)

@color = 'red'

when options[:alerttype].match(/info/i)

@color = 'green'

end if options[:alerttype]

end

def report

if @msg

@msg_options[:color]=(@color || 'yellow')

client = HipChat::Client.new(@api_token, @hipchat_options)

client[@room_name].send(@to_user, [@msg_prefix, @msg].join(' '), @msg_options)

end

end

end

end

begin

opts = Trollop::options do

opt :message, "Use monkey mode" ,:type => :string \# flag --monkey, default false

opt :name, "Monkey name", :type => :string \# string --name <s>, default nil

opt :apitoken, "HIPCHAT API Token 2.0", :type=> :string

opt :roomname, "Room id from Hipchat", :type=> :string

opt :alerttype, "warning/critical/info", :type => :string

end

[ :message, :apitoken, :roomname].each do |key|

Trollop::die "arguments required --\#{key}" unless opts[key]

end

hipchat=HipChat::NotifyRoomCli.new(opts[:apitoken],opts[:roomname],opts[:message],opts)

hipchat.report

rescue Errno::ENOENT => err

abort "hip_chat_cli: \#{err.message}"

end

Script used on server

following is a script which should which we need to configure to use this code with a icinga server

Service notification

root@hubot0:/etc/icinga2/scripts\# cat hipchat-service-notification.sh

#!/bin/sh

template=\`cat <<TEMPLATE

\*\*\*\*\* Icinga \*\*\*\*\*

Notification Type: \$NOTIFICATIONTYPE

Service: \$SERVICEDESC

Host: \$HOSTALIAS

Address: \$HOSTADDRESS

State: \$SERVICESTATE

Date/Time: \$LONGDATETIME

Additional Info: \$SERVICEOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

TEMPLATE

\`

\#/usr/bin/printf "%b" "\$template" | mail -s "\$NOTIFICATIONTYPE - \$HOSTDISPLAYNAME - \$SERVICEDISPLAYNAME is \$SERVICESTATE" \$USEREMAIL

dir="\$(readlink -f \$(dirname \$0))"

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "\$SERVICESTATE"

Host notification

#root@hubot0:/etc/icinga2/scripts\# cat mail-host-notification-hipchat.sh

#!/bin/sh

dir="\$(readlink -f \$(dirname \$0))"

template=\`cat <<EOF

HOST DOWN \$HOSTALIAS; Address: \$HOSTADDRESS; State: \$HOSTSTATE ; Date/Time: \$LONGDATETIME

Additional Info: \$HOSTOUTPUT

Comment: [\$NOTIFICATIONAUTHORNAME] \$NOTIFICATIONCOMMENT

EOF

\`

ruby \$dir/notify --message "\$(/usr/bin/printf "%b" "\$template")" --name icinga --apitoken "<TOKEN>" --roomname 2614946 --alerttype "warning"

root@hubot0:/etc/icinga2/scripts\#

Change in command.conf for Icinga server

This will change the Icinga server to support the hipchat adapter ,
root@hubot0:/etc/icinga2/scripts\# cat ../conf.d/commands.conf

/\* Command objects \*/

object NotificationCommand "mail-host-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/mail-host-notification-hipchat.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

HOSTSTATE = "\$host.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

HOSTOUTPUT = "\$host.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

object NotificationCommand "mail-service-notification" {

import "plugin-notification-command"

command = [ SysconfDir + "/icinga2/scripts/hipchat-service-notification.sh" ]

env = {

NOTIFICATIONTYPE = "\$notification.type\$"

SERVICEDESC = "\$service.name\$"

HOSTALIAS = "\$host.display_name\$"

HOSTADDRESS = "\$address\$"

SERVICESTATE = "\$service.state\$"

LONGDATETIME = "\$icinga.long_date_time\$"

SERVICEOUTPUT = "\$service.output\$"

NOTIFICATIONAUTHORNAME = "\$notification.author\$"

NOTIFICATIONCOMMENT = "\$notification.comment\$"

HOSTDISPLAYNAME = "\$host.display_name\$"

SERVICEDISPLAYNAME = "\$service.display_name\$"

USEREMAIL = "\$user.email\$"

}

}

Example notification

Roadmap

  1. Add hipchat user for Icinga server
  2. Configure to talk to groups using token from user settings in
    hipchat
  3. Change notification files and update token+room_ids
All these steps are required to make sure hubot can take actions on events. For now hubot can not take actions if the user Hubot and notifying user is matched. It is required to prevent race conditions.

Labels:

Comments

Post a Comment

Popular posts

Helm generic springboot templates

With the dramatically increasing demand for container orchestration specifically Kubernetes, demand to template K8S manifests(Json/Yaml) also came to light. To handle increasing manifests, new CRDs(Custom resource definition), etc… it became obvious that we need a package manager somewhat like yum, apt, etc… However, the nature of Kubernetes manifest is very different than what one used to have with Yum and Apt. These manifests required a lot of templates which is now supported by Helm, a tool written in GoLang with custom helm functions and pipelines. Neutral background on templating Templating has been a driver for configuration management for a long time. While it may seem trivial for users coming from Ansible, Chef, Puppet, Salt, etc…, it is not. Once one moves to Kubernetes, the very first realization is hard declarative approach that Kubernetes follows. It is difficult to make generic templating with declarative form since each application may have some unique feature and r
Post a Comment
Read more

Allowing users to have ssh access

Allowing users to have ssh access Hi Readers, It is one of the tasks we need to complete in order to allow users to log-in into your server without compromising your security. We are going to accomplish the following tasks, 1.) Allow the users for given domain only 2.) Must allow access to a given domain 3.) Block access for a specific domain. These questions are asked in RedHat certification examination RHCE6 We are going to complete the above mention task using iptables To give proper example we are taking 192.168.20.0/255.255.255.0 as our domain and 192.168.21.0/255.255.255.0 as other domain. Assuming that your system is a fresh installation we can remove all rules previously applied. # iptables -F  The above mention command will flush all the previously applied rules. Insert a rule in your input chain by below mention command, # iptables -I INPUT -s <ip of your domain>/<subnet mask> -p <protocal tcp/udp> --dport <port> -j AC
Post a Comment
Read more

Istio multicluster, gotchas ....

istio.md Istio lets you connect, secure, control, and observe services. At a high level, Istio helps reduce the complexity of these deployments, and eases the strain on your development teams. It is a completely open source service mesh that layers transparently onto existing distributed applications. It is also a platform, including APIs that let it integrate into any logging platform, or telemetry or policy system. Istio’s diverse feature set lets you successfully, and efficiently, run a distributed microservice architecture, and provides a uniform way to secure, connect, and monitor microservices. In context of Vuclip istio allows us to reduce the code and environment configurations while keeping the similar or more feature sets at our disposal. Since istio is designed to bridge the gap for both development teams and SRE, it is essential to see and visualize that in practice. Istio will affect us in our ability to connect , secure(HTTPs TLS, mtls [Phase-2]), control(external comm
Post a Comment
Read more

Mysql Scripts in Linux Command Line Terminal

In MySQL user always tries to connect with the server via command line interface such as remote login like 'ssh' then they troubled themselves in using graphical interface and get the required output quickly. So they need to automate that task. In this Blog I am going to suggest you a way to do this ... First of all we need to know that a MySQL service is kept in /etc/init.d/mysqld and we need to invoke that in order to use mysql server and client. To do this use following command in terminal : service /etc/init.d/mysqld start or service mysqld start When mysql server is started make a directory like /home/anduril/shubham_Script and after entering into that directory use this command in your terminal. vi connect_string.sh It will create a file with the name connect_string.sh you can give any name you want. Press 'i' to edit and enter the following text : mysqlshow -u root -proot mysqladmin version -u root -proot mysqladmin variables -u root -proot mysqladmin ping -u roo
Post a Comment
Read more

Enter your email address: