Hubot and Hipchat

Many of us have struggle with concept of L1, and L2 support and maintenance which can be very expensive and transitioning is a living hell. What we wish is the ability for both of them to work together and make a smooth transition.

So I have started to think it would be cool to make this happen for a organization which runs 1200+ servers and have teams to support product and infra both. The Idea was to start with collaboration between teams which can be extended to request and support basis. To extend it further we discovered  that product team works for 24x7 and infra team is 18x7. Now it is important to design it in a way that product team can manage it 24x7 with minimum and no help from infra team(L2). 

Solution:

Solution was to use a collaboration tool which supports API for our tools to maintain alert level. For that please refer my previous blog describing configuration of Icinga and Hipchat. Next logical step is to configure hipchat to support robot system which can perform housekeeping like infra, with appropiate roles and permission. This is where we discovered Hubot a robot which can respond to certain request on chat channels. An opensource tool written in NodeJS, with a community support. So we can pretty much do anything that NodeJS has to offer us. 

Installation of Hubot: 

In order to install hubot we need to install NodeJS+Npm first, which is out of scope of this document. We are going to use Redis-brain so we need to install redis-server as well. With this little pre-configuration lets start the installation of hubot:   

1. npm install -g coffee-script
2. sudo apt-get install redis-server
3. npm install -g hubot
4. hubot -v # to verify
5. apt-get install libexpat1-dev libicu-dev
6. npm install --save hubot-hipchat
7. npm install -g yo

Steps above will allow you to install hubot, now we need to write our scripts to support our requirements. Now I am hoping that you might be having a configuration tool to manage this infra already, it could be anything from Puppet, CHEF, Ansible to in house tool. If you are not having any such tool, well life pretty much sucks. 

Now i am presuming that you do not have any such tools in place, then your system admin is a traditional system admin, aka GOD. Let write a custom script for GOD,

Lets create a file in script folder inside hubot and name it execute-script.coffee

util= require 'util'
fs = require 'fs'
path= require 'path'
module.exports = (robot) ->
    robot.respond /execute-command (.*?)( on (.*))?$/i, (msg) ->
        msg.send "MSG_ENVELOPE: #{msg.envelope.user.roles} #{process.env.HUBOT_AUTH_ADMIN} "
        if robot.auth.hasRole(msg.envelope.user,'superadmin')
            # msg.send "Command is :  #{msg.match[1]} for host: #{msg.match[3]}"
            #command = "ssh hubot@#{msg.match[3]} #{msg.match[1]}"
            command = "parallel-ssh -i  -O StrictHostKeyChecking=no  -l hubot -H \"#{msg.match[3]}\" -v -i \"#{msg.match[1]}\" "
            @exec = require('child_process').exec
            msg.send "This is the command #{command}."

            @exec command, (error, stdout, stderr) ->
                msg.send error
                msg.send stdout
                msg.send stderr
        else 
            msg.send "To execute these commands you need to be part of superadmin group"

This small script will allow you to login into any of the system in infra controlled by ssh keys and and hubot group role. Of course you have to install role module given by hubot first. 

# npm install "hubot-auth" --save 

The above will write the code to package.json and manage an entry inside external-scripts.json file as well.

Lets test it now,

Since this is configured with paralled-ssh this will allow you to manage whole set of servers using their ip. This gives us some control and audit out of box with custom roles based on room and role. 

Happy chatting.