nagios-herald 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+--- 
+SHA1: 
+  metadata.gz: 335b4477670958112568a92c559b299ee3c992bf
+  data.tar.gz: dd284b572df136d6afd4b18ec80312923a3b5b8f
+SHA512: 
+  metadata.gz: 711870cc088ec314c383a2a97f16a43a2af7c00284b3858f23dad0e00d8727ba284634f505c674f86cb4c09b602343c7f464c9e85b626a969c3880ab59216e3a
+  data.tar.gz: 4b2981598b689cf8390c62aae5bb863f6c916cdf90cf884a693835e33a585202f11043dc6781324475e55b8f6869315c89e7776dfd731a935b055570b295361f

data/.gitignore

@@ -0,0 +1,8 @@
+*.swp
+*.pyc
+.DS_STORE
+etc/config.yml
+mail.html
+*.gem
+.idea
+Gemfile.lock

data/.travis.yml

@@ -0,0 +1,9 @@
+language: ruby
+rvm:
+ - 1.9.2
+sudo: false
+cache: bundler
+before_install:
+ - "echo 'gem: --no-ri --no-rdoc' >> ~/.gemrc"
+before_script:
+ - bundle exec mailcatcher

data/CHANGELOG.md

@@ -0,0 +1,11 @@
+## 0.0.2 (23rd June, 2014)
+
+Features:
+
+    - LogstashQuery helper, check_logstash formatter, CHANGELOG added 
+    
+## 0.0.1 (6th June, 2014)
+
+Features:
+
+    - Initial Release

data/CONTRIBUTING.md

@@ -0,0 +1,28 @@
+# Contribute
+
+To contribute to ``nagios-herald``, you need to follow a few easy steps:
+
+1. Fork the repo.
+2. Clone your fork.
+3. Hack on your favorite bits like the formatters, helpers, etc.
+4. If you are adding new functionality, document it in its own file under ``docs/``.
+5. If necessary, rebase your commits into logical chunks, without errors.
+6. Verify your code by running the test suite, and adding additional tests if you can.
+7. Push the branch up to GitHub.
+8. Send a pull request.
+
+We'll do our best to get your changes in!
+
+## How to Run Tests
+
+The tests are written using Ruby's minitest.
+
+Name your tests with the ``test_`` prefix and place them in the ``test/unit/``
+directory.
+
+When you're ready to run your tests, simply run ``rake``:
+
+```
+% rake test
+```
+

data/Gemfile

@@ -0,0 +1,5 @@
+source "https://rubygems.org"
+gemspec
+
+gem 'rake'
+gem 'mailcatcher'

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) [year] [fullname]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,94 @@
+# nagios-herald [![Travis Build Status](https://travis-ci.org/etsy/nagios-herald.svg?branch=master)](https://travis-ci.org/etsy/nagios-herald)
+
+``nagios-herald`` is a project that aims to make it easy to provide context in Nagios alerts.
+
+It was created from a desire to supplement an on-call engineer's awareness of conditions surrounding a notifying event. In other words, if a computer is going to page me at 3AM, I expect it to do some work for me to help me understand what's failing.
+
+## Customizing Nagios Alerts
+
+Nagios is a time-tested monitoring and alerting tool used by many Operations teams to keep an eye
+on the shop.  It does an excellent job of executing scheduled checks, determining when a threshold has been exceeded, and sending alerts.
+
+Past experience with Nagios has shown that, typically, those alerts provide little information beyond the fact that a host is down or a service is not responding as defined by check thresholds. It's bad enough to be woken up by an alert; it would make the on-call experience more bearable if the alerts could tell the engineer more about what's going on.  But what's useful in an alert?
+
+When notified, an engineer often performs a set of procedures to gather information about the event before attempting to correct it.  Imagine being able to automatically perform those procedures (or some subset) at the time of the alert. Imagine further, that the results of those procedures are embedded in the alert!
+
+Enter ``nagios-herald``!
+
+### Generic Nagios Alert
+
+Using the canonical (and oft-maligned) disk space check, here's an example notification:
+
+![vanilla-nagios-alert](/docs/images/vanilla-nagios.png)
+
+While it does provide necessary information, it could be formatted for better legibility.  For example,
+the following line, which contains the information we need, is dense and may be difficult to
+parse in the wee hours of the morning:
+
+    Additional Info: DISK WARNING - free space: / 1597 MB (8% inode=57%):
+    /dev/shm 24127 MB (100% inode=99%): /boot 152 MB (83% inode=99%):
+
+Common questions would be **"Which volume is problematic?"** or
+**"Why is this considered a 'WARNING' alert?"**  In this example, it's not readily apparent what
+those answers are.  Let's add that context with ``nagios-herald``.
+
+### Nagios Alert with Context
+
+``nagios-herald`` can **highlight and colorize text**, **embed images** (such as Ganglia graphs), **include search results**, and much more.
+
+The previous disk space alert example can be tailored to look like this:
+
+![html nagios email](docs/images/nagios-herald.png)
+
+Notice the handy **stack bar** that clearly illustrates the problematic volume?  See that **Ganglia graph**
+showing disk space utilization for the node in the last 24 hours. Curious why the alert fired?  Check
+the **highlighted ``df`` output** that neatly defines which threshold was exceeded and why.
+
+**NOTE**: In this example, the Nagios check ran ``df`` and supplied that input.
+
+### More Examples
+
+For more examples of ``nagios-herald`` in action, see the [example alerts](/docs/example_alerts.md) page.
+
+This is possible because ``nagios-herald`` provides extensible formatters.
+
+## Formatters
+
+Adding context to alerts is done by the formatters. Formatters generate all the content that may
+be used by one or more message types. For example, text returned by a Nagios check
+can be highlighted to grab the operator's attention.
+
+To learn more, see the [formatters](/docs/formatters.md) page.
+
+# Installing ``nagios-herald``
+
+Installation of ``nagios-herald`` is as easy as cloning this repository to a location of your choice.
+To enable ``nagios-herald`` to send notifications, [configure Nagios](/docs/nagios-config.md) and,
+optionally, write a [config.yml](/docs/config.md) file. At a minimum, specify the ``logfile`` configuration
+variable.
+
+## Dependencies
+
+### Ruby Gems
+
+``nagios-herald`` and its tools depend on the following Ruby gems:
+
+* ``app_conf``
+* ``choice``
+* ``mail``
+
+### Stack Bars
+
+Generating stack bars requires the following (which are included in this project for your convenience):
+
+* Python
+* Python Image Library (PIL)
+
+## Configuring Nagios
+
+To configure Nagios to use ``nagios-herald`` for notifications, see the [Nagios Configuration](/docs/nagios-config.md) page.
+
+### Icinga Support
+
+``nagios-herald`` supports Icinga too! Follow the instructions [here](/docs/config.md#icinga-support) to make it so.
+

data/Rakefile

@@ -0,0 +1,9 @@
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/unit/test_*.rb']
+  t.verbose = true
+end
+
+task :default => :test

data/bin/draw_stack_bars

@@ -0,0 +1,76 @@
+#!/usr/bin/env python
+
+# Draw a set of 2 values stack bars
+# Requires PIL and python > 2.6
+# Install PIL using: yum install python-imaging
+
+import sys
+import os
+from optparse import OptionParser
+
+try:
+    import PIL
+except ImportError:
+    sys.stderr.write("The Python Image Library (PIL) is required to use the draw_stack_bar module")
+    sys.stderr.write(" - you may try : 'sudo pip install pillow'\n")
+    sys.exit(1)
+
+lib_path = os.path.normpath(os.path.join(os.path.dirname(__file__), '../', 'lib', 'stackbars'))
+sys.path.append(lib_path)
+from grouped_stackbars import GroupedStackbars
+
+def parseData(data):
+    series = []
+    for i in data:
+        label, value = i.split('=')
+        try:
+            series.append((label.strip(), int(value)))
+        except:
+            print "Invalid format - value is not a number %s in '%s'" % (value, i)
+            sys.exit(1)
+    return series
+
+def drawStackBars(series, options):
+    stacked_bars = GroupedStackbars(options)
+    stacked_bars.render(series)
+    stacked_bars.save(options.get('output_file'))
+
+def parse_options():
+    usage = "usage: %prog [options] label=value label=value"
+
+    parser = OptionParser(usage = usage)
+
+    parser.add_option("-w", "--width", type="int",
+                      help="Picture width",
+                      dest="width", default=300)
+
+    parser.add_option("-o", "--output", type="string",
+                      help="The file name to save the results in",
+                      dest="output_file", default='stacked_bars.png')
+
+    parser.add_option("--bar-area-ratio", type="float",
+                      help="How wide should the bar area be compared to the total width",
+                      dest="bar_area_ratio", default=.75)
+
+    parser.add_option("--bar-height-ratio", type="float",
+                    help="How tall should the bars be compared to their width",
+                    dest="bar_height_ratio", default=.2)
+
+    parser.add_option("--bar-border", type="int",
+                    help="Width of the border around the bars",
+                    dest="bar_border", default=1)
+
+    options, data = parser.parse_args()
+    if not data:
+        sys.stderr.write("No  data to plot !\n\n")
+        parser.print_help()
+        sys.exit(1)
+    return options, data
+
+if __name__=='__main__':
+    options, data = parse_options()
+    series = parseData(data)
+    drawStackBars(series, vars(options))
+
+
+

data/bin/dump_nagios_env.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Dump the environment as Nagios sees it; useful for generating environment files to test with new formatters
+#
+# CONFIG NOTES
+# In commands.cfg, add a definition similar to the following:
+#
+# 'dump-env' command definition; dump Nagios' environment (use sparingly)
+#define command {
+#        command_name    dump-env
+#        command_line    /usr/local/nagios-herald/bin/dump_nagios_env.sh
+#}
+#
+# In contacts.cfg, append the 'dump-env' short command to one's 'service_notification_commands' like so:
+#
+#define contact {
+#        contact_name                    ops
+#        alias                           Ops Engineer
+#        email                           ops@example.com
+#        use                             generic-contact
+#        service_notification_commands   notify-service-by-email,dump-env
+#}
+
+LOGFILE="/tmp/nagios_env_$(date +%Y%m%d).log"
+date >> $LOGFILE
+env >> $LOGFILE

data/bin/get_ganglia_graph

@@ -0,0 +1,82 @@
+#!/usr/bin/env ruby
+
+require 'choice'
+require 'app_conf'
+
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'nagios-herald'
+
+program_name = File.basename($0)
+
+Choice.options do
+  header ""
+  header "#{program_name} is used to generate images from Ganglia graphs."
+  header ""
+  header "It takes one or more hostnames and a single metric to build the appropriate URI(s) from which to generate images."
+  header "The script outputs the full path for each of the images that have been written."
+  header ""
+
+  option :config_file do
+    short "-c"
+    long  "--config-file"
+    desc  "Specify an alternate location for the config file."
+    default File.expand_path(File.join(File.dirname(__FILE__), '..', 'etc', 'config.yml'))
+  end
+
+  option :hosts, :required => true do
+    short "-h"
+    long  "--host *HOST"
+    desc  "The FQDN of the host required to look up a metric/image in Ganglia. Specify multiple hosts with multiple \'-h\' arguments."
+  end
+
+  option :metric, :required => true do
+    short "-m"
+    long  "--metric"
+    desc  "The name of the metric we want to see."
+  end
+
+  option :path do
+    short "-p"
+    long  "--path"
+    desc  "An optional path in which to write the image files."
+    desc  "DEFAULT: /tmp"
+    default "/tmp"
+  end
+
+  option :range do
+    short "-r"
+    long  "--range"
+    desc  "The range of time for which the graph should look back."
+    desc  "Acceptable values are the same as thouse used by Ganglia (i.e '8hours', '8h, '1day', '1d', '3weeks', '3w')"
+    desc  "DEFAULT: 8h (8 hours)"
+    default "8h"
+  end
+
+  footer ""
+  footer "EXAMPLES"
+  footer "--------"
+  footer "1. Passing a single hostname and metric to #{program_name}:"
+  footer ""
+  footer "#{program_name} -h mysql01.example.com -m part_max_used"
+  footer ""
+  footer "2. Passing multiple hostnames and a metric to #{program_name}:"
+  footer ""
+  footer "#{program_name} -h mysql01.example.com -h memcached01.example.com -m disk_free"
+  footer ""
+  footer "3. Including an optional time range (12 hours):"
+  footer ""
+  footer "#{program_name} -h mysql01.example.com -h memcached01.example.com -m disk_free -r 12h"
+  footer ""
+end
+
+options = Choice.choices
+
+# Load the config
+abort("Config file not found #{options.config_file}") unless File.exists? options.config_file
+config = AppConf.new
+config.load( options.config_file )
+
+ganglia = NagiosHerald::Helpers::GangliaGraph.new
+
+image_paths = ganglia.get_graphs( options.hosts, options.metric, options.path, options.range )
+puts image_paths.join( ", " ) if image_paths

data/bin/get_graph

@@ -0,0 +1,50 @@
+#!/usr/bin/env ruby
+
+require 'choice'
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'nagios-herald'
+
+program_name = File.basename($0)
+
+Choice.options do
+  header ""
+  header "#{program_name} is used to generate images from given URIs."
+  header ""
+  header "It takes one or more URIs from which to generate images."
+  header "The script outputs the full path for each of the images that have been written."
+  header ""
+
+  option :path do
+    short "-p"
+    long  "--path"
+    desc  "An optional path in which to write the image files."
+    desc  "DEFAULT: /tmp"
+    default "/tmp"
+  end
+
+  option :uris, :required => true do
+    short "-u"
+    long  "--uri *URI"
+    desc  "The URI from which to generate an image. Specify multiple URIs with multiple \'-u\' arguments."
+  end
+
+  footer ""
+  footer "EXAMPLES"
+  footer "--------"
+  footer "1. Passing a single URI to #{program_name}:"
+  footer ""
+  footer "#{program_name} -u \"http://graphite.example.com/render/?target=mysql.connections.count\""
+  footer ""
+  footer "2. Passing multiple URIs to #{program_name}:"
+  footer "(line breaks added for clarity)"
+  footer ""
+  footer "#{program_name} -u \"http://graphite.example.com/render/?target=mysql.connections.count\" \\"
+  footer " -u \"http://graphite.example.com/render/?target=apache.requests_per_second\""
+  footer ""
+end
+
+path = Choice[:path] if Choice[:path]
+uris = Choice[:uris] if Choice[:uris]
+
+image_paths = NagiosHerald::Helpers::UrlImage.download_images( uris, path )
+puts image_paths.join( ", " )

data/bin/get_graphite_graph

@@ -0,0 +1,58 @@
+#!/usr/bin/env ruby
+
+require 'choice'
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'nagios-herald'
+
+program_name = File.basename($0)
+
+Choice.options do
+  header ""
+  header "#{program_name} is used to generate images from Graphite URIs."
+  header ""
+  header "It takes a URI from which to generate a Graphite image."
+  header "Optionally, an historical graph can be generated to compare to the first."
+  header "The script outputs the full path for each of the images that have been written."
+  header ""
+
+  option :historical_graph do
+    short "-h"
+    long  "--historical-graph"
+    desc  "Generate an historical graph to compare to the original image (i.e. past 24-hour period)."
+    default false
+  end
+
+  option :path do
+    short "-p"
+    long  "--path"
+    desc  "An optional path in which to write the image files."
+    desc  "DEFAULT: /tmp"
+    default "/tmp"
+  end
+
+  option :uri, :required => true do
+    short "-u"
+    long  "--uri"
+    desc  "The URI from which to generate an image."
+  end
+
+  footer ""
+  footer "EXAMPLES"
+  footer "--------"
+  footer "1. Passing a URI to #{program_name}:"
+  footer ""
+  footer "#{program_name} -u \"http://graphite.example.com/render/?target=mysql.connections.count\""
+  footer ""
+  footer "2. Passing a URI to #{program_name} and generating an historical graph:"
+  footer ""
+  footer "#{program_name} -u \"http://graphite.example.com/render/?target=mysql.connections.count\" -h"
+  footer ""
+end
+
+historical_graph = true if Choice[:historical_graph]
+path = Choice[:path]
+uri = Choice[:uri]
+
+graphite = NagiosHerald::Helpers::GraphiteGraph.new
+image_paths = graphite.get_graph(uri, path, historical_graph)
+puts image_paths.join(", ") if image_paths

data/bin/nagios-herald

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'nagios-herald'
+require 'nagios-herald/executor'
+
+NagiosHerald::Executor.new.announce

data/bin/splunk_alert_frequency

@@ -0,0 +1,54 @@
+#!/usr/bin/env ruby
+
+#
+# splunk_alert_frequency.rb
+#
+
+require 'choice'
+require 'app_conf'
+$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..', 'lib')
+require 'nagios-herald'
+
+program_name = File.basename( $0 )
+Choice.options do
+  header ""
+  header "#{program_name} will search Splunk for Nagios events, that match the given service name, in the past n days."
+  header ""
+
+  option :duration do
+    short "-d"
+    long  "--duration"
+    desc  "The number of days to search back through Nagios events. DEFAULT: 1 day"
+    default 1
+  end
+
+  option :hostname, :required => true do
+    short "-h"
+    long  "--hostname"
+    desc  "[REQUIRED] The hostname to search for. Expects the Etsy Nagios-type hostname (i.e. web0200.ny4)."
+    desc  "NOTE: If '--service-name' is not specified, will look for alert where this host was in the DOWN state."
+  end
+
+  option :service_name do
+    short "-s"
+    long  "--service-name"
+    desc  "The name of the service to search for.  It may be necessary to surround the service name with double quotes."
+    default nil
+  end
+
+  option :config_file do
+    short "-c"
+    long  "--config-file"
+    desc  "Specify an alternate location for the config file."
+    default File.expand_path(File.join(File.dirname(__FILE__), '..', 'etc', 'config.yml'))
+  end
+
+  footer ""
+end
+
+config  = AppConf.new
+config.load( Choice[:config_file] )
+
+reporter = NagiosHerald::Helpers::SplunkReporter.new(config['splunk']['url'],config['splunk']['username'], config['splunk']['password'])
+puts reporter.get_alert_frequency(Choice[:hostname], Choice[:service_name], {:duration => Choice[:duration]})
+