nagios-herald 0.0.2

data/docs/formatters.md

@@ -0,0 +1,180 @@
+# Formatters
+
+Adding context to alerts is done by the formatters. Formatters generate all the content that may
+be used by one or more message types. For example, text returned by a Nagios check
+can be highlighted to grab the operator's attention. Helper functions can be called to generate
+additional content such as image attachments or search results.
+
+All content generated by a formatter is stored in a hash (a class instance variable in the
+formatter object). The hash looks similar to the following:
+
+```ruby
+@content = {
+  :attachments => [],
+  :html => {
+    :additonal_info => "",
+    :additional_details => "" 
+  },
+  :subject => "",
+  :text => {
+    :additonal_info => "",
+    :additional_details => ""
+  }
+}
+```
+
+**NOTE**: The subkeys under the ``:html`` and ``:text`` keys are generated by formatter methods of the
+same name.
+
+## Formatter Methods
+
+The base formatter (``Formatter::Base``) defines a set of core formatting methods useful for generating content.
+These methods can be overridden in subclassed formatters to do anything. **The only limit is your
+imagination.**
+
+    ack_info
+    additional_details
+    additional_info
+    alert_ack_url
+    host_info
+    notes
+    notification_info
+    recipients_email_link
+    short_ack_info
+    short_state_detail
+    state_info
+
+``Formatter::Base`` also includes basic methods that are used to store or manipulate content in the
+``@content`` hash:
+
+    add_attachments - Add an attachment path to be referenced in an Email message.
+    add_html - Concatenate HTML for the given section.
+    add_text - Concatenate text for the given section.
+    line_break - Generate a line break for both text and HTML content.
+
+See below for an example of content generated using some of the above methods.
+
+![nagios-herald-formatter-content-example.png](/docs/images/nagios-herald-formatter-content-example.png)
+
+## Writing the Formatter
+
+To write a formatter, create a new formatter class (i.e. ``check_disk.rb``) and inherit from ``NagiosHerald::Formatter``.
+
+```ruby
+module NagiosHerald
+  module Formatter
+    class CheckDisk < NagiosHerald::Formatter
+      include NagiosHerald::Logging
+
+      def additional_details
+      section = __method__  # this defines the section key in the formatter's content hash
+      add_text(section, "nagios-herald makes alerting more bearable.")
+      add_html(section, "<b>nagios-herald</b> makes alerting more bearable.<br>")
+      end
+
+    end
+  end
+end
+```
+
+### Naming Convention
+
+* Built-in formatters live in ``lib/nagios-herald/formatters/``.
+* Custom formatters (those **you'll** write) can live in any location.
+ * Specify the location of your custom formatters via the ``--formatter-dir`` option
+ on the command line or the ``formatter_dir`` variable in the configuration file.
+* The file names **MUST** lower-cased and underscored.
+* The class names **MUST** be CamelCased.
+
+For example, our ``CheckDisk`` formatter class would reside in a file named ``check_disk.rb``.
+
+This is necessary as ``nagios-herald`` preloads all known formatters when it starts so that it can
+dynamically instantiate the appropriate formatter it needs when it's ready.
+
+### Overriding Methods
+
+Any of the core formatting methods can be overridden in your subclass.  It's recommended that the helper methods
+(i.e. ``add_(attachment|html|text)`` not be overridden.
+
+An example of an overridden ``additional_info`` method could be:
+
+```ruby
+def additional_info
+  section = __method__  # this defines the section key in the formatter's content hash
+  hostname  = get_nagios_var("NAGIOS_HOSTNAME")
+  add_text(section, "#{hostname} blew up!")
+  add_html(section, "#{hostname} <b>blew</b> up!")
+end
+```
+
+## Helpers
+
+Helpers are libraries available to all formatters that can be used to supplement the content they generate. For example, a helper can pull in external information (such as a graph) that is relevant to a service that Nagios is alerting on.
+
+To learn more, see the [helpers](/docs/helpers.md) page.
+
+## Testing Your Formatter
+
+**PLEASE TEST YOUR FORMATTER. NOT DOING SO INCREASES THE POSSIBILITY THAT A NEW FORMATTER WILL PREVENT DELIVERY OF CRITICAL ALERTS.**
+
+There are two ways that formatters can be tested: unit tests and manually running ``nagios-herald``.
+
+### Unit Tests
+
+**THIS IS A WORK IN PROGRESS. BETTER TESTING IS ON THE ROADMAP**
+
+### ``nagios-herald``
+
+``nagios-herald`` can be called manually from the command line to test new formatters:
+
+```
+./bin/nagios-herald --env-file ../test/env_files/check_mem.vars --formatter=check_mem -r ops@example.com -y nagios@example.com --message-type email -u http://nagios.example.com --trace
+```
+
+For a full listing of available options, run ``nagios-herald --help``.
+
+### A Note About Nagios Data
+
+Nagios stores important information in environment variables.  The formatter methods can retrieve that
+information by using the ``get_nagios_var()`` method.  For reference, see an
+example [environment file](/test/env_files/nagios_vars.EXAMPLE).
+
+**NOTE**: Do not directly call ``ENV['YOUR_VAR']`` in your Ruby code as it will be harder to test.
+
+### Testing the Formatter with Offline Data
+
+Because Nagios stores information in environment variables that are generated during runtime, ``nagios-herald``
+provides a few example environment files that can be used for testing. Files in ``test/env_files/`` can be specified via the
+``--env-file`` argument to mimic an alerting event.  During normal operation ``nagios-herald`` grabs
+the information it needs from Nagios' environment variables.
+
+One can generate environment files for testing by using the [dump_nagios_env.sh](/docs/tools.md#dump_nagios_env.sh) tool.
+
+**NOTE**: ``--no-send`` forces ``nagios-herald`` to output content to the terminal.
+
+```
+./bin/nagios-herald --no-send --env-file ../test/env_files/check_disk.vars --formatter=check_disk -r ops@example.com -y nagios@example.com --message-type email -u http://nagios.example.com --trace
+------------------
+Subject : ** PROBLEM Service ALERT: web.example.com/Disk Space is CRITICAL **
+------------------
+Host: web.example.com
+Service: Disk Space
+
+State is now: CRITICAL for 0d 0h 5m 12s (was CRITICAL) after 3 / 3 checks
+
+Additional info:
+ DISK CRITICAL - free space: / 7002 MB (82% inode 60%): /data 16273093 MB (99% inode 99%):
+
+Additional Details:
+Filesystem            Size  Used Avail Use% Mounted on
+/dev/sda               40G   33G  7.2G  82% /
+tmpfs                 2.5G   83M  2.4G   4% /dev/shm
+/dev/sdb               16G   16G  158M  99% /data
+
+Sent to ops-engineer
+Notification sent at: Thu May 16 21:06:38 UTC 2013 (notification number 1)
+
+Acknowledge this alert: http://nagios.example.com/nagios/cgi-bin/cmd.cgi?cmd_typ=34&host=web.example.com&service=Disk%20Space%0A%3Cbr%3E
+
+------------------
+```

data/docs/helpers.md

@@ -0,0 +1,12 @@
+# Helpers
+
+Helpers are libraries available to all formatters that can be used to supplement the content they generate. For example, a helper can pull in external information (such as a graph) that is relevant to a service that Nagios is alerting on.
+
+``nagios-herald`` comes with the following helpers as examples:
+
+* GangliaGraph - Downloads relevant graphs to be embedded in (email) messages.
+* SplunkReporter - Queries Splunk for information to add to an alert.
+* LogstashQuery - Queries logstash with either a kibaba-style query or a file containing query JSON
+* UrlImage - Generic helper to download images.
+
+Helpers are located in ``lib/nagios-herald/helpers/``.

data/docs/messages.md

@@ -0,0 +1,16 @@
+# Messages
+
+While [formatters](/docs/formatters.md) are crucial to contextualizing alerts, it doesn't matter much if the alerts are never sent. This is where messages come in to play.
+
+``nagios-herald`` provides a base Message class (``Message::Base``) from which all Message subclasses inherit.  ``Message::Base`` knows that all messages have content and recipients; it's the job of the Message subclasses to define additional behavior (such as how to send a message).
+
+Three message types are defined in ``nagios-herald``: Email, Pager, and IRC.  Feel free to cook up any new message types (i.e. CarrierPigeon, SmokeSignal) as you see fit.
+
+## Getting Content into Messages
+
+Content generated by formatters is assigned to a message in ``Executor`` via the following code:
+
+```ruby
+formatter.generate_message_content
+message.content = formatter.content
+```

data/docs/nagios-config.md

@@ -0,0 +1,74 @@
+# Nagios Configuration
+
+## Notification Command Configuration
+
+Nagios needs to be instructed to use ``nagios-herald`` for notifications. The main notification program in
+``nagios-herald`` is called ``nagios-herald``.
+
+To configure Nagios to generate alerts via ``nagios-herald``, update the relevant stanza(s) in ``commands.cfg``.
+The below example assumes ``nagios-herald`` is installed in ``/usr/local/bin``.
+
+```
+# email
+define command {
+    command_name    notify-host-by-email
+    command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type email --formatter=$_HOSTMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com
+}
+
+# specify the location of custom formatters
+define command {
+    command_name    notify-service-by-email
+    command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type email --formatter=$_SERVICEMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com --formatter-dir=/usr/local/nagios-herald-formatters
+}
+
+# pager
+define command {
+    command_name    notify-host-by-pager
+    command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type pager --formatter=$_HOSTMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com
+}
+
+define command {
+    command_name    notify-service-by-pager
+    command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type pager --formatter=$_SERVICEMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com
+}
+
+# IRC (optional)
+define command {
+    command_name    notify-service-by-irc
+    command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type irc --formatter=$_SERVICEMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com
+}
+
+define command {
+    command_name    notify-service-by-irc
+    command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type irc --formatter=$_SERVICEMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com
+}
+```
+
+**NOTE**: ``$_HOSTMESSAGE_FORMATTER_NAME$`` and ``$_SERVICEMESSAGE_FORMATTER_NAME$`` are Nagios macros, generated
+from a custom variable in a relevant service check definition.
+
+### Tracing Errors
+If ``nagios-herald`` is failing to notify, you can turn up the logging via the ``--trace`` option to gather
+more information, like so:
+
+```
+command_line    /usr/local/nagios-herald/bin/nagios-herald --message-type email --formatter=$_SERVICEMESSAGE_FORMATTER_NAME$ --nagios-cgi-url=http://nagios.example.com/nagios/cgi-bin/cmd.cgi --reply-to=nagios@example.com --trace
+```
+
+## Service Configuration
+
+To apply a formatter to a service check, add the ``_message_formatter_name`` custom variable in the relevant stanza in ``services.cfg``:
+The value of the variable is the name of the formatter.
+
+```
+define service {
+    hostgroup_name                  web_servers
+    service_description             Disk Space
+    notification_interval           20
+    check_command                   check_nrpe!check_disk
+    _message_formatter_name         check_disk            <<<< custom variable
+    contact_groups                  ops
+}
+```
+
+**NOTE**: If no formatter is specified, ``nagios-herald`` defaults to the base formatter and generates generic content for messages.

data/docs/tools.md

@@ -0,0 +1,79 @@
+# Tools
+
+This project comes with a few tools you can use that mimic the behavior of helpers and formatters.  These are
+really useful for testing functionality outside of ``nagios-herald``.
+
+The following tools reside under the ``bin/`` directory.
+
+<a name="dump_nagios_env.sh"></a>
+## ``dump_nagios_env.sh`` - Dump Nagios Environment
+
+``dump_nagios_env.sh`` is a handy script that dumps the Nagios environment variables at the time an alert fires.
+Follow the instuctions in the [script](/bin/dump_nagios_env.sh) to configure and use it.
+
+The environment file that is generated can be used to test new formatters.
+
+## ``draw_stack_bars`` - Draw Stack Bars 
+
+``draw_stack_bars`` can generate horizontal stacked bars.
+
+TODO: This needs better documentation. Are we missing command line arguments in the below output?
+
+```
+$ ./draw_stack_bars -h
+usage: draw_stack_bars [options] label=value label=value
+
+options:
+  -h, --help            show this help message and exit
+  -w WIDTH, --width=WIDTH
+                        Picture width
+  -o OUTPUT_FILE, --output=OUTPUT_FILE
+                        The file name to save the results in
+  --bar-area-ratio=BAR_AREA_RATIO
+                        How wide should the bar area be compared to the total
+                        width
+  --bar-height-ratio=BAR_HEIGHT_RATIO
+                        How tall should the bars be compared to their width
+  --bar-border=BAR_BORDER
+                        Width of the border around the bars
+```
+
+For example: ``./draw_stack_bars /var=10`` will generate:
+
+![stack bars](/docs/images/stack-bars.png)
+
+## ``get_ganglia_graph`` - Draw Ganglia Graphs
+
+``get_ganglia_graph`` will download one or more images of Ganglia graphs.
+
+```
+$ ./get_ganglia_graph -h
+Usage: get_ganglia_graph [-hmpr]
+
+get_ganglia_graph is used to generate images from Ganglia graphs.
+
+It takes one or more hostnames and a single metric to build the appropriate URI(s) from which to generate images.
+The script outputs the full path for each of the images that have been written.
+
+    -h, --host *HOST                 The FQDN of the host required to look up a metric/image in Ganglia. Specify multiple hosts with multiple '-h' arguments.
+    -m, --metric                     The name of the metric we want to see.
+    -p, --path                       An optional path in which to write the image files.
+                                     DEFAULT: /tmp
+    -r, --range                      The range of time for which the graph should look back.
+                                     Acceptable values are the same as thouse used by Ganglia (i.e '8hours', '8h, '1day', '1d', '3weeks', '3w')
+                                     DEFAULT: 8h (8 hours)
+
+EXAMPLES
+--------
+1. Passing a single hostname and metric to get_ganglia_graph:
+
+    get_ganglia_graph -h mysql01.example.com -m part_max_used
+
+2. Passing multiple hostnames and a metric to get_ganglia_graph:
+
+    get_ganglia_graph -h mysql01.example.com -h memcached01.example.com -m disk_free
+
+3. Including an optional time range (12 hours):
+
+    get_ganglia_graph -h mysql01.example.com -h memcached01.example.com -m disk_free -r 12h
+```

data/etc/config.yml.example

@@ -0,0 +1,14 @@
+# define the FQDN of servers we call on to provide context in notifications
+servers:
+  ganglia: ganglia.example.com
+  graphite: graphite.example.com
+splunk:
+  url: https://splunk.example.com:8089/services/search/jobs
+  username: splunkuser
+  password: splunkpass
+logstash:
+  url: http://logstash.example.com:9200
+  result_field_trucate: 200
+  num_results: 10
+logfile: /tmp/nagios-herald.log
+formatter_dir: /usr/local/nagios-herald-formatters

data/etc/readme.md

@@ -0,0 +1,2 @@
+# Chef
+Chef will create the config we need in this directory.

data/lib/nagios-herald/config.rb

@@ -0,0 +1,25 @@
+require 'app_conf'
+
+module NagiosHerald
+  module Config
+    extend self
+
+    @config = {}
+    attr_accessor :config
+
+    # Public: Load the configuration file for use globally.
+    #
+    # options - The options hash built from command-line arguments.
+    #
+    # Returns a hash of the parsed config file merged with the command line options.
+    def load(options = {})
+      abort("Config file not found #{options['config_file']}") unless File.exists? options['config_file']
+      @config = AppConf.new
+      @config.load(options['config_file'])
+      @config = @config.to_hash
+      @config.merge!(options)   # runtime options should override
+      @config
+    end
+
+  end
+end