fluent-plugin-sensu 0.0.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in fluent-plugin-sensu.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,13 @@
+Copyright (c) 2015- MIYAKAWA Taku
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

data/README.md

@@ -0,0 +1,344 @@
+# fluent-plugin-sensu
+
+[Fluentd](http://fluentd.org) output plugin to send check results to
+[sensu-client](https://sensuapp.org/docs/latest/clients),
+which is an agent process of Sensu monitoring framework.
+
+## Configuration
+
+### Example configuration
+
+```apache
+<match ddos>
+  type sensu
+
+  # Connection settings
+  server localhost
+  port 3030
+
+  # Payload settings
+
+  ## The check is named "ddos_detection"
+  check_name ddos_detection
+
+  ## The severity is read from the field "level"
+  check_status_field level
+</match>
+```
+
+### Plugin type
+
+Tye type of this plugin is `sensu`.
+Specify `type sensu` in the match section.
+
+### Connection setting
+
+* `server` (default is "localhost")
+  * The IP address or the hostname of the host running sensu-client daemon.
+* `port` (default is 3030)
+  * The TCP port number of the [Sensu client socket](
+    https://sensuapp.org/docs/latest/clients#client-socket-input)
+    on which sensu-client daemon is listening.
+
+### Check payload setting
+
+The payload of a check result is a JSON object
+which contains attributes as follows.
+Attributes are indicated
+by [JSONPath](http://goessner.net/articles/JsonPath/) expressions.
+
+* `$.name`
+  * The check name to identify the check.
+* `$.output`
+  * An arbitrary string to describe the check result.
+  * This attribute is often used to contain metric values.
+* `$.status`
+  * The severity of the check result.
+  * 0 (OK), 1 (WARNING), 2 (CRITICAL) or 3 (UNKNOWN or CUSTOM).
+
+The check result can also contain other attributes.
+This plugin supports the attributes below.
+
+* `$.type`
+  * Either "standard" or "metric".
+  * If the attribute is set to "standard", the sensu-server creates an event
+    only when the status is not OK or when the status is changed to OK.
+  * If the attribute is set to "metric", the sensu-server creates an event
+    even if the status is OK.
+    It is useful when Sensu sends check results to metrics collectors
+    such as [Graphite](http://graphite.wikidot.com/).
+* `$.ttl`
+  * The time to live (TTL) in seconds,
+    until the check result is considered stale.
+    If TTL expires, sensu-server creates an event.
+  * This attribute is useful when you want to be notified
+    when logs are not output for a certain period.
+  * Same as `freshness_threshold` in Nagios.
+* `$.handlers`
+  * The names of handlers which process events created for the check.
+* `$.low_flap_threshold` and `$.high_flap_threshold`
+  * Threshold percentages to determine the status is considered "flapping,"
+    or the state is changed too frequently.
+  * Same as the options in Nagios.
+    See the description of [Flap Detection](
+    https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/flapping.html)
+    in Nagios.
+* `$.source`
+  * The source of the check, such as servers or network switches.
+  * If this attribute is not specified,
+    the host of sensu-client is considered as the source.
+* `$.executed`
+  * The timestamp on which the check is executed.
+  * Note that there is also another timestamp attribute named `issued`,
+    which is automatically measured by sensu-client process.
+    Uchiwa, the default dashboard of Sensu, displays `issued`
+    as the timestamp of check results.
+
+This plugin additionally adds "fluentd" attribute to the check result.
+The value of the attribute is a JSON object
+whoes elements are input to the plugin.
+
+* `$.fluentd.tag`
+  * The tag of the Fluentd data.
+* `$.fluentd.time`
+  * The time of the Fluentd data, in seconds since the Unix epoch.
+* `$.fluentd.record`
+  * The record of the Fluentd data.
+
+#### `name` attribute
+
+The check name is determined as below.
+
+1. The field specified by `check_name_field` option,
+   if present and valid (highest priority)
+  * The valid values are strings composed of ASCII alphanumerics,
+    underscores, periods, and hyphens.
+2. or `check_name` option, if present
+  * The valid values are same as above.
+3. or the tag name, if valid
+  * The valid values are same as above.
+4. or "fluent-plugin-sensu" (lowest priority)
+
+#### `output` attribute
+
+The check output is determined as below.
+
+1. The field specified by `check_output_field` option,
+   if present (highest priority)
+2. or `check_output` option, if present
+3. or JSON notation of the record (lowest priority)
+
+#### `status` attribute
+
+The severity of the check result is determined as below.
+
+1. The field specified by `check_status_field` option,
+   if present and permitted (highest priority)
+  * The values permitted to the field for each status (case insensitive):
+    * status 0: an integer ``0``
+      and strings ``"0"``, ``"OK"``
+    * status 1: an integer ``1``
+      and strings ``"1"``, ``"WARNING"``, ``"warn"``
+    * status 2: an integer ``2``
+      and strings ``"2"``, ``"CRITICAL"``, ``"crit"``
+    * status 3: an integer ``3``
+      and strings ``"3"``, ``"UNKNOWN"``, ``"CUSTOM"``
+2. or `check_status` option, if present
+  * The permitted values for each status (case insensitive):
+    * status 0: ``0`` and ``OK``
+    * status 1: ``1``, ``WARNING``, ``warn``
+    * status 2: ``2``, ``CRITICAL``, ``crit``
+    * status 3: ``3``, ``UNKNOWN``, ``CUSTOM``
+  * If the value is not permitted, it causes a configuration error.
+3. or `3`, which means UNKNOWN or CUSTOM (lowest priority)
+
+"warn" and "crit" come from
+[fluent-plugin-notifier](https://github.com/tagomoris/fluent-plugin-notifier).
+
+#### `type` attribute
+
+The check type is determined as below.
+
+1. `check_type` option (highest priority)
+  * The value must be a string ``"standard"`` or ``"metric"``.
+2. or "standard" (lowest priority)
+
+#### `ttl` attribute
+
+The TTL seconds till expiration is determined as below.
+
+1. `check_ttl` option (highest priority)
+  * The value must be an integer which represents the TTL seconds.
+2. or N/A (lowest priority)
+  * It means no expiration detection is performed.
+
+#### `handlers` attributes
+
+The handlers which process check results are determined as below.
+
+1. `check_handlers` option (highest priority)
+  * The value must be an array of strings which represent handler names.
+2. or `["default"]` (lowest priority)
+
+#### `low_flap_threshold` and `high_flap_threshold` attributes
+
+The threshold percentages for flap detection are determined as below.
+
+1. `check_low_flap_threshold`
+   and `check_high_flap_threshold` options (highest priority)
+  * The values must be integers of threshold percentages.
+2. or N/A (lowest priority)
+  * It means no flap detection is performed.
+
+The two options either must be specified together,
+not specified at all.
+
+If the options are specified,
+the following condition must be true:
+`0 <= check_low_flap_threshold <= check_high_flap_threshold <= 100`.
+
+#### `source` attribute
+
+The source of the checks is determined as below.
+
+1. The field specified by `check_source_field` option,
+   if present and valid (highest priority)
+2. or `check_source` option
+3. or N/A (lowest priority)
+  * It means the host of sensu-client is considered as the check source.
+
+#### `executed` attribute
+
+The executed timestamp is determined as below.
+
+1. The field specified by `check_executed_field`
+   if present and valid (highest priority)
+  * The value must be an integer
+    which represents seconds since the Unix epoch.
+2. The time of the Fluentd record (lowest priority)
+
+### Buffering
+
+The default value of `flush_interval` option is set to 1 second.
+It means that check results are delayed at most 1 second
+before being sent.
+
+Except for `flush_interval`,
+the plugin uses default options
+for buffered output plugins (defined in Fluent::BufferedOutput class).
+
+You can override buffering options in the configuration.
+For example:
+
+```apache
+<match ddos>
+  type sensu
+  ...snip...
+  buffer_type file
+  buffer_path /var/lib/fluentd/buffer/ddos
+  flush_interval 0.1
+  try_flush_interval 0.1
+</match>
+```
+
+## Use case: "too many server errors" alert
+
+#### Situation
+
+Assume you have a web server which runs:
+
+* Apache HTTP server
+* Fluentd
+* sensu-client
+  * which listens to the TCP port 3030 for [Sensu client socket](
+    https://sensuapp.org/docs/latest/clients#client-socket-input).
+
+You want to be notified when Apache responds too many server errors,
+for example 5 errors per minute as WARNING,
+and 50 errors per minute as CRITICAL.
+
+#### Configuration
+
+The setting for Fluentd utilizes
+[fluent-plugin-datacounter](https://github.com/tagomoris/fluent-plugin-datacounter),
+[fluent-plugin-record-reformer](https://github.com/sonots/fluent-plugin-record-reformer),
+and of course [fluent-plugin-sensu](https://github.com/miyakawataku/fluent-plugin-sensu).
+Install those plugins and add configuration as below.
+
+```apache
+# Parse Apache access log
+<source>
+  type tail
+  tag access
+  format apache2
+
+  # The paths vary by setup
+  path /var/log/httpd/access_log
+  pos_file /var/lib/fluentd/pos/httpd-access_log.pos
+</source>
+
+# Count 5xx errors per minute
+<match access>
+  type datacounter
+  tag count.access
+  unit minute
+  aggregate all
+  count_key code
+  pattern1 error ^5\d\d$
+</match>
+
+# Calculate the severity level
+<match count.access>
+  type record_reformer
+  tag server_errors
+  enable_ruby true
+  <record>
+    level ${error_count < 5 ? 'OK' : error_count < 50 ? 'WARNING' : 'CRITICAL'}
+  </record>
+</match>
+
+# Send checks to sensu-client
+<match server_errors>
+  type sensu
+  server localhost
+  port 3030
+
+  check_name server_errors
+  check_type standard
+  check_status_field level
+  check_ttl 100
+</match>
+```
+
+The TTL is set to 100 seconds here,
+because the check must be sent for each 60 seconds,
+plus 40 seconds as a margin.
+
+### Alternatives
+
+You can use [record\_transformer](
+http://docs.fluentd.org/articles/filter_record_transformer) filter
+instead of `fluent-plugin-record-reformer`
+on Fluentd 0.12.0 and above.
+
+If you are concerned with scalability,
+[fluent-plugin-norikra](https://github.com/norikra/fluent-plugin-norikra)
+may be a better option than datacounter and record\_reformer.
+
+Another alternative configuration for the use case is
+sending the error count to Graphite using [fluent-plugin-graphite](
+https://github.com/studio3104/fluent-plugin-graphite),
+and making Sensu monitor the value on Graphite with [check-data.rb](
+https://github.com/sensu/sensu-community-plugins/blob/master/plugins/graphite/check-data.rb).
+
+## Installation
+
+Install `fluent-plugin-sensu` gem.
+
+# Contributing
+
+Submit an [issue](https://github.com/miyakawataku/fluent-plugin-sensu/issues)
+or a [pull request](https://github.com/miyakawataku/fluent-plugin-sensu/pulls).
+
+Feedback to [@miyakawa\_taku](https://twitter.com/miyakawa_taku) on Twitter
+is also welcome.

data/Rakefile

@@ -0,0 +1,13 @@
+# -*- encoding: utf-8 -*-
+# vim: et sw=2 sts=2
+
+require "bundler/gem_tasks"
+require 'rake/testtask'
+
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test

data/fluent-plugin-sensu.gemspec

@@ -0,0 +1,28 @@
+# coding: utf-8
+# vim: et sw=2 sts=2
+
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |spec|
+  spec.name          = "fluent-plugin-sensu"
+  spec.version       = '0.0.1'
+  spec.authors       = ["MIYAKAWA Taku"]
+  spec.email         = ["miyakawa.taku@gmail.com"]
+  spec.description   = 'Fluentd output plugin to send checks' +
+                       ' to sensu-client.'
+  spec.summary       = spec.description
+  spec.homepage      = ""
+  spec.license       = "Apache License, v2.0"
+
+  spec.files         = `git ls-files`.split($/)
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
+  spec.require_paths = ["lib"]
+
+  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "rr"
+  spec.add_runtime_dependency "fluentd"
+end

data/lib/fluent/plugin/out_sensu.rb

@@ -0,0 +1,288 @@
+# -*- encoding: utf-8 -*-
+# vim: et sw=2 sts=2
+
+module Fluent
+
+  # Fluentd output plugin to send checks to sensu-client.
+  class SensuOutput < Fluent::BufferedOutput
+    Fluent::Plugin.register_output('sensu', self)
+
+    #
+    # Config parameters
+    #
+
+    # The IP address or the hostname of the host running sensu-client.
+    config_param :server, :string, :default => 'localhost'
+
+    # The port to which sensu-client is listening.
+    config_param :port, :integer, :default => 3030
+
+    # Options for "name" attribute.
+    config_param :check_name, :string, :default => nil
+    config_param :check_name_field, :string, :default => nil
+
+    # Pattern for check names.
+    CHECK_NAME_PATTERN = /\A[\w.-]+\z/
+
+    # Options for "output" attribute.
+    config_param :check_output_field, :string, :default => nil
+    config_param :check_output, :string, :default => nil
+
+    # Options for "status" attribute.
+    config_param :check_status_field, :string, :default => nil
+    config_param(:check_status, :default => 3) { |status_str|
+      check_status = SensuOutput.normalize_status(status_str)
+      if not check_status
+        raise Fluent::ConfigError,
+          "invalid 'check_status': #{status_str}; 'check_status' must be" +
+          " 0/1/2/3, OK/WARNING/CRITICAL/CUSTOM, warn/crit"
+      end
+      check_status
+    }
+
+    # Pattern for OK status.
+    OK_PATTERN = /\A(0|OK)\z/i
+
+    # Pattern for WARNING status.
+    WARNING_PATTERN = /\A(1|WARNING|warn)\z/i
+
+    # Pattern for CRITICAL status.
+    CRITICAL_PATTERN = /\A(2|CRITICAL|crit)\z/i
+
+    # Pattern for UNKNOWN status.
+    UNKNOWN_PATTERN = /\A(3|UNKNOWN|CUSTOM)\z/i
+
+    # Option for "type" attribute.
+    config_param(:check_type, :default => 'standard') { |check_type|
+      if not ['standard', 'metric'].include?(check_type)
+        raise Fluent::ConfigError,
+          "invalid 'check_type': #{check_type}; 'check_type' must be" +
+          ' either "standard" or "metric".'
+      end
+      check_type
+    }
+
+    # Option for "ttl" attribute.
+    config_param :check_ttl, :integer, :default => nil
+
+    # Option for "handlers" attribute.
+    config_param(:check_handlers, :default => ['default']) { |handlers_str|
+      error_message = "invalid 'check_handlers': #{handlers_str};" +
+                      " 'check_handlers' must be an array of strings."
+      obj = JSON.load(handlers_str)
+      raise Fluent::ConfigError, error_message if not obj.is_a?(Array)
+      array = obj
+      all_string_elements = array.all? { |e| e.is_a?(String) }
+      raise Fluent::ConfigError, error_message if not all_string_elements
+      array
+    }
+
+    # Options for flapping thresholds.
+    config_param :check_low_flap_threshold, :integer, :default => nil
+    config_param :check_high_flap_threshold, :integer, :default => nil
+
+    # Options for "source" attribute.
+    config_param :check_source_field, :string, :default => nil
+    config_param :check_source, :string, :default => nil
+
+    # Options for "executed" attribute.
+    config_param :check_executed_field, :string, :default => nil
+
+    # Load modules.
+    private
+    def initialize
+      super
+      require 'json'
+    end
+
+    # Read and validate the configuration.
+    public
+    def configure(conf)
+      super
+      reject_invalid_check_name
+      reject_invalid_flapping_thresholds
+    end
+
+    # Reject check_name option if invalid
+    private
+    def reject_invalid_check_name
+      if @check_name && @check_name !~ CHECK_NAME_PATTERN
+        raise ConfigError,
+          "check_name must be a string consisting of one or more" +
+          " ASCII alphanumerics, underscores, periods, and hyphens"
+      end
+    end
+
+    # Reject invalid check_low_flap_threshold and check_high_flap_threshold
+    private
+    def reject_invalid_flapping_thresholds
+      if @check_low_flap_threshold.nil? ^ @check_high_flap_threshold.nil?
+        raise ConfigError,
+          "'check_low_flap_threshold' and 'check_high_flap_threshold'" +
+          " specified togher, or not specified at all."
+      end
+      if @check_low_flap_threshold
+        in_order = 0 <= @check_low_flap_threshold &&
+          @check_low_flap_threshold <= @check_high_flap_threshold &&
+          @check_high_flap_threshold <= 100
+        if not in_order
+          raise ConfigError,
+            "the following condition must be true:" +
+            " 0 <= check_low_flap_threshold <= check_high_flap_threshold <= 100"
+        end
+      end
+    end
+
+    # Pack the tuple (tag, time, record).
+    public
+    def format(tag, time, record)
+      [tag, time, record].to_msgpack
+    end
+
+    # Send a check
+    public
+    def write(chunk)
+      results = []
+      chunk.msgpack_each { |(tag, time, record)|
+        payload = {
+          'name' => determine_check_name(tag, record),
+          'output' => determine_output(tag, record),
+          'status' => determine_status(tag, record),
+          'type' => @check_type,
+          'handlers' => @check_handlers,
+          'executed' => determine_executed_time(tag, time, record),
+          'fluentd' => {
+            'tag' => tag,
+            'time' => time.to_i,
+            'record' => record,
+          },
+        }
+        add_attribute_if_present(
+          payload, 'ttl', @check_ttl)
+        add_attribute_if_present(
+          payload, 'low_flap_threshold', @check_low_flap_threshold)
+        add_attribute_if_present(
+          payload, 'high_flap_threshold', @check_high_flap_threshold)
+        add_attribute_if_present(
+          payload, 'source', determine_source(tag, record))
+        send_check(@server, @port, payload)
+      }
+    end
+
+    # Send a check to sensu-client.
+    private
+    def send_check(server, port, payload)
+      json = payload.to_json
+      sensu_client = TCPSocket.open(@server, @port)
+      begin
+        sensu_client.puts(json)
+      ensure
+        sensu_client.close
+      end
+    end
+
+    # Adds an attribute to the payload if present.
+    private
+    def add_attribute_if_present(payload, name, value)
+      payload[name] = value if value
+    end
+
+    # Determines "name" attribute of a check.
+    private
+    def determine_check_name(tag, record)
+      # Field specified by check_name_field option
+      if @check_name_field
+        check_name = record[@check_name_field]
+        return check_name if check_name =~ CHECK_NAME_PATTERN
+        log.warn('Invalid check name in the field.' +
+                 ' Fallback to check_name option, tag,' +
+                 ' or constant "fluent-plugin-sensu".',
+                 :tag => tag,
+                 :check_name_field => @check_name_field,
+                 :value => check_name)
+        # Fall through
+      end
+      # check_name option
+      return @check_name if @check_name
+      # Tag
+      return tag if tag =~ CHECK_NAME_PATTERN
+      # Default value
+      log.warn('Invalid check name in the tag.' +
+               'Fallback to the constant "fluent-plugin-sensu".',
+               :tag => tag)
+      return 'fluent-plugin-sensu'
+    end
+
+    # Determines "output" attribute of a check.
+    private
+    def determine_output(tag, record)
+      # Read from the field
+      if @check_output_field
+        check_output = record[@check_output_field]
+        return check_output if check_output
+        log.warn('the field for "output" attribute is absent',
+                :tag => tag, :check_output_field => @check_output_field)
+        # Fall through
+      end
+      # Returns the option value
+      return @check_output if @check_output
+      # Default to JSON notation of the record
+      return record.to_json
+    end
+
+    # Determines "status" attribute of a check.
+    private
+    def determine_status(tag, record)
+      # Read from the field
+      if @check_status_field
+        status_field_val = record[@check_status_field]
+        if status_field_val
+          check_status = SensuOutput.normalize_status(status_field_val)
+          return check_status if check_status
+        end
+        log.warn('the field for "status" attribute is invalid',
+                :tag => tag, :check_status_field => @check_status_field,
+                :value => status_field_val)
+      end
+      # Returns the default
+      return @check_status
+    end
+
+    # Converts variations of status values to an integer.
+    private
+    def self.normalize_status(status_val)
+      status_str = status_val.to_s
+      return 0 if status_str =~ OK_PATTERN
+      return 1 if status_str =~ WARNING_PATTERN
+      return 2 if status_str =~ CRITICAL_PATTERN
+      return 3 if status_str =~ UNKNOWN_PATTERN
+      return nil
+    end
+
+    # Determines "source" attribute of a check.
+    private
+    def determine_source(tag, record)
+      if @check_source_field
+        source = record[@check_source_field]
+        return source if source
+        log.warn('the field for "source" attribute is absent',
+                :tag => tag, :check_source_field => @check_source_field)
+      end
+      return @check_source
+    end
+
+    # Determines "executed" attribute of a check.
+    private
+    def determine_executed_time(tag, time, record)
+      if @check_executed_field
+        executed = record[@check_executed_field]
+        return executed if executed.is_a?(Integer)
+        log.warn('the field for "executed" attribute is absent',
+                :tag => tag, :check_executed_field => @check_executed_field)
+      end
+      return time
+    end
+
+  end
+
+end