sensu-extensions-influxdb 2.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 1274c88a4a5ac95ad0fd334e410db0c8b05c5b09
+  data.tar.gz: 7263d85b3abc0409346a72636f19f0f03789b57b
+SHA512:
+  metadata.gz: 91e9bfd2aedd0a2855c8820d9dd8dd75278216db86a7b0780c9317574798c8aace7cdc6584f7e8f3a50f2496e55a3a5e162c5005d3d504cb762d91a01a6e26a1
+  data.tar.gz: c470b545e5e484e0cc643a750269e4790be696b48751e820e9f1641107db165e4a395ced4c6b84a63222d2883b70f6bec3dd63144bc1ce41795cf140de2580ca

data/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+## [2.0.1] - 2017-05-14
+### Added
+- Released as Gem
+
+## [1.7] - 2017-04-24
+### Added
+- Event tags support (@ganeshk254)
+- SSL support (@PxSonny)

data/LICENSE

@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Johnny Horvi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

data/README.md

@@ -0,0 +1,190 @@
+sensu-influxdb-extension ![circle ci build status](https://circleci.com/gh/jhrv/sensu-influxdb-extension.png?circle-token=:circle-token)
+========================
+
+[Sensu](https://sensuapp.org/) extension for sending metrics to [InfluxDB](https://influxdb.com/) using [line protocol](https://docs.influxdata.com/influxdb/latest/write_protocols/line_protocol_reference)
+
+It handles both metrics on the graphite message format "key value timestamp\n" as well as line protocol directly by setting the extension in **proxy mode**. 
+
+# How it works
+
+For every sensu-event, it will grab the output and transform each line into a line protocol data point. The point will contain tags defined on the check and sensu client (optional).
+It will buffer up points until it reaches the configured length or maximum age (see **buffer_size** and **buffer_max_age**), and then post the data directly to the [InfluxDB write endpoint](https://docs.influxdata.com/influxdb/latest/tools/api/#write).
+
+Example line of graphite data-format ([metric_path] [value] [timestamp]\n):
+
+will be transformed into the following data-point ([line protocol](https://influxdb.com/docs/v0.9/write_protocols/line.html))...
+
+```
+key_a[,<tags>] value=6996 1435216969\n...
+```
+
+# Proxy mode
+
+If the extension is configured to be in proxy mode, it will skip the transformation step and assume that the data is valid [line protocol](https://docs.influxdata.com/influxdb/latest/write_protocols/line_protocol_reference).
+It will not take into account any tags defined in the sensu-configuration.
+
+# Getting started
+
+1) Add the *sensu-influxdb-extension.rb* to the sensu extensions folder (/etc/sensu/extensions)
+
+2) Create your InfluxDB configuration for Sensu (or copy and edit *influxdb-extension.json.tmpl*) inside the sensu config folder (/etc/sensu/conf.d). 
+
+Example of a minimal configuration file
+```
+{
+    "influxdb-extension": {
+        "hostname": "influxdb.mydomain.tld",
+        "database": "metrics",
+    }
+}
+```
+
+## Full list of configuration options
+
+| variable          | default value         |
+| ----------------- | --------------------- |
+| hostname          |       none (required) |
+| port              |                  8086 | 
+| database          |       none (required) |
+| proxy_mode        |                 false |
+| buffer_size       |           100 (lines) |
+| buffer_max_age    |          10 (seconds) |
+| ssl               |                 false |
+| ssl_ca_file (*)   |                  none |
+| ssl_verify        |                  true |
+| precision         |                s (**) |
+| retention_policy  |                  none |
+| username          |                  none |
+| password          |                  none |
+
+(*) Optional file with trusted CA certificates  
+(**) s = seconds. Other valid options are n, u, ms, m, h. See [influxdb docs](https://influxdb.com/docs/v0.9/write_protocols/write_syntax.html) for more details
+
+
+3) Add the extension to your sensu-handler configuration 
+
+```
+"handlers": {
+    "metrics": {
+        "type": "set",
+        "handlers": [ "influxdb-extension" ]
+    }
+    ...
+ }
+
+```
+
+4) Configure your metric/check-definitions to use this handler
+
+```
+"checks": {
+    "metric_cpu": {
+        "type": "metric",
+        "command": "/etc/sensu/plugins/metrics/cpu-usage.rb",
+        "handlers": [ "metrics" ],
+        ...
+ }
+```
+
+5) Restart your sensu-server and sensu-client(s)
+
+You should see the following output in the sensu-server logs if all is working correctly:
+
+```
+{"timestamp":"2015-06-21T13:37:04.256753+0200","level":"info","message":"influxdb-extension:
+Successfully initialized config: hostname: ....
+```
+
+# Tags (optional)
+
+If you want to tag your InfluxDB measurements (great for querying, as tags are indexed), you can define this on the sensu-client as well as on the checks definition. 
+
+Example sensu-client definition:
+
+```
+{
+    "client": {
+        "name": "app_env_hostname",
+        "address": "my-app-in-env.domain.tld",
+        "subscriptions": [],
+        "tags": {
+            "environment": "dev",
+            "application": "myapp",
+            "hostname": "my-app-in-env.domain.tld"
+        }
+    }
+}
+```
+
+Example check definition:
+
+```
+{
+  "checks": {
+    "metric_cpu": {
+      "command": "/opt/sensu/embedded/bin/ruby /path/to/script.rb",
+      "interval": 20,
+      "standalone": true,
+      "type": "metric",
+      "handlers": [
+        "metrics"
+      ],
+      "tags": {
+        "mytag": "xyz"
+      }
+    } 
+  }
+}
+```
+
+... will turn into the following tags for that point: **,environment=dev,application=myapp,hostname=my-app-in-env.domain.tld,mytag=xyz**
+
+If both the client and the check tags have the same key, the one defined on the check will overwrite/win the merge.
+
+The tags will be sorted alphabetically for InfluxDB performance, and tags with empty values will be skipped.
+
+### Event Output Tags
+
+This extension already provides check level and client level tags and now can provide event output tags. This will help us reducing number of sensu checks and provide better flexibility and control.
+
+Example -
+Let's say we configured the sensu check output to be :
+
+```
+app.downloads.eventtags.platform.iOS 26 1476047752
+app.downloads.eventtags.platform.android 52 1476047752
+app.downloads.eventtags.platform.others 12 1476047752
+```
+
+The extension will split the output of the measurement on eventtags. Then it will slice the second part into tag key and values. From above example, the transformed output will be -
+```
+measurement = app.downloads, tags = platform => iOS , value = 26, timestamp = 1476047752 
+measurement = app.downloads, tags = platform => android , value = 52, timestamp = 1476047752
+measurement = app.downloads, tags = platform => others , value = 12, timestamp = 1476047752
+```
+
+You can create multiple tags also, for example :
+
+```
+app.downloads.eventtags.platform.iOS.device.iPad 92 1476047752
+```
+will be transformed to :
+```
+measurement = app.downloads, tags = platform => iOS;device => iPad , value = 92, timestamp = 1476047752 
+```
+
+The event output tags will be merged with client and check definition tags and sent to InfluxDB as usual.
+
+# Performance
+
+The extension will buffer up points until it reaches the configured **buffer_size** length or **buffer_max_age**, and then post all the points in the buffer to InfluxDB. 
+Depending on your load, you will want to tune these configurations to match your environment.
+
+Example:
+If you set the **buffer_size** to 1000, and you have a event-frequency of 100 per second, it will give you about a ten second lag before the data is available through the InfluxDB query API.
+
+**buffer_size** / event-frequency = latency 
+
+However, if you set the **buffer_max_age** to 5 seconds, it will flush the buffer each time it exeeds this limit.
+
+I recommend testing different **buffer_size**s and **buffer_max_age**s depending on your environment and requirements.

data/lib/sensu/extensions/influxdb.rb

@@ -0,0 +1,167 @@
+#!/usr/bin/env ruby
+# coding: utf-8
+
+require "sensu/extension"
+require "net/http"
+require "json"
+
+module Sensu::Extension
+  class InfluxDB < Handler
+
+    @@extension_name = "influxdb-extension"
+
+    def name
+      @@extension_name
+    end
+
+    def description
+      "Transforms and sends metrics to InfluxDB"
+    end
+
+    def post_init
+      influxdb_config = settings[@@extension_name] || Hash.new
+      validate_config(influxdb_config)
+
+      hostname         = influxdb_config[:hostname] || "127.0.0.1"
+      port             = influxdb_config[:port] || "8086"
+      database         = influxdb_config[:database]
+      ssl              = influxdb_config[:ssl] || false
+      ssl_ca_file      = influxdb_config[:ssl_ca_file]
+      ssl_verify       = if influxdb_config.key?(:ssl_verify) then influxdb_config[:ssl_verify] else true end
+      precision        = influxdb_config[:precision] || "s"
+      retention_policy = influxdb_config[:retention_policy]
+      rp_queryparam    = if retention_policy.nil? then "" else "&rp=#{retention_policy}" end
+      protocol         = if ssl then "https" else "http" end
+      username         = influxdb_config[:username]
+      password         = influxdb_config[:password]
+      auth_queryparam  = if username.nil? or password.nil? then "" else "&u=#{username}&p=#{password}" end
+      @BUFFER_SIZE     = if influxdb_config.key?(:buffer_size) then influxdb_config[:buffer_size].to_i else 100 end
+      @BUFFER_MAX_AGE  = if influxdb_config.key?(:buffer_max_age) then influxdb_config[:buffer_max_age].to_i else 10 end
+      @PROXY_MODE      = influxdb_config[:proxy_mode] || false
+
+      string = "#{protocol}://#{hostname}:#{port}/write?db=#{database}&precision=#{precision}#{rp_queryparam}#{auth_queryparam}"
+      @uri = URI(string)
+      @http = Net::HTTP::new(@uri.host, @uri.port)
+      if ssl
+        @http.ssl_version = :TLSv1
+        @http.use_ssl = true
+        @http.verify_mode = if ssl_verify then OpenSSL::SSL::VERIFY_PEER else OpenSSL::SSL::VERIFY_NONE end
+        @http.ca_file = ssl_ca_file
+      end
+      @buffer = []
+      @buffer_flushed = Time.now.to_i
+
+      @logger.info("#{@@extension_name}: successfully initialized config: hostname: #{hostname}, port: #{port}, database: #{database}, uri: #{@uri.to_s}, username: #{username}, buffer_size: #{@BUFFER_SIZE}, buffer_max_age: #{@BUFFER_MAX_AGE}")
+    end
+
+    def run(event)
+      begin
+
+        if buffer_too_old? or buffer_too_big?
+          flush_buffer
+        end
+
+        event = JSON.parse(event)
+        output = event["check"]["output"]
+
+        if not @PROXY_MODE
+          client_tags = event["client"]["tags"] || Hash.new
+          check_tags = event["check"]["tags"] || Hash.new
+          tags = create_tags(client_tags.merge(check_tags))
+        end
+
+        output.split(/\r\n|\n/).each do |point|
+            if not @PROXY_MODE
+              measurement, field_value, timestamp = point.split(/\s+/)
+
+              if not is_number?(timestamp)
+                @logger.debug("invalid timestamp, skipping line in event #{event}")
+                next
+              end
+
+              # Get event output tags
+              if measurement.include?('eventtags')
+                only_measurement, tagstub = measurement.split('.eventtags.',2)
+                event_tags = Hash.new()
+                tagstub.split('.').each_slice(2) do |key, value|
+                  event_tags[key] = value
+                end
+                measurement = only_measurement
+                tags = create_tags(client_tags.merge(check_tags).merge(event_tags))
+              end
+
+              point = "#{measurement}#{tags} value=#{field_value} #{timestamp}"
+            end
+
+            @buffer.push(point)
+            @logger.debug("#{@@extension_name}: stored point in buffer (#{@buffer.length}/#{@BUFFER_SIZE})")
+        end
+        yield 'ok', 0
+      rescue => e
+        @logger.error("#{@@extension_name}: unable to handle event #{event} - #{e}")
+        yield 'error', 2
+      end
+    end
+
+    def create_tags(tags)
+        begin
+            # sorting tags alphabetically in order to increase influxdb performance
+            sorted_tags = Hash[tags.sort]
+
+            tag_string = ""
+
+            sorted_tags.each do |tag, value|
+                next if value.to_s.empty? # skips tags without values
+                tag_string += ",#{tag}=#{value}"
+            end
+
+            @logger.debug("#{@@extension_name}: created tags: #{tag_string}")
+            tag_string
+        rescue => e
+            @logger.debug("#{@@extension_name}: unable to create tag string from #{tags} - #{e.backtrace.to_s}")
+            ""
+        end
+    end
+
+    def send_to_influxdb(payload)
+      request = Net::HTTP::Post.new(@uri.request_uri)
+      request.body = payload
+
+      @logger.debug("#{@@extension_name}: writing payload #{payload} to endpoint #{@uri.to_s}")
+      response = @http.request(request)
+      @logger.debug("#{@@extension_name}: influxdb http response code = #{response.code}, body = #{response.body}")
+    end
+
+    def flush_buffer
+      payload = @buffer.join("\n")
+      send_to_influxdb(payload)
+      @buffer = []
+      @buffer_flushed = Time.now.to_i
+    end
+
+    def buffer_too_old?
+      buffer_age = Time.now.to_i - @buffer_flushed
+      buffer_age >= @BUFFER_MAX_AGE
+    end
+
+    def buffer_too_big?
+      @buffer.length >= @BUFFER_SIZE
+    end
+
+    def validate_config(config)
+      if config.nil?
+        raise ArgumentError, "no configuration for #{@@extension_name} provided. exiting..."
+      end
+
+      ["hostname", "database"].each do |required_setting|
+        if config[required_setting].nil?
+          raise ArgumentError, "required setting #{required_setting} not provided to extension. this should be provided as json element with key #{@@extension_name}. exiting..."
+        end
+      end
+    end
+
+    def is_number?(input)
+      true if Integer(input) rescue false
+    end
+  end
+end

metadata

@@ -0,0 +1,133 @@
+--- !ruby/object:Gem::Specification
+name: sensu-extensions-influxdb
+version: !ruby/object:Gem::Version
+  version: 2.0.1
+platform: ruby
+authors:
+- Johnny Horvi
+- Terje Sannum
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2017-05-14 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sensu-extension
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.6'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sensu-logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: sensu-settings
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: InfluxDB extension for Sensu
+email:
+- johnny@horvi.no
+- terje@offpiste.org
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/sensu/extensions/influxdb.rb
+- LICENSE
+- README.md
+- CHANGELOG.md
+homepage: https://github.com/jhrv/sensu-influxdb-extension
+licenses: []
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14.1
+signing_key: 
+specification_version: 4
+summary: InfluxDB extension for Sensu
+test_files: []