fluent-plugin-statsite 0.0.4 → 0.0.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 18bf875092d09bd490fd29b12b3749aeefc44248
-  data.tar.gz: 522741407db13e270fbd60fa523020d0428ecb5a
+  metadata.gz: 19a955bd7d1edfb5cc432376589593364d276a3a
+  data.tar.gz: 67c75225376f3a128618ff3dfbc7fd584f5d1ed0
 SHA512:
-  metadata.gz: 0f48dfb35f0409afa99d40aa18bc8599b98bafa13d78fdc2a177aa2404fd8bd173ee92380c08bc1e73015563fec80535768a69ecd9f7fb6ddec6a9aa0b690399
-  data.tar.gz: e61528aee4cd10d6ad2821d1cdbe5ecd48d4ef02c97b0f04fa7f3dbf87addeef6528c149ecba9897236f9d7edd5e642001b0080dc464b8e8a1f59b06547959d4
+  metadata.gz: 4b0f6e5e0c20122d87141bd21e9a3e2bf7cc326399de0b2a648a0c72866bea9a62e34ca1d5870e70e8c0890719424305020faea7312b3c0ba15b613e78fcc8e1
+  data.tar.gz: b18f846e67dbf88bc580ee5ef9a902d6ce04a24754a31c13a094aa4ab6b181c0270bc6f4c6534585a9c9ce0ed8d19dc2ec0372654f6cf7a09d8270dd04914284

data/README.md

@@ -1,5 +1,7 @@
 # Statsite Fluentd Plugin
 
+[![Build Status](https://travis-ci.org/choplin/fluent-plugin-statsite.svg?branch=master)](https://travis-ci.org/choplin/fluent-plugin-statsite)
+
 This plugin calculates various useful metrics using [Statsite by armon](http://armon.github.io/statsite/).
 
  [Statsite](http://armon.github.io/statsite/) is very cool software. Statsite works as daemon service, receiving events from tcp/udp, aggregating these events with specified methods, and sending the results via pluggable sinks. Statsite is written in C, cpu and memory efficient, and employ some approximate algorithms for unique sets and percentiles.
@@ -8,6 +10,67 @@ This plugin calculates various useful metrics using [Statsite by armon](http://a
 
  Statsite process is launched as a child process from this plugin internally. All you have to do place statsite the binary under $PATH, or set the path of statsite binary as parameter. Neither config files or daemon process is not required. Besides, the communication between the plugin and the Statsite process takes place through STDIN/STDOUT, so no network port will be used.
 
+## Quickstart
+
+Assume that nginx log events like below come.
+
+```json
+{
+  "remote_addr":"114.170.6.118",
+  "remote_user":"-",
+  "time_local":"20/Jul/2014:18:25:50 +0000",
+  "request":"GET /foo HTTP/1.1",
+  "status":"200",
+  "body_bytes_sent":"911",
+  "http_referer":"-",
+  "http_user_agent":"Mozilla/5.0 (Linux; U; Android 4.2.2; ja-jp; SO-04E Build/10.3.1.B.0.256) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30",
+  "request_time":"0.058",
+  "upstream_addr":"192.168.222.180:80",
+  "upstream_response_time":"0.058"
+}
+```
+
+and you set a fluentd config as,
+
+```
+<match **>
+  type statsite_filter
+  tag statsite
+  metrics [
+    "status_${status}:1|c",
+    "request_time|${request_time}|ms"}
+  ]
+  histograms [
+    {"prefix": "request_time", "min": 0, "max": 1, "width": 0.1}
+  ]
+  statsite_flush_interval 1s
+  flush_interval 1s
+</match>
+```
+
+then you will get events such as below every specified seconds.
+
+```json
+statsite 1406124737 {"type":"counts","key":"status_500","value":1.0}
+statsite 1406124737 {"type":"counts","key":"status_200","value":3.0}
+statsite 1406124737 {"type":"counts","key":"status_302","value":1.0}
+statsite 1406124737 {"type":"timers","key":"request_time","value":12.0,"statistic":"sum"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":40.0,"statistic":"sum_sq"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":2.4,"statistic":"mean"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":1.0,"statistic":"lower"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":5.0,"statistic":"upper"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":5,"statistic":"count"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":1.67332,"statistic":"stdev"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":2.0,"statistic":"median"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":5.0,"statistic":"p95"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":5.0,"statistic":"p99"}
+statsite 1406124737 {"type":"timers","key":"request_time","value":12.0,"statistic":"rate"}
+```
+
+## Prerequisite
+
+You have to install Statsite on the machine where fluentd is running.
+
 ## Installation
 
 `$ fluent-gem install fluent-plugin-statsite`
@@ -20,44 +83,167 @@ Please refer to [Statsite official page](http://armon.github.io/statsite/).
 
 ## Configuration
 
-It is strongly recommended to use '[V1 config format](http://docs.fluentd.org/articles/config-file#v1-format)' because this plugin requires to set deeply nested parameters. 
+It is strongly recommended to use '[V1 config format](http://docs.fluentd.org/articles/config-file#v1-format)' because this plugin requires to set deeply nested parameters.
+
+### Parameter
+
+key                     | type   | description                                                                                                                                                                                      | required | default
+---                     | ---    | ---                                                                                                                                                                                              | ---      | ---
+tag                     | string | The tag of output events.                                                                                                                                                                        | yes      |
+metrics                 | array  | How to retrive statsite messages from fluentd event. see the details below.                                                                                                                      | yes      |
+histograms              | array  | THe statstie histogram settings. see the details below.                                                                                                                                          | no       | []
+statiste_path           | string | The path of statsite command. Leave this blank if statsite places under $PATH.                                                                                                                   | yes      | statsite
+statsite_flush_interval | time   | The interval at which statsite flush aggregated results.                                                                                                                                         | no       | 10s
+stream_cmd              | string | This is the command that statsite invokes every flush_interval seconds to handle the metrics. It can be any executable. It should read inputs over stdin and exit with status code 0 on success. | no       | cat
+time_eps                | float  | The upper bound on error for timer estimates. Please refer to statsite official page.                                                                                                            | no       | 0.01
+set_eps                 | float  | The upper bound on error for unique set estimates. Please refer to statsite official page.                                                                                                       | no       | 0.02
+child_respawn           | string | How many times statsite will be respawned in case of unexpected exit.                                                                                                                            | no       |
+
+### Metrics
+
+Metrics parameter specifies how to form messages to send to statsite from each fluentd event.
 
-### Example
+Top level of **metrics* parameter must be an array which contains strings or hashes. If you set multliple elements in an array, equivalent number of messages will be sent to statsite for one fluentd event.
+
+For example, given this metrics setting,
 
 ```
-<match **>
-  type statsite_filter
-  tag statsite
-  metrics [
-    "${status}:1|c",
-    {"key": "request_time", "value": "request_time", "type": "ms"}
-  ]
-  histograms [
-    {"prefix": "request_time", "min": 0, "max": 1, "width": 0.1}
-  ]
-  statsite_path "statsite"
-  statsite_flush_interval 1s
-  timer_eps 0.01
-  set_eps 0.01
-  child_respawn 5
-</match>
+metrics [
+    {"key": "key_1", "value": "1", "type": "s"}
+    {"key": "key_2", "value": "1", "type": "s"}
+]
 ```
 
-### Parameter
+and this fluentd event is comming,
+
+```
+{"foo": "f", "bar": "b", "hoge": "h"}
+```
+
+then the events below will be sent to statsite
+
+```
+key_1:1|c
+key_2:1|c
+```
+
+Every element of the array must foloow the one of these format, the string format, and the hash format. Both they have the same fields semantically. See the details of these formats below.
 
-TODO
+#### Fields
 
-### Metrics Format
+key        | type   | description            | required
+---        | ---    | ---                    | ---
+key        | string | message key            | yes
+value_time | string | message value          | yes
+type       | enum   | statsite message type. | yes
 
-You can specify metrics in two format, string style, and hash style.
+With these settings, A message sent to statsite is "${key}:${value}|${type}\n"
 
-#### String style
+You should also see [Statsite official page](http://armon.github.io/statsite/) to see the statsite supports in detail.
 
-TODO
+#### String format
+
+String format almost the same as Statsite's event protocol, though this format supports variable substitution in key and value field, which is described in detail below.
+
+##### Example
+
+```
+metrics [
+    "status_${status}:1|c"
+]
+```
+
+#### Hash format
+
+Hash format is expressed as JSON Object.
+
+##### Example
+
+```json
+metrics [
+    {"key": "status_${status}", "value": "1", "type": "c"}
+]
+```
+
+#### Message Type
+
+Please refer to [Statsite official page](http://armon.github.io/statsite/).
+
+type | description
+---  | ---
+kv   | Simple Key/Value.
+g    | Gauge, similar to kv but only the last value per key is retained
+ms   | Timer.
+h    | Alias for timer
+c    | Counter.
+s    | Unique Set
+
+#### Variable substitution
+
+Both in string and hash format, key and value field support substitution of variable.
+
+When the string "${*subst_key*}" exists key or value field, this will be replaces with the corresponding value in the fluentd event.
+
+For example, given this metrics setting,
+
+```
+metrics [
+    {"key": "key_${foo}_${bar}", "value": "${hoge}", "type": "c"}
+]
+```
+
+and this fluentd event is comming,
+
+```
+{"foo": "f", "bar": "b", "hoge": "1"}
+```
+
+then the events below will be sent to statsite
+
+```
+key_f_b:1|s
+```
+
+When any one of the substitution key does not exists in the fluentd event, no messages will be sent to statsite for that metric element.
+
+For example, given this metrics setting,
+
+```
+metrics [
+    {"key": "key_${foo}", "value": "1", "type": "c"},
+    {"key": "key_${bar}", "value": "1", "type": "c"}
+]
+```
+
+and this fluentd event is comming,
+
+```
+{"foo": "f"}
+
+then the events below will be sent to statsite
+
+```
+key_f:1|c
+```
+
+### Histograms
+
+#### Example
+
+```json
+histogram [
+    {"prefix": "request_time", "min": 0, "max": 1, "width": 0.1}
+]
+```
 
-#### Hash style
+#### Fields
 
-TODO
+key    | type   | description                                                                        | required
+---    | ---    | ---                                                                                | ---
+prefix | string | This is the key prefix to match on. This is also used as a suffix of section name. | no
+min    | float  | The minimum bound on the histogram.                                                | yes
+max    | float  | The maximum bound on the histogram.                                                | yes
+width  | float  | The width of each bucket between the min and max.                                  | yes
 
 ## Copyright
 

data/fluent-plugin-statsite.gemspec

@@ -4,13 +4,12 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-plugin-statsite"
-  spec.version       = "0.0.4"
+  spec.version       = "0.0.7"
   spec.authors       = ["OKUNO Akihiro"]
   spec.email         = ["choplin.choplin@gmail.com"]
   spec.summary       = %q{Fluentd statsite plugin}
   spec.description   = %q{Fluentd plugin which caluculate statistics using statsite}
-  spec.homepage      = ""
-  spec.homepage         = "https://github.com/choplin/fluent-plugin-statsite"
+  spec.homepage      = "https://github.com/choplin/fluent-plugin-statsite"
   spec.license       = "Apache-2.0"
 
   spec.files         = `git ls-files -z`.split("\x0")

data/lib/fluent/plugin/out_statsite_filter.rb

@@ -1,5 +1,6 @@
 require_relative 'statsite/child_process'
-require_relative 'statsite/format'
+require_relative 'statsite/formatter'
+require_relative 'statsite/parser'
 require_relative 'statsite/metric'
 require_relative 'statsite/histogram'
 
@@ -20,6 +21,7 @@ module Fluent
     config_param :histograms,              :array,    :default => []
     config_param :statsite_path,           :string,   :default => 'statsite'
     config_param :statsite_flush_interval, :time,     :default => 10
+    config_param :stream_cmd,              :string,   :default => 'cat'
     config_param :timer_eps,               :float,    :default => 0.01
     config_param :set_eps,                 :float,    :default => 0.02
     config_param :child_respawn,           :string,   :default => nil
@@ -143,7 +145,7 @@ log_level = INFO
 flush_interval = #{@statsite_flush_interval}
 timer_eps = #{@timer_eps}
 set_eps = #{@set_eps}
-stream_cmd = cat
+stream_cmd = #{@stream_cmd}
 
 #{@histograms.map(&:to_ini).join("\n\n")}
       CONFIG

data/lib/fluent/plugin/statsite/formatter.rb

@@ -0,0 +1,13 @@
+module Fluent
+  module StatsitePlugin
+    class StatsiteFormatter
+      def initialize(metrics)
+        @metrics = metrics
+      end
+
+      def call(record)
+        @metrics.map{|m| m.convert(record)}.select{|m| not m.nil?}.join('')
+      end
+    end
+  end
+end

data/lib/fluent/plugin/statsite/parser.rb

@@ -0,0 +1,91 @@
+module Fluent
+  module StatsitePlugin
+    module Parser
+      def parse_line(line)
+        k,v,t = line.chomp.split('|')
+        record = build_record(k,v)
+        [t.to_i, record]
+      end
+
+      def build_record(k,v)
+        type, key, statistic, range = k.split(".", 4)
+
+        case type
+        when 'timers' then 1
+          if statistic == 'histogram'
+            {type: type, key: key, value: v.to_i, statistic: statistic, range: range[4..-1]}
+          elsif statistic == 'count'
+            {type: type, key: key, value: v.to_i, statistic: statistic}
+          else
+            {type: type, key: key, value: v.to_f, statistic: statistic}
+          end
+        when 'kv', 'gauges', 'counts'
+          {type: type, key: key, value: v.to_f}
+        when 'sets'
+          {type: type, key: key, value: v.to_i}
+        end
+      end
+    end
+
+    class StatsiteParser
+      include Parser
+
+      def initialize(on_message)
+        @on_message = on_message
+      end
+
+      def call(io)
+        io.each_line(&method(:each_line))
+      end
+
+      def each_line(line)
+        time, record = parse_line(line)
+        raise "out_statsite: failed to parse a line. '#{line}'" if record.nil?
+
+        @on_message.call(time, record)
+      end
+    end
+
+    class StatsiteAggregateParser
+      # TODO: should be configurable?
+      FLUSH_WAIT = 0.5
+
+      def initialize(on_message, coolio_loop)
+        @on_message = on_message
+        @loop = coolio_loop
+        @buf = {}
+      end
+
+      def call(io)
+        io.each_line(&method(:each_line))
+      end
+
+      def each_line(line)
+        record = parse_line(line)
+
+        raise "out_statsite: failed to parse a line. '#{line}'" if record.nil?
+
+        timer = TimerWatcher(FLUSH_WAIT, $log, method(&:flush))
+      end
+
+      def flush
+        @on_message.call(t.to_i, record)
+      end
+
+      class TimerWatcher < Coolio::TimerWatcher
+        def initialize(duration, log, callback)
+          @callback = callback
+          @log = log
+          super(interval, repeat)
+        end
+
+        def on_timer
+          @callback.call
+        rescue
+          @log.error $!.to_s
+          @log.error_backtrace
+        end
+      end
+    end
+  end
+end

data/test/test_formatter.rb

@@ -1,5 +1,5 @@
 require 'helper'
-require 'fluent/plugin/statsite/format'
+require 'fluent/plugin/statsite/formatter'
 
 include Fluent::StatsitePlugin
 

data/test/test_parser.rb

@@ -1,5 +1,5 @@
 require 'helper'
-require 'fluent/plugin/statsite/format'
+require 'fluent/plugin/statsite/parser'
 
 include Fluent::StatsitePlugin
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-statsite
 version: !ruby/object:Gem::Version
-  version: 0.0.4
+  version: 0.0.7
 platform: ruby
 authors:
 - OKUNO Akihiro
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-07-21 00:00:00.000000000 Z
+date: 2014-08-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -70,10 +70,11 @@ files:
 - fluent-plugin-statsite.gemspec
 - lib/fluent/plugin/out_statsite_filter.rb
 - lib/fluent/plugin/statsite/child_process.rb
-- lib/fluent/plugin/statsite/format.rb
+- lib/fluent/plugin/statsite/formatter.rb
 - lib/fluent/plugin/statsite/histogram.rb
 - lib/fluent/plugin/statsite/metric.rb
 - lib/fluent/plugin/statsite/metric_format.rb
+- lib/fluent/plugin/statsite/parser.rb
 - test/helper.rb
 - test/test_formatter.rb
 - test/test_histogram.rb

data/lib/fluent/plugin/statsite/format.rb

@@ -1,47 +0,0 @@
-module Fluent
-  module StatsitePlugin
-    class StatsiteParser
-      def initialize(on_message)
-        @on_message = on_message
-      end
-
-      def call(io)
-        io.each_line(&method(:each_line))
-      end
-
-      def each_line(line)
-        k,v,t = line.chomp.split('|')
-        type, key, statistic, range = k.split(".", 4)
-
-        record = case type
-                 when 'timers' then 1
-                   if statistic == 'histogram'
-                     {type: type, key: key, value: v.to_i, statistic: statistic, range: range[4..-1]}
-                   elsif statistic == 'count'
-                     {type: type, key: key, value: v.to_i, statistic: statistic}
-                   else
-                     {type: type, key: key, value: v.to_f, statistic: statistic}
-                   end
-                 when 'kv', 'gauges', 'counts'
-                   {type: type, key: key, value: v.to_f}
-                 when 'sets'
-                   {type: type, key: key, value: v.to_i}
-                 end
-
-        raise "out_statsite: failed to parse a line. '#{line}'" if record.nil?
-
-        @on_message.call(t.to_i, record)
-      end
-    end
-
-    class StatsiteFormatter
-      def initialize(metrics)
-        @metrics = metrics
-      end
-
-      def call(record)
-        @metrics.map{|m| m.convert(record)}.select{|m| not m.nil?}.join('')
-      end
-    end
-  end
-end