fluent-plugin-anomalydetect 0.1.2 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5dc986d4d340cf2115aaf591724f1f7215dbbbd8
-  data.tar.gz: 9cb4fcb75b4500f544a1d81e1159e60ffdc956d3
+  metadata.gz: a090405a428a2cbcb4cc192885582f624ecb7daa
+  data.tar.gz: 657abfb5723ee0fe3e339d272b4925c30bf3d5ab
 SHA512:
-  metadata.gz: d414c7d4fc72620c9b35ec135145b67d7042315e3e0280339c0a5e08526c0da9ec104432b60321ff761c00fae1db63e56fb75dd4f758979a604910b41ba74067
-  data.tar.gz: f48841a679ffe543fbba58bdf27443653f50ed0e144f94055562edff663f9e30aa5782f1cc7ad62ace7b9cb922f13c5c509ce622ac669e44cb55d5ab9e919b61
+  metadata.gz: 6b0c52886fda3615b96aa689503d7e1b877708345b24dd66c4673b8a47259e6df70da29dff177711af15137999f3b06f2b5bed748c00df8d56b27b5dcb701506
+  data.tar.gz: 662ed34868f9eec666af86a9a495bd83f5c4f177a5fafc551ea4d0eeb918b30e503966c6daa834272fd35739657702f8731daa11d596b83235eb42817517770f

data/README.md

@@ -0,0 +1,186 @@
+# Fluent::Plugin::Anomalydetect
+
+To detect anomaly for log stream, use this plugin.
+Then you can find changes in logs casually.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'fluent-plugin-anomalydetect'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install fluent-plugin-anomalydetect
+
+## Usage
+
+    <source>
+      type file
+      ...
+      tag access.log
+    </source>
+
+    <match access.**>
+      type anomalydetect
+      tag anomaly.access
+      tick 86400
+    </match>
+
+    <match anomaly.access>
+      type file
+      ...
+    </match>
+
+Then the plugin output anomaly log counts in each day.
+
+This plugin watches a value of input record number in the interval set with `tick`.
+
+If you want to watch a value for a target field <fieldname> in data, write below:
+
+    <match access.**>
+      type anomalydetect
+      tag anomaly.access
+      tick 86400
+      target fieldname
+    </match>
+
+## more configuration
+
+    <match access.**>
+      type anomalydetect
+      tag anomaly.access
+      tick 86400
+      target fieldname
+      outlier_term 7
+      outlier_discount 0.5
+      smooth_term 7
+      score_term 28
+      score_discount 0.01
+    </match>
+
+If you want to know detail of these parameters, see "Theory".
+
+    <match access.**>
+      type anomalydetect
+      ...
+      store_file /path/to/anomalydetect.dat
+    </match>
+
+If "store_file" option was specified, a historical stat will be stored to the file at shutdown, and it will be restored on started.
+
+
+    <match access.**>
+      type anomalydetect
+      ...
+      threshold 3
+    </match>
+
+If "threshold" option was specified, plugin only ouput when the anomalyscore is more than threshold.
+
+    <match access.**>
+      type anomalydetect
+      ...
+      trend up
+    </match>
+
+If "trend" option was specified, plugin only ouput when the input data tends to up (or down).
+
+## Parameters
+
+- outlier\_term
+
+- outlier\_discount
+
+- smooth\_term
+
+- score\_term
+
+- score\_discount
+
+- tick
+
+    The time interval to watch in seconds.
+
+- tag
+
+    The output tag name. Required for aggregate `all`. Default is `anomaly`.
+
+- add_tag_prefix
+
+    Add tag prefix for output message. Required for aggregate `tag`.
+
+- remove_tag_prefix
+
+    Remove tag prefix for output message.
+
+- aggragate
+
+    Process data for each `tag` or `all`. The default is `all`.
+
+- target
+
+    Watch a value of a target field in data. If not specified, the number of records is watched (default). The output would become like:
+
+        {"outlier":1.783,"score":4.092,"target":10}
+
+- threshold
+
+    Emit message only if the score is greater than the threshold. Default is `-1.0`.
+
+- trend
+
+    Emit message only if the input data trend is `up` (or `down`). Default is nil.
+
+- store\_file
+
+    Store the learning results into a file, and reload it on restarting.
+
+- targets
+
+    Watch target fields in data. Specify by comma separated value like `x,y`. The output messsages would be like:
+
+        {"x_outlier":1.783,"x_score":4.092,"x":10,"y_outlier":2.310,"y_score":3.982,"y":3}
+
+- thresholds
+
+    Threahold values for each target. Specify by comma separated value like `1.0,2.0`. Use with `targets` option.
+
+- outlier\_suffix
+
+    Change the suffix of emitted messages of `targets` option. Default is `_outlier`.
+
+- score\_suffix
+
+    Change the suffix of emitted messages of `targets` option. Default is `_score`.
+
+- target\_suffix
+
+    Change the suffix of emitted messages of `targets` option. Default is `` (empty).
+
+- suppress\_tick
+
+    Suppress to emit output messsages during specified seconds after starting up.
+
+
+## Theory
+"データマイニングによる異常検知" http://amzn.to/XHXNun
+
+# ToDo
+
+## FFT algorithms
+
+# Copyright
+
+* Copyright
+
+  * Copyright (c) 2013- Muddy Dixon
+  * Copyright (c) 2013- Naotoshi Seo
+
+* License
+
+  * Apache License, Version 2.0

data/fluent-plugin-anormalydetect.gemspec

@@ -3,7 +3,7 @@ lib = File.expand_path('../lib', __FILE__)
 
 Gem::Specification.new do |gem|
   gem.name          = "fluent-plugin-anomalydetect"
-  gem.version       = "0.1.2"
+  gem.version       = "0.1.3"
   gem.authors       = ["Muddy Dixon"]
   gem.email         = ["muddydixon@gmail.com"]
   gem.description   = %q{detect anomal sequential input casually}
@@ -15,7 +15,9 @@ Gem::Specification.new do |gem|
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
-  
+
   gem.add_development_dependency "rake"
+  gem.add_development_dependency "pry"
+  gem.add_development_dependency "pry-nav"
   gem.add_runtime_dependency "fluentd"
 end

data/lib/fluent/plugin/out_anomalydetect.rb

@@ -1,7 +1,7 @@
 module Fluent
   class AnomalyDetectOutput < Output
     Fluent::Plugin.register_output('anomalydetect', self)
-    
+
     require_relative 'change_finder'
     require 'pathname'
 
@@ -11,10 +11,19 @@ module Fluent
     config_param :score_term, :integer, :default => 14
     config_param :score_discount, :float, :default => 0.1
     config_param :tick, :integer, :default => 60 * 5
+    config_param :suppress_tick, :integer, :default => 0
     config_param :tag, :string, :default => "anomaly"
+    config_param :add_tag_prefix, :string, :default => nil
+    config_param :remove_tag_prefix, :string, :default => nil
+    config_param :aggregate, :string, :default => 'all'
     config_param :target, :string, :default => nil
+    config_param :targets, :string, :default => nil
+    config_param :outlier_suffix, :string, :default => '_outlier'
+    config_param :score_suffix, :string, :default => '_score'
+    config_param :target_suffix, :string, :default => ''
     config_param :store_file, :string, :default => nil
-    config_param :threshold, :float, :default => -1.0
+    config_param :threshold, :float, :default => nil
+    config_param :thresholds, :string, :default => nil
     config_param :trend, :default => nil do |val|
       case val.downcase
       when 'up'
@@ -22,24 +31,16 @@ module Fluent
       when 'down'
         :down
       else
-        raise ConfigError, "out_anomaly treand should be 'up' or 'down'"
+        raise ConfigError, "out_anomaly trend should be 'up' or 'down'"
       end
     end
 
-    attr_accessor :outlier
-    attr_accessor :score
-    attr_accessor :record_count
-
-    attr_accessor :outlier_buf
-
-    attr_accessor :records
-
     def configure (conf)
       super
-      unless 0 < @outlier_discount and @outlier_discount < 1 
-        raise Fluent::ConfigError, "discount ratio should be between (0, 1)" 
+      unless 0 < @outlier_discount and @outlier_discount < 1
+        raise Fluent::ConfigError, "discount ratio should be between (0, 1)"
       end
-      unless 0 < @score_discount and @score_discount < 1 
+      unless 0 < @score_discount and @score_discount < 1
         raise Fluent::ConfigError, "discount ratio should be between (0, 1)"
       end
       if @outlier_term < 1
@@ -54,25 +55,113 @@ module Fluent
       if @tick < 1
         raise Fluent::ConfigError, "tick timer should be greater than 1 sec"
       end
+      if @suppress_tick < 0
+        raise Fluent::ConfigError, "`suppress_tick` must be greater or equal to 0 sec"
+      end
       if @store_file
         f = Pathname.new(@store_file)
         if (f.exist? && !f.writable_real?) || (!f.exist? && !f.parent.writable_real?)
           raise Fluent::ConfigError, "#{@store_file} is not writable"
         end
       end
-      @outlier_buf = []
-      @outlier  = ChangeFinder.new(@outlier_term, @outlier_discount)
-      @score    = ChangeFinder.new(@score_term, @score_discount)
+
+      case @aggregate
+      when 'all'
+        raise Fluent::ConfigError, "anomalydetect: `tag` must be specified with aggregate all" if @tag.nil?
+      when 'tag'
+        raise Fluent::ConfigError, "anomalydetect: `add_tag_prefix` must be specified with aggregate tag" if @add_tag_prefix.nil?
+      else
+        raise Fluent::ConfigError, "anomalydetect: aggregate allows tag/all"
+      end
+
+      @tag_prefix = "#{@add_tag_prefix}." if @add_tag_prefix
+      @tag_prefix_match = "#{@remove_tag_prefix}." if @remove_tag_prefix
+      @tag_proc =
+        if @tag_prefix and @tag_prefix_match
+          Proc.new {|tag| "#{@tag_prefix}#{lstrip(tag, @tag_prefix_match)}" }
+        elsif @tag_prefix_match
+          Proc.new {|tag| lstrip(tag, @tag_prefix_match) }
+        elsif @tag_prefix
+          Proc.new {|tag| "#{@tag_prefix}#{tag}" }
+        elsif @tag
+          Proc.new {|tag| @tag }
+        else
+          Proc.new {|tag| tag }
+        end
+
+      if @target and @targets
+        raise Fluent::ConfigError, "anomalydetect: Either of `target` or `targets` can be specified"
+      end
+      if @targets
+        @targets = @targets.split(',')
+      end
+      @output_each_proc =
+        if @targets
+          Proc.new {|outlier, score, val, target| {"#{target}#{@outlier_suffix}" => outlier, "#{target}#{@score_suffix}" => score, "#{target}#{@target_suffix}" => val } }
+        else
+          Proc.new {|outlier, score, val, target| {"outlier" => outlier, "score" => score, "target" => val} }
+        end
+
+      if @threshold and @thresholds
+        raise Fluent::ConfigError, "anomalydetect: Either of `threshold` or `thresholds` can be specified"
+      end
+      if thresholds = @thresholds
+        if @targets.nil?
+          raise Fluent::ConfigError, "anomalydetect: `thresholds` must be specified together with `targets`"
+        end
+        @thresholds = {}
+        thresholds.split(',').map.with_index {|threshold, idx| @thresholds[@targets[idx]]= threshold.to_f }
+        if @thresholds.size != @targets.size
+          raise Fluent::ConfigError, "anomalydetect: The size of `thresholds` must be same with the size of `targets`"
+        end
+      else
+        @threshold = -1.0 if @threshold.nil? # for lower compatibility
+      end
+      @threshold_proc =
+        if @thresholds
+          Proc.new {|target| @thresholds[target] }
+        else
+          Proc.new {|target| @threshold }
+        end
+
+      @records = {}
+      @outliers = {}
+      @outlier_bufs = {}
+      @scores = {}
 
       @mutex = Mutex.new
+    end
+
+    # for test
+    attr_reader :thresholds
+    attr_reader :threshold_proc
 
-      @record_count = @target.nil?
+    def outlier_bufs(tag, target = nil)
+      @outlier_bufs[tag] ||= {}
+      @outlier_bufs[tag][target] ||= []
+    end
+
+    def outliers(tag, target = nil)
+      @outliers[tag] ||= {}
+      @outliers[tag][target] ||= ChangeFinder.new(@outlier_term, @outlier_discount)
+    end
+
+    def scores(tag, target = nil)
+      @scores[tag] ||= {}
+      @scores[tag][target] ||= ChangeFinder.new(@score_term, @score_discount)
+    end
+
+    def init_records(tags)
+      records = {}
+      tags.each do |tag|
+        records[tag] = []
+      end
+      records
     end
 
     def start
       super
       load_from_file
-      init_records
       start_watch
     rescue => e
       $log.warn "anomalydetect: #{e.class} #{e.message} #{e.backtrace.first}"
@@ -89,62 +178,20 @@ module Fluent
       $log.warn "anomalydetect: #{e.class} #{e.message} #{e.backtrace.first}"
     end
 
-    def load_from_file
-      return unless @store_file
-      f = Pathname.new(@store_file)
-      return unless f.exist?
-
-      begin
-        f.open('rb') do |f|
-          stored = Marshal.load(f)
-          if (( stored[:outlier_term]     == @outlier_term ) &&
-              ( stored[:outlier_discount] == @outlier_discount ) &&
-              ( stored[:score_term]       == @score_term ) &&
-              ( stored[:score_discount]   == @score_discount ) &&
-              ( stored[:smooth_term]      == @smooth_term ))
-          then
-            @outlier  = stored[:outlier]
-            @outlier_buf = stored[:outlier_buf]
-            @score    = stored[:score]
-          else
-            $log.warn "anomalydetect: configuration param was changed. ignore stored data"
-          end
-        end
-      rescue => e
-        $log.warn "anomalydetect: Can't load store_file #{e}"
-      end
-    end
-
-    def store_to_file
-      return unless @store_file
-      begin
-        Pathname.new(@store_file).open('wb') do |f|
-          Marshal.dump({
-            :outlier          => @outlier,
-            :outlier_buf         => @outlier_buf,
-            :score            => @score,
-            :outlier_term     => @outlier_term,
-            :outlier_discount => @outlier_discount,
-            :score_term       => @score_term,
-            :score_discount   => @score_discount,
-            :smooth_term      => @smooth_term,
-          }, f)
-        end
-      rescue => e
-        $log.warn "anomalydetect: Can't write store_file #{e}"
-      end
-    end
-
     def start_watch
       @watcher = Thread.new(&method(:watch))
     end
 
     def watch
-      @last_checked = Fluent::Engine.now
+      @started = @last_checked = Fluent::Engine.now
+      @suppress = true
       loop do
         begin
           sleep 0.5
           now = Fluent::Engine.now
+          if @suppress and (now - @started >= @suppress_tick)
+            @suppress = false
+          end
           if now - @last_checked >= @tick
             flush_emit(now - @last_checked)
             @last_checked = now
@@ -155,67 +202,151 @@ module Fluent
       end
     end
 
-    def init_records
-      @records = []
-    end
-
     def flush_emit(step)
-      output = flush
-      if output
-        Fluent::Engine.emit(@tag, Fluent::Engine.now, output)
+      outputs = flush
+      outputs.each do |tag, output|
+        emit_tag = @tag_proc.call(tag)
+        Fluent::Engine.emit(emit_tag, Fluent::Engine.now, output) if output and !output.empty?
       end
     end
 
     def flush
-      flushed, @records = @records, init_records
-
-      val = if @record_count
-              flushed.size
-            else
-              filtered = flushed.map {|record| record[@target] }.compact
-              return nil if filtered.empty?
-              filtered.inject(:+).to_f / filtered.size
+      flushed_records, @records = @records, init_records(tags = @records.keys)
+      outputs = {}
+      flushed_records.each do |tag, records|
+        output =
+          if @targets
+            @targets.each_with_object({}) do |target, output|
+              output_each = flush_each(records, tag, target)
+              output.merge!(output_each) if output_each
             end
+          elsif @target
+            flush_each(records, tag, @target)
+          else
+            flush_each(records, tag)
+          end
+        outputs[tag] = output if output
+      end
+      outputs
+    end
 
-      outlier = @outlier.next(val)
-
-      @outlier_buf.push outlier
-      @outlier_buf.shift if @outlier_buf.size > @smooth_term
-      outlier_avg = @outlier_buf.empty? ? 0.0 : @outlier_buf.inject(:+).to_f / @outlier_buf.size
-
-      score = @score.next(outlier_avg)
+    def flush_each(records, tag, target = nil)
+      val = get_value(records, target)
+      outlier, score, mu = get_score(val, tag, target) if val
+      threshold = @threshold_proc.call(target)
 
-      $log.debug "out_anomalydetect:#{Thread.current.object_id} flushed:#{flushed} val:#{val} outlier:#{outlier} outlier_buf:#{@outlier_buf} score:#{score}"
-      if @threshold < 0 or (@threshold >= 0 and score > @threshold)
+      return nil if @suppress
+      if score and threshold < 0 or (threshold >= 0 and score > threshold)
         case @trend
         when :up
-          return nil if val < @outlier.mu
+          return nil if val < mu
         when :down
-          return nil if val > @outlier.mu
+          return nil if val > mu
         end
-        {"outlier" => outlier, "score" => score, "target" => val}
+        @output_each_proc.call(outlier, score, val, target)
       else
         nil
       end
     end
 
-    def tick_time(time)
-      (time - time % @tick).to_s
+    def get_value(records, target = nil)
+      if target
+        compacted_records = records.map {|record| record[target] }.compact
+        return nil if compacted_records.empty?
+        compacted_records.inject(:+).to_f / compacted_records.size # average
+      else
+        records.size.to_f # num of records
+      end
     end
 
-    def push_records(records)
+    def get_score(val, tag, target = nil)
+      outlier = outliers(tag, target).next(val)
+      mu = outliers(tag, target).mu
+
+      outlier_buf = outlier_bufs(tag, target)
+      outlier_buf.push outlier
+      outlier_buf.shift if outlier_buf.size > @smooth_term
+      outlier_avg = outlier_buf.empty? ? 0.0 : outlier_buf.inject(:+).to_f / outlier_buf.size
+
+      score = scores(tag, target).next(outlier_avg)
+
+      $log.debug "out_anomalydetect:#{Thread.current.object_id} tag:#{tag} val:#{val} outlier:#{outlier} outlier_buf:#{outlier_buf} score:#{score} mu:#{mu}"
+
+      [outlier, score, mu]
+    end
+
+    def push_records(tag, records)
       @mutex.synchronize do
-        @records.concat(records)
+        @records[tag] ||= []
+        @records[tag].concat(records)
       end
     end
 
     def emit(tag, es, chain)
       records = es.map { |time, record| record }
-      push_records records
+      if @aggregate == 'all'
+        push_records(:all, records)
+      else
+        push_records(tag, records)
+      end
 
       chain.next
     rescue => e
       $log.warn "anomalydetect: #{e.class} #{e.message} #{e.backtrace.first}"
     end
+
+    def load_from_file
+      return unless @store_file
+      f = Pathname.new(@store_file)
+      return unless f.exist?
+
+      begin
+        f.open('rb') do |f|
+          stored = Marshal.load(f)
+          if (( stored[:outlier_term]     == @outlier_term ) &&
+              ( stored[:outlier_discount] == @outlier_discount ) &&
+              ( stored[:score_term]       == @score_term ) &&
+              ( stored[:score_discount]   == @score_discount ) &&
+              ( stored[:smooth_term]      == @smooth_term ) &&
+              ( stored[:aggregate]        == @aggregate ))
+          then
+            @outliers     = stored[:outliers]
+            @outlier_bufs = stored[:outlier_bufs]
+            @scores       = stored[:scores]
+          else
+            $log.warn "anomalydetect: configuration param was changed. ignore stored data"
+          end
+        end
+      rescue => e
+        $log.warn "anomalydetect: Can't load store_file #{e}"
+      end
+    end
+
+    def store_to_file
+      return unless @store_file
+      begin
+        Pathname.new(@store_file).open('wb') do |f|
+          Marshal.dump({
+            :outliers         => @outliers,
+            :outlier_bufs     => @outlier_bufs,
+            :scores           => @scores,
+            :outlier_term     => @outlier_term,
+            :outlier_discount => @outlier_discount,
+            :score_term       => @score_term,
+            :score_discount   => @score_discount,
+            :smooth_term      => @smooth_term,
+            :aggregate        => @aggregate,
+          }, f)
+        end
+      rescue => e
+        $log.warn "anomalydetect: Can't write store_file #{e}"
+      end
+    end
+
+    private
+
+    def lstrip(string, substring)
+      string.index(substring) == 0 ? string[substring.size..-1] : string
+    end
   end
 end

data/test/helper.rb

@@ -8,6 +8,7 @@ rescue Bundler::BundlerError => e
   exit e.status_code
 end
 require 'test/unit'
+require 'pry'
 
 $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
 $LOAD_PATH.unshift(File.dirname(__FILE__))

data/test/plugin/test_out_anomalydetect.rb

@@ -15,7 +15,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     smooth_term 3
     target y
   ]
-  
+
   def create_driver (conf=CONFIG, tag="debug.anomaly")
     Fluent::Test::OutputTestDriver.new(Fluent::AnomalyDetectOutput, tag).configure(conf)
   end
@@ -30,7 +30,6 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     assert_equal 300, d.instance.tick
     assert_nil d.instance.target
     assert_equal 'anomaly', d.instance.tag
-    assert d.instance.record_count
 
     d = create_driver
     assert_equal 28, d.instance.outlier_term
@@ -41,7 +40,6 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     assert_equal 10, d.instance.tick
     assert_equal "y", d.instance.target
     assert_equal 'test.anomaly', d.instance.tag
-    assert !d.instance.record_count
 
     assert_raise(Fluent::ConfigError) {
       d = create_driver %[
@@ -83,18 +81,29 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
         tick 0
       ]
     }
-  end
-
-  def test_array_init
-    d = create_driver
-    assert_equal [], d.instance.outlier_buf
-    assert_nil d.instance.records  # @records is initialized at start, not configure
-  end
-
-  def test_sdar
-    d = create_driver
-    assert_instance_of Fluent::ChangeFinder, d.instance.outlier
-    assert_instance_of Fluent::ChangeFinder, d.instance.score
+    assert_raise(Fluent::ConfigError) {
+      d = create_driver %[
+        target y
+        targets x,y,z
+      ]
+    }
+    assert_raise(Fluent::ConfigError) {
+      d = create_driver %[
+        threshold 1.0
+        thresholds 1.0,2.0
+      ]
+    }
+    assert_raise(Fluent::ConfigError) {
+      d = create_driver %[
+        thresholds 1,2
+      ]
+    }
+    assert_raise(Fluent::ConfigError) {
+      d = create_driver %[
+        targets x,y,z
+        thresholds 1
+      ]
+    }
   end
 
   def test_emit_record_count
@@ -109,12 +118,12 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     ]
 
     data = 10.times.map { (rand * 100).to_i } + [0]
-    d.run do 
+    d.run do
       data.each do |val|
         (0..val - 1).each do ||
           d.emit({'y' => 1})
         end
-        r = d.instance.flush
+        r = d.instance.flush[:all]
         assert_equal val, r['target']
       end
     end
@@ -136,7 +145,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     d.run do
       data.each do |val|
         d.emit({'y' => val})
-        r = d.instance.flush
+        r = d.instance.flush[:all]
         assert_equal val, r['target']
       end
     end
@@ -157,7 +166,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     d.run do
       10.times do
         d.emit({'foobar' => 999.99})
-        r = d.instance.flush
+        r = d.instance.flush[:all]
         assert_equal nil, r
       end
     end
@@ -168,11 +177,11 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     reader = CSV.open("test/stock.2432.csv", "r")
     header = reader.take(1)[0]
     d = create_driver
-    d.run do 
+    d.run do
       reader.each_with_index do |row, idx|
         break if idx > 5
         d.emit({'y' => row[4].to_i})
-        r = d.instance.flush
+        r = d.instance.flush[:all]
         assert r['target']
         assert r['outlier']
         assert r['score']
@@ -191,15 +200,15 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     ]
 
     d.run do
-      assert_equal [], d.instance.outlier_buf
+      assert_equal([], d.instance.outlier_bufs(:all))
       d.emit({'x' => 1})
       d.emit({'x' => 1})
       d.emit({'x' => 1})
-      d.instance.flush
+      d.instance.flush[:all]
       d.emit({'x' => 1})
       d.emit({'x' => 1})
       d.emit({'x' => 1})
-      d.instance.flush
+      d.instance.flush[:all]
     end
     assert File.exist? file
 
@@ -207,7 +216,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
       store_file #{file}
     ]
     d2.run do
-      assert_equal 2, d2.instance.outlier_buf.size
+      assert_equal 2, d2.instance.outlier_bufs(:all).size
     end
 
     File.unlink file
@@ -220,11 +229,11 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     d = create_driver %[
       threshold 1000
     ]
-    d.run do 
+    d.run do
       reader.each_with_index do |row, idx|
         break if idx > 5
         d.emit({'y' => row[4].to_i})
-        r = d.instance.flush
+        r = d.instance.flush[:all]
         assert_equal nil, r
       end
     end
@@ -237,11 +246,11 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
     d = create_driver %[
       threshold 1
     ]
-    d.run do 
+    d.run do
       reader.each_with_index do |row, idx|
         break if idx > 5
         d.emit({'y' => row[4].to_i})
-        r = d.instance.flush
+        r = d.instance.flush[:all]
         assert_not_equal nil, r
       end
     end
@@ -258,7 +267,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
       d.emit({'y' => 0.0}); d.instance.flush
       d.emit({'y' => 0.0}); d.instance.flush
       d.emit({'y' => 0.0}); d.instance.flush
-      d.emit({'y' => -1.0}); r = d.instance.flush
+      d.emit({'y' => -1.0}); r = d.instance.flush[:all]
       assert_equal nil, r
     end
 
@@ -267,7 +276,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
       d.emit({'y' => -1.0}); d.instance.flush
       d.emit({'y' => -1.0}); d.instance.flush
       d.emit({'y' => -1.0}); d.instance.flush
-      d.emit({'y' => 0.0}); r = d.instance.flush
+      d.emit({'y' => 0.0}); r = d.instance.flush[:all]
       assert_not_equal nil, r
     end
   end
@@ -282,7 +291,7 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
       d.emit({'y' => 0.0}); d.instance.flush
       d.emit({'y' => 0.0}); d.instance.flush
       d.emit({'y' => 0.0}); d.instance.flush
-      d.emit({'y' => -1.0}); r = d.instance.flush
+      d.emit({'y' => -1.0}); r = d.instance.flush[:all]
       assert_not_equal nil, r
     end
 
@@ -292,8 +301,121 @@ class AnomalyDetectOutputTest < Test::Unit::TestCase
       d.emit({'y' => -1.0}); d.instance.flush
       d.emit({'y' => -1.0}); d.instance.flush
       d.emit({'y' => 0.0})
-      r = d.instance.flush
+      r = d.instance.flush[:all]
       assert_equal nil, r
     end
   end
+
+  def test_aggregate_tag
+    d = create_driver %[
+      outlier_term 28
+      outlier_discount 0.05
+      score_term 28
+      score_discount 0.05
+      tick 10
+      smooth_term 3
+      aggregate tag
+      add_tag_prefix test
+    ]
+
+    data = 10.times.map { (rand * 100).to_i } + [0]
+    d.run do
+      data.each do |val|
+        (0..val - 1).each do ||
+          d.emit({'y' => 1})
+        end
+        r = d.instance.flush['debug.anomaly']
+        assert_equal val, r['target']
+      end
+    end
+  end
+
+  def test_targets
+    d = create_driver %[
+      targets x,y
+    ]
+    data = 10.times.map { (rand * 100).to_i } + [0]
+    d.run do
+      data.each do |val|
+        d.emit({'x' => val, 'y' => val})
+        r = d.instance.flush[:all]
+        assert_equal val, r['x']
+        assert_equal val, r['y']
+      end
+    end
+  end
+
+  def test_targets_default_suffix
+    d = create_driver %[
+      targets x,y
+    ]
+    data = 1.times.map { (rand * 100).to_i } + [0]
+    d.run do
+      data.each do |val|
+        d.emit({'x' => val, 'y' => val})
+        r = d.instance.flush[:all]
+        assert r.has_key?('x')
+        assert r.has_key?('y')
+        assert r.has_key?('x_outlier')
+        assert r.has_key?('x_score')
+        assert r.has_key?('y_outlier')
+        assert r.has_key?('y_score')
+      end
+    end
+  end
+
+  def test_targets_suffix
+    d = create_driver %[
+      targets x,y
+      outlier_suffix
+      score_suffix _anomaly
+      target_suffix _target
+    ]
+    data = 1.times.map { (rand * 100).to_i } + [0]
+    d.run do
+      data.each do |val|
+        d.emit({'x' => val, 'y' => val})
+        r = d.instance.flush[:all]
+        assert r.has_key?('x_target')
+        assert r.has_key?('y_target')
+        assert r.has_key?('x')
+        assert r.has_key?('x_anomaly')
+        assert r.has_key?('y')
+        assert r.has_key?('y_anomaly')
+      end
+    end
+  end
+
+  def test_targets_thresholds
+    d = create_driver %[
+      targets x,y
+      thresholds 1,2
+    ]
+    d.run do
+      thresholds = d.instance.thresholds
+      assert_equal 1, thresholds['x']
+      assert_equal 2, thresholds['y']
+
+      threshold_proc = d.instance.threshold_proc
+      assert_equal 1, threshold_proc.call('x')
+      assert_equal 2, threshold_proc.call('y')
+    end
+  end
+
+  def test_suppress_tick
+    d = create_driver %[
+      tick 10
+      suppress_tick 30
+      target y
+    ]
+
+    data = 10.times.map { (rand * 100).to_i } + [0]
+    d.run do
+      data.each do |val|
+        d.emit({'y' => val})
+        r = d.instance.flush[:all]
+        assert_equal nil, r
+      end
+    end
+  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-anomalydetect
 version: !ruby/object:Gem::Version
-  version: 0.1.2
+  version: 0.1.3
 platform: ruby
 authors:
 - Muddy Dixon
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-08-20 00:00:00.000000000 Z
+date: 2014-01-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake
@@ -24,6 +24,34 @@ dependencies:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry-nav
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: fluentd
   requirement: !ruby/object:Gem::Requirement
@@ -47,7 +75,7 @@ extra_rdoc_files: []
 files:
 - .gitignore
 - Gemfile
-- README.rdoc
+- README.md
 - Rakefile
 - fluent-plugin-anormalydetect.gemspec
 - lib/fluent/plugin/change_finder.rb
@@ -74,7 +102,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.0.2
+rubygems_version: 2.0.3
 signing_key: 
 specification_version: 4
 summary: detect anomal sequential input casually

data/README.rdoc

@@ -1,107 +0,0 @@
-= Fluent::Plugin::Anomalydetect
-
-To detect anomaly for log stream, use this plugin.
-Then you can find changes in logs casually.
-
-= Installation
-
-Add this line to your application's Gemfile:
-
-    gem 'fluent-plugin-anomalydetect'
-
-And then execute:
-
-    $ bundle
-
-Or install it yourself as:
-
-    $ gem install fluent-plugin-anomalydetect
-
-== Usage
-
-    <source>
-      type file
-      ...
-      tag access.log
-    </source>
-
-    <match access.**>
-      type anomalydetect
-      tag anomaly.access
-      tick 86400
-    </match>
-
-    <match anomaly.access>
-      type file
-      ...
-    </match>
-
-Then the plugin output anomaly log counts in each day.
-
-This plugin watches a value of input record number in the interval set with `tick`.
-
-If you want to watch a value for a target field <fieldname> in data, write below:
-
-    <match access.**>
-      type anomalydetect
-      tag anomaly.access
-      tick 86400
-      target fieldname
-    </match>
-
-== more configuration
-
-    <match access.**>
-      type anomalydetect
-      tag anomaly.access
-      tick 86400
-      target fieldname
-      outlier_term 7
-      outlier_discount 0.5
-      smooth_term 7
-      score_term 28
-      score_discount 0.01
-    </match>
-
-If you want to know detail of these parameters, see "Theory".
-    
-    <match access.**>
-      type anomalydetect
-      ...
-      store_file /path/to/anomalydetect.dat
-    </match>
-
-If "store_file" option was specified, a historical stat will be stored to the file at shutdown, and it will be restored on started.
-
-
-    <match access.**>
-      type anomalydetect
-      ...
-      threshold 3
-    </match>
-
-If "threshold" option was specified, plugin only ouput when the anomalyscore is more than threshold.
-
-    <match access.**>
-      type anomalydetect
-      ...
-      trend up
-    </match>
-
-If "trend" option was specified, plugin only ouput when the input data tends to up (or down). 
-
-== Theory
-"データマイニングによる異常検知" http://amzn.to/XHXNun
-
-= TODO
-
-== threshold
-
-fluentd outputs value when the outlier value over threshold
-
-== FFT algorithms
-
-= Copyright
-
-Copyright:: Copyright (c) 2013- Muddy Dixon
-License::   Apache License, Version 2.0