RubyGems - fluent-plugin-woothee - Versions diffs - 0.2.2 → 1.0.0 - Mend

fluent-plugin-woothee 0.2.2 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (10) hide show

checksums.yaml +4 -4
data/.travis.yml +0 -1
data/README.md +60 -109
data/fluent-plugin-woothee.gemspec +2 -2
data/lib/fluent/plugin/filter_woothee.rb +56 -60
data/test/helper.rb +0 -1
data/test/plugin/test_filter_woothee.rb +76 -81
metadata +3 -6
data/lib/fluent/plugin/out_woothee.rb +0 -154
data/test/plugin/test_out_woothee.rb +0 -364

checksums.yaml CHANGED

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 6adc6d94e1e14d85619c6f5f320ff6c9e1b7ec77
-  data.tar.gz: 76a0c6592d1e33e6625bd8cb58839d64d1f7344d
+  metadata.gz: cf79bb6e1d07bdddb4cae76b7ae7674fbe7b460f
+  data.tar.gz: 6d1c7933f26b6f2b7bb7df6aacb11f645ca6f649
 SHA512:
-  metadata.gz: 0446a8bc15e2e7cca56d407c0085574fd821ae24dfec837b0526b509e50f5c6baa18cf5b80c18dc595b4abc5775e76f8fbb6ec6ecf893c92c1fe0a7df25c4603
-  data.tar.gz: 06e8582442bc4409103f92e3b40eadd4b41e0d9cf51731ebdb96e95d09ca4cd5204f6807ed8148d3daeb2a0b77403cd167d73ef8ec565485d8575018905fe49f
+  metadata.gz: 2b93cc550a0584bd74421878da8efd1da5d97c7e8f7fe09f5e98bf5cabd0b4d30c283220a5503fb5298304358a5a6986b592039394f4e06bdfef0b446f0d7034
+  data.tar.gz: e0444b74e92eb0bdf2cb6968b82d9998bed32b29ec025aa8a6c6272a62185e050f2b44610e9cb00a285144dc2185ad15867157c6644d7e02cb4b14c5397dbd41

data/.travis.yml CHANGED

@@ -1,7 +1,6 @@
 language: ruby
 sudo: false
 rvm:
-  - 2.0.0
   - 2.1
   - 2.2
   - 2.3.0

data/README.md CHANGED

@@ -1,133 +1,84 @@
 # fluent-plugin-woothee
-## WootheeOutput
-'fluent-plugin-woothee' is a Fluentd plugin to parse UserAgent strings and to filter/drop specified categories of user terminals (like 'pc', 'smartphone' and so on).
+'fluent-plugin-woothee' is a Fluentd filter plugin to parse UserAgent strings and to filter/drop specified categories of user terminals (like 'pc', 'smartphone' and so on).
 'woothee' is multi-language user-agent strings parser project. See: https://github.com/woothee/woothee
 ## Configuration
-To add woothee parser result into matched messages:
-    <match input.**>
-      type woothee
-      key_name agent
-      remove_prefix input
-      add_prefix merged
-      merge_agent_info yes
-    </match>
-Output messages with tag 'merged.**' has attributes like 'agent\_name', 'agent\_category' and 'agent\_os' from woothee parser result. If you want to change attribute names, or want to merge more attributes of browser vendor and its version, write configurations as below:
-    <match input.**>
-      type woothee
-      key_name agent
-      remove_prefix input
-      add_prefix merged
-      merge_agent_info yes
-      out_key_name ua_name
-      out_key_category ua_category
-      out_key_os ua_os
-      out_key_os_version ua_os_version
-      out_key_version ua_version
-      out_key_vendor ua_vendor
-    </match>
-To re-emit messages with specified user-agent categories (and merge woothee parser result), configure like this:
-    <match input.**>
-      type woothee
-      key_name agent
-      filter_categories pc,smartphone,mobilephone,appliance
-      remove_prefix input
-      add_prefix merged
-      merge_agent_info yes
-    </match>
-Or, you can specify categories to drop (and not to merge woothee result):
-    <match input.**>
-      type woothee
-      key_name agent
-      drop_categories crawler
-      remove_prefix input
-      add_prefix merged
-      merge_agent_info false # default
-    </match>
-### Fast Crawler Filter
-If you want to drop __almost__ all of messages with crawler's user-agent, and not to merge woothee result, you just specify plugin type:
-    <match input.**>
-      type woothee_fast_crawler_filter
-      key_name useragent
-      tag filtered
-    </match>
-'fluent-plugin-woothee' uses 'Woothee.is_crawler' of woothee with this configuration, fast and incomplete method to judge user-agent is crawler or not.
-If you want to drop all of crawlers completely, specify 'type woothee' and 'drop_categories crawler'.
-## WootheeFilter
-This is filter version of 'fluent-plugin-woothee'.
-Note that this filter version does not have rewrite tag functionality.
-## Configuration
-To add woothee parser result into filtered messages:
-    <filter input.**>
-      type woothee
-      key_name agent
-      merge_agent_info yes
-    </filter>
-Filtered messages with non-modified tag has attributes like 'agent\_name', 'agent\_category' and 'agent\_os' from woothee parser result. If you want to change attribute names, or want to merge more attributes of browser vendor and its version, write configurations as below:
-    <filter input.**>
-      type woothee
-      key_name agent
-      merge_agent_info yes
-      out_key_name ua_name
-      out_key_category ua_category
-      out_key_os ua_os
-      out_key_os_version ua_os_version
-      out_key_version ua_version
-      out_key_vendor ua_vendor
-    </filter>
-To filter messages with specified user-agent categories (and merge woothee parser result), configure like this:
-    <filter input.**>
-      type woothee
-      key_name agent
-      filter_categories pc,smartphone,mobilephone,appliance
-      merge_agent_info yes
-    </filter>
+To add woothee parser result into messages:
+    <label @accesslog>
+      <filter input.**>
+        @type woothee
+        key_name agent
+        merge_agent_info yes
+      </filter>
+      <match ...>
+      </match>
+    </label>
+Result messages has attributes like 'agent\_name', 'agent\_category' and 'agent\_os' from woothee parser result. If you want to change attribute names, or want to merge more attributes of browser vendor and its version, write configurations as below:
+    <label @accesslog>
+      <filter input.**>
+        @type woothee
+        key_name agent
+        merge_agent_info yes
+        out_key_name ua_name
+        out_key_category ua_category
+        out_key_os ua_os
+        out_key_os_version ua_os_version
+        out_key_version ua_version
+        out_key_vendor ua_vendor
+      </filter>
+      <match ...>
+      </match>
+    </label>
+To pass messages only with specified user-agent categories (and merge woothee parser result), configure like this:
+    <label @accesslog>
+      <filter input.**>
+        @type woothee
+        key_name agent
+        merge_agent_info yes
+        filter_categories pc,smartphone,mobilephone,appliance
+      </filter> # logs of other categories will be dropped
+      # ...
+    </label>
 Or, you can specify categories to drop (and not to merge woothee result):
-    <filter input.**>
-      type woothee
-      key_name agent
-      drop_categories crawler
-      merge_agent_info false # default
-    </filter>
+    <label @accesslog>
+      <filter input.**>
+        @type woothee
+        key_name agent
+        merge_agent_info false # default
+        drop_categories crawler
+      </filter>
+      # ...
+    </label>
 ### Fast Crawler Filter
 If you want to drop __almost__ all of messages with crawler's user-agent, and not to merge woothee result, you just specify plugin type:
     <filter input.**>
-      type woothee_fast_crawler_filter
+      @type woothee_fast_crawler_filter
       key_name useragent
     </filter>
-'fluent-plugin-woothee' uses 'Woothee.is_crawler' of woothee with this configuration, fast and incomplete method to judge user-agent is crawler or not.
+'fluent-plugin-woothee' uses 'Woothee.is\_crawler' of woothee with this configuration, fast and incomplete method to judge user-agent is crawler or not.
 If you want to drop all of crawlers completely, specify 'type woothee' and 'drop_categories crawler'.
+### Output plugin
+The output version of woothee plugin is not supported in versions for Fluentd v0.14.
 ## TODO
 * patches welcome!

data/fluent-plugin-woothee.gemspec CHANGED

@@ -2,7 +2,7 @@
 Gem::Specification.new do |gem|
   gem.name          = "fluent-plugin-woothee"
-  gem.version       = "0.2.2"
+  gem.version       = "1.0.0"
   gem.authors       = ["TAGOMORI Satoshi"]
   gem.email         = ["tagomoris@gmail.com"]
   gem.description   = %q{parsing by Project Woothee. See https://github.com/woothee/woothee }
@@ -17,6 +17,6 @@ Gem::Specification.new do |gem|
   gem.add_development_dependency "rake"
   gem.add_development_dependency "test-unit", "~> 3.0.2"
-  gem.add_runtime_dependency "fluentd", "< 0.14.0"
+  gem.add_runtime_dependency "fluentd", ">= 0.14.0"
   gem.add_runtime_dependency "woothee", ">= 1.0.0"
 end

data/lib/fluent/plugin/filter_woothee.rb CHANGED

@@ -1,46 +1,47 @@
-class Fluent::WootheeFilter < Fluent::Filter
+require 'fluent/plugin/filter'
+require 'woothee'
+class Fluent::Plugin::WootheeFilter < Fluent::Plugin::Filter
   Fluent::Plugin.register_filter('woothee', self)
   Fluent::Plugin.register_filter('woothee_fast_crawler_filter', self)
-  config_param :fast_crawler_filter_mode, :bool, :default => false
+  config_param :fast_crawler_filter_mode, :bool, default: false
   config_param :key_name, :string
-  config_param :filter_categories, :default => [] do |val|
-    val.split(',').map(&:to_sym)
-  end
-  config_param :drop_categories, :default => [] do |val|
-    val.split(',').map(&:to_sym)
-  end
+  config_param :filter_categories, :array, value_type: :string, default: []
+  config_param :drop_categories, :array, value_type: :string, default: []
   attr_accessor :mode
-  config_param :merge_agent_info, :bool, :default => false
-  config_param :out_key_name, :string, :default => 'agent_name'
-  config_param :out_key_category, :string, :default => 'agent_category'
-  config_param :out_key_os, :string, :default => 'agent_os'
-  config_param :out_key_os_version, :string, :default => nil # supress output
-  config_param :out_key_version, :string, :default => nil # supress output
-  config_param :out_key_vendor, :string, :default => nil # supress output
-  def initialize
-    super
-    require 'woothee'
-  end
+  config_param :merge_agent_info, :bool, default: false
+  config_param :out_key_name,       :string, default: 'agent_name'
+  config_param :out_key_category,   :string, default: 'agent_category'
+  config_param :out_key_os,         :string, default: 'agent_os'
+  config_param :out_key_os_version, :string, default: nil # supress output in default
+  config_param :out_key_version,    :string, default: nil # supress output in default
+  config_param :out_key_vendor,     :string, default: nil # supress output in default
   def configure(conf)
+    specified_type_name = conf['@type']
     super
-    if conf['type'] == 'woothee_fast_crawler_filter' or @fast_crawler_filter_mode
+    @filter_categories = @filter_categories.map(&:to_sym)
+    @drop_categories = @drop_categories.map(&:to_sym)
+    if specified_type_name == 'woothee_fast_crawler_filter' || @fast_crawler_filter_mode
       @fast_crawler_filter_mode = true
-      if @filter_categories.size > 0 or @drop_categories.size > 0 or @merge_agent_info
+      if @filter_categories.size > 0 || @drop_categories.size > 0 || @merge_agent_info
         raise Fluent::ConfigError, "fast_crawler_filter cannot be specified with filter/drop/merge options"
       end
+      define_singleton_method(:filter, method(:filter_fast_crawler))
       return
     end
-    if @filter_categories.size > 0 and @drop_categories.size > 0
+    if @filter_categories.size > 0 && @drop_categories.size > 0
       raise Fluent::ConfigError, "both of 'filter' and 'drop' categories specified"
     elsif @filter_categories.size > 0
       unless @filter_categories.reduce(true){|r,i| r and Woothee::CATEGORY_LIST.include?(i)}
@@ -56,52 +57,47 @@ class Fluent::WootheeFilter < Fluent::Filter
       @mode = :through
     end
-    if @mode == :through and not @merge_agent_info
+    if @mode == :through && ! @merge_agent_info
       raise Fluent::ConfigError, "configured not to do nothing (not to do either filter/drop nor addition of parser result)"
     end
-  end
-  def fast_crawler_filter_stream(tag, es)
-    new_es = Fluent::MultiEventStream.new
+    define_singleton_method(:filter, method(:filter_standard))
+  end
-    es.each do |time,record|
-      unless Woothee.is_crawler(record[@key_name] || '')
-        new_es.add(time, record.dup)
-      end
+  def filter(tag, time, record)
+    # dynamically overwritten by #configure
+    if @fast_crawler_filter_mode
+      filter_fast_crawler(tag, time, record)
+    else
+      filter_standard(tag, time, record)
     end
-    new_es
   end
-  def normal_filter_stream(tag, es)
-    new_es = Fluent::MultiEventStream.new
-    es.each do |time,record|
-      parsed = Woothee.parse(record[@key_name] || '')
-      category = parsed[Woothee::ATTRIBUTE_CATEGORY]
-      next if @mode == :filter and not @filter_categories.include?(category)
-      next if @mode == :drop and @drop_categories.include?(category)
-      if @merge_agent_info
-        record = record.merge({
-            @out_key_name => parsed[Woothee::ATTRIBUTE_NAME],
-            @out_key_category => parsed[Woothee::ATTRIBUTE_CATEGORY].to_s,
-            @out_key_os => parsed[Woothee::ATTRIBUTE_OS]
-          })
-        record[@out_key_os_version] = parsed[Woothee::ATTRIBUTE_OS_VERSION] if @out_key_os_version
-        record[@out_key_version] = parsed[Woothee::ATTRIBUTE_VERSION] if @out_key_version
-        record[@out_key_vendor] = parsed[Woothee::ATTRIBUTE_VENDOR] if @out_key_vendor
-      end
-      new_es.add(time, record.dup)
+  def filter_fast_crawler(tag, time, record)
+    if Woothee.is_crawler(record[@key_name] || '')
+      nil
+    else
+      record
     end
-    new_es
   end
-  def filter_stream(tag, es)
-    if @fast_crawler_filter_mode
-      fast_crawler_filter_stream(tag, es)
-    else
-      normal_filter_stream(tag, es)
+  def filter_standard(tag, time, record)
+    parsed = Woothee.parse(record[@key_name] || '')
+    category = parsed[Woothee::ATTRIBUTE_CATEGORY]
+    return nil if @mode == :filter && !@filter_categories.include?(category)
+    return nil if @mode == :drop && @drop_categories.include?(category)
+    if @merge_agent_info
+      record = record.merge({
+          @out_key_name => parsed[Woothee::ATTRIBUTE_NAME],
+          @out_key_category => parsed[Woothee::ATTRIBUTE_CATEGORY].to_s,
+          @out_key_os => parsed[Woothee::ATTRIBUTE_OS]
+      })
+      record[@out_key_os_version] = parsed[Woothee::ATTRIBUTE_OS_VERSION] if @out_key_os_version
+      record[@out_key_version] = parsed[Woothee::ATTRIBUTE_VERSION] if @out_key_version
+      record[@out_key_vendor] = parsed[Woothee::ATTRIBUTE_VENDOR] if @out_key_vendor
     end
+    record
   end
-end if defined?(Fluent::Filter)
+end

data/test/helper.rb CHANGED

@@ -22,7 +22,6 @@ unless ENV.has_key?('VERBOSE')
   $log = nulllogger
 end
-require 'fluent/plugin/out_woothee'
 require 'fluent/plugin/filter_woothee'
 class Test::Unit::TestCase

data/test/plugin/test_filter_woothee.rb CHANGED

@@ -1,4 +1,5 @@
 require 'helper'
+require 'fluent/test/driver/filter'
 class Fluent::WootheeFilterTest < Test::Unit::TestCase
   # fast crawler filter
@@ -36,13 +37,11 @@ drop_categories crawler,misc
 ]
   def setup
-    omit("Use fluentd v0.12 or later") unless defined?(Fluent::Filter)
     Fluent::Test.setup
   end
-  def create_driver(conf=CONFIG1,tag='test')
-    Fluent::Test::FilterTestDriver.new(Fluent::WootheeFilter, tag).configure(conf)
+  def create_driver(conf=CONFIG1)
+    Fluent::Test::Driver::Filter.new(Fluent::Plugin::WootheeFilter).configure(conf)
   end
   class TestConfigure < self
@@ -105,53 +104,51 @@ drop_categories crawler,misc
   def test_filter_fast_crawler_filter_stream
     d = create_driver CONFIG0
     time = Time.parse('2012-07-20 16:19:00').to_i
-    d.run do
-      d.filter({'useragent' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', 'value' => 1}, time)
-      d.filter({'useragent' => 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)', 'value' => 2}, time)
-      d.filter({'useragent' => 'Mozilla/5.0 (iPad; U; CPU OS 4_3_2 like Mac OS X; ja-jp) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5', 'value' => 3}, time)
-      d.filter({'useragent' => 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)', 'value' => 4}, time)
-      d.filter({'useragent' => 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)', 'value' => 5}, time)
-      d.filter({'useragent' => 'Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)', 'value' => 6}, time)
-      d.filter({'useragent' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; ja-jp) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1', 'value' => 7}, time)
-      d.filter({'useragent' => 'Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)', 'value' => 8}, time)
+    d.run(default_tag: 'test') do
+      d.feed(time, {'useragent' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', 'value' => 1})
+      d.feed(time, {'useragent' => 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)', 'value' => 2})
+      d.feed(time, {'useragent' => 'Mozilla/5.0 (iPad; U; CPU OS 4_3_2 like Mac OS X; ja-jp) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5', 'value' => 3})
+      d.feed(time, {'useragent' => 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)', 'value' => 4})
+      d.feed(time, {'useragent' => 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)', 'value' => 5})
+      d.feed(time, {'useragent' => 'Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)', 'value' => 6})
+      d.feed(time, {'useragent' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; ja-jp) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1', 'value' => 7})
+      d.feed(time, {'useragent' => 'Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)', 'value' => 8})
     end
-    filtered = d.filtered_as_array
+    filtered = d.filtered
     assert_equal 4, filtered.size
-    assert_equal 'test', filtered[0][0]
-    assert_equal time, filtered[0][1]
-    assert_equal 'Mozilla/5.0 (iPad; U; CPU OS 4_3_2 like Mac OS X; ja-jp) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5', filtered[0][2]['useragent']
-    assert_equal 3, filtered[0][2]['value']
-    assert_equal 2, filtered[0][2].keys.size
+    assert_equal time, filtered[0][0]
+    assert_equal 'Mozilla/5.0 (iPad; U; CPU OS 4_3_2 like Mac OS X; ja-jp) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5', filtered[0][1]['useragent']
+    assert_equal 3, filtered[0][1]['value']
+    assert_equal 2, filtered[0][1].keys.size
-    assert_equal 4, filtered[1][2]['value']
-    assert_equal 6, filtered[2][2]['value']
-    assert_equal 7, filtered[3][2]['value']
+    assert_equal 4, filtered[1][1]['value']
+    assert_equal 6, filtered[2][1]['value']
+    assert_equal 7, filtered[3][1]['value']
   end
   # through & merge
   def test_filter_through
-    d = create_driver(CONFIG1, 'test.message')
+    d = create_driver(CONFIG1)
     time = Time.parse('2012-07-20 16:40:30').to_i
-    d.run do
-      d.filter({'value' => 0, 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'}, time)
-      d.filter({'value' => 1, 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.filter({'value' => 2, 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.filter({'value' => 3, 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'}, time)
-      d.filter({'value' => 4, 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'}, time)
-      d.filter({'value' => 5, 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'}, time)
-      d.filter({'value' => 6, 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'}, time)
-      d.filter({'value' => 7, 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'}, time)
+    d.run(default_tag: 'test.message') do
+      d.feed(time, {'value' => 0, 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'})
+      d.feed(time, {'value' => 1, 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'})
+      d.feed(time, {'value' => 2, 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'})
+      d.feed(time, {'value' => 3, 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'})
+      d.feed(time, {'value' => 4, 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'})
+      d.feed(time, {'value' => 5, 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'})
+      d.feed(time, {'value' => 6, 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'})
+      d.feed(time, {'value' => 7, 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'})
     end
-    filtered = d.filtered_as_array
+    filtered = d.filtered
     assert_equal 8, filtered.size
-    assert_equal 'test.message', filtered[0][0]
-    assert_equal time, filtered[0][1]
+    assert_equal time, filtered[0][0]
     # 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'
-    m = filtered[0][2]
+    m = filtered[0][1]
     assert_equal 0, m['value']
     assert_equal 'Internet Explorer', m['agent_name']
     assert_equal 'pc', m['agent_category']
@@ -159,49 +156,49 @@ drop_categories crawler,misc
     assert_equal 5, m.keys.size
     # 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = filtered[1][2]
+    m = filtered[1][1]
     assert_equal 1, m['value']
     assert_equal 'Firefox', m['agent_name']
     assert_equal 'pc', m['agent_category']
     assert_equal 'Windows Vista', m['agent_os']
     # 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = filtered[2][2]
+    m = filtered[2][1]
     assert_equal 2, m['value']
     assert_equal 'Firefox', m['agent_name']
     assert_equal 'pc', m['agent_category']
     assert_equal 'Linux', m['agent_os']
     # 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'
-    m = filtered[3][2]
+    m = filtered[3][1]
     assert_equal 3, m['value']
     assert_equal 'Safari', m['agent_name']
     assert_equal 'smartphone', m['agent_category']
     assert_equal 'Android', m['agent_os']
     # 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'
-    m = filtered[4][2]
+    m = filtered[4][1]
     assert_equal 4, m['value']
     assert_equal 'docomo', m['agent_name']
     assert_equal 'mobilephone', m['agent_category']
     assert_equal 'docomo', m['agent_os']
     # 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'
-    m = filtered[5][2]
+    m = filtered[5][1]
     assert_equal 5, m['value']
     assert_equal 'PlayStation Vita', m['agent_name']
     assert_equal 'appliance', m['agent_category']
     assert_equal 'PlayStation Vita', m['agent_os']
     # 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'
-    m = filtered[6][2]
+    m = filtered[6][1]
     assert_equal 6, m['value']
     assert_equal 'Google Desktop', m['agent_name']
     assert_equal 'misc', m['agent_category']
     assert_equal 'UNKNOWN', m['agent_os']
     # 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'
-    m = filtered[7][2]
+    m = filtered[7][1]
     assert_equal 7, m['value']
     assert_equal 'msnbot', m['agent_name']
     assert_equal 'crawler', m['agent_category']
@@ -210,26 +207,25 @@ drop_categories crawler,misc
   # filter & merge
   def test_filter_stream
-    d = create_driver(CONFIG2, 'test.message')
+    d = create_driver(CONFIG2)
     time = Time.parse('2012-07-20 16:40:30').to_i
-    d.run do
-      d.filter({'value' => 0, 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'}, time)
-      d.filter({'value' => 1, 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.filter({'value' => 2, 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.filter({'value' => 3, 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'}, time)
-      d.filter({'value' => 4, 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'}, time)
-      d.filter({'value' => 5, 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'}, time)
-      d.filter({'value' => 6, 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'}, time)
-      d.filter({'value' => 7, 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'}, time)
+    d.run(default_tag: 'test.message') do
+      d.feed(time, {'value' => 0, 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'})
+      d.feed(time, {'value' => 1, 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'})
+      d.feed(time, {'value' => 2, 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'})
+      d.feed(time, {'value' => 3, 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'})
+      d.feed(time, {'value' => 4, 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'})
+      d.feed(time, {'value' => 5, 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'})
+      d.feed(time, {'value' => 6, 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'})
+      d.feed(time, {'value' => 7, 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'})
     end
-    filtered = d.filtered_as_array
+    filtered = d.filtered
     assert_equal 6, filtered.size
-    assert_equal 'test.message', filtered[0][0]
-    assert_equal time, filtered[0][1]
+    assert_equal time, filtered[0][0]
     # 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'
-    m = filtered[0][2]
+    m = filtered[0][1]
     assert_equal 8, m.keys.size
     assert_equal 0, m['value']
     assert_equal 'Internet Explorer', m['ua_name']
@@ -240,7 +236,7 @@ drop_categories crawler,misc
     assert_equal '10.0', m['ua_version']
     # 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = filtered[1][2]
+    m = filtered[1][1]
     assert_equal 1, m['value']
     assert_equal 'Firefox', m['ua_name']
     assert_equal 'pc', m['ua_category']
@@ -250,7 +246,7 @@ drop_categories crawler,misc
     assert_equal '9.0.1', m['ua_version']
     # 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = filtered[2][2]
+    m = filtered[2][1]
     assert_equal 2, m['value']
     assert_equal 'Firefox', m['ua_name']
     assert_equal 'pc', m['ua_category']
@@ -260,7 +256,7 @@ drop_categories crawler,misc
     assert_equal '9.0.1', m['ua_version']
     # 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'
-    m = filtered[3][2]
+    m = filtered[3][1]
     assert_equal 3, m['value']
     assert_equal 'Safari', m['ua_name']
     assert_equal 'smartphone', m['ua_category']
@@ -270,7 +266,7 @@ drop_categories crawler,misc
     assert_equal '4.0', m['ua_version']
     # 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'
-    m = filtered[4][2]
+    m = filtered[4][1]
     assert_equal 4, m['value']
     assert_equal 'docomo', m['ua_name']
     assert_equal 'mobilephone', m['ua_category']
@@ -280,7 +276,7 @@ drop_categories crawler,misc
     assert_equal 'N505i', m['ua_version']
     # 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'
-    m = filtered[5][2]
+    m = filtered[5][1]
     assert_equal 5, m['value']
     assert_equal 'PlayStation Vita', m['ua_name']
     assert_equal 'appliance', m['ua_category']
@@ -292,47 +288,46 @@ drop_categories crawler,misc
   # drop & non-merge
   def test_filter_drop
-    d = create_driver(CONFIG3, 'test.message')
+    d = create_driver(CONFIG3)
     time = Time.parse('2012-07-20 16:40:30').to_i
-    d.run do
-      d.filter({'value' => 0, 'user_agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'}, time)
-      d.filter({'value' => 1, 'user_agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.filter({'value' => 2, 'user_agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.filter({'value' => 3, 'user_agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'}, time)
-      d.filter({'value' => 4, 'user_agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'}, time)
-      d.filter({'value' => 5, 'user_agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'}, time)
-      d.filter({'value' => 6, 'user_agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'}, time)
-      d.filter({'value' => 7, 'user_agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'}, time)
+    d.run(default_tag: 'test.message') do
+      d.feed(time, {'value' => 0, 'user_agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'})
+      d.feed(time, {'value' => 1, 'user_agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'})
+      d.feed(time, {'value' => 2, 'user_agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'})
+      d.feed(time, {'value' => 3, 'user_agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'})
+      d.feed(time, {'value' => 4, 'user_agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'})
+      d.feed(time, {'value' => 5, 'user_agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'})
+      d.feed(time, {'value' => 6, 'user_agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'})
+      d.feed(time, {'value' => 7, 'user_agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'})
     end
-    filtered = d.filtered_as_array
+    filtered = d.filtered
     assert_equal 6, filtered.size
-    assert_equal 'test.message', filtered[0][0]
-    assert_equal time, filtered[0][1]
+    assert_equal time, filtered[0][0]
     # 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'
-    m = filtered[0][2]
+    m = filtered[0][1]
     assert_equal 0, m['value']
     assert_equal 2, m.keys.size
     # 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = filtered[1][2]
+    m = filtered[1][1]
     assert_equal 1, m['value']
     # 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = filtered[2][2]
+    m = filtered[2][1]
     assert_equal 2, m['value']
     # 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'
-    m = filtered[3][2]
+    m = filtered[3][1]
     assert_equal 3, m['value']
     # 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'
-    m = filtered[4][2]
+    m = filtered[4][1]
     assert_equal 4, m['value']
     # 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'
-    m = filtered[5][2]
+    m = filtered[5][1]
     assert_equal 5, m['value']
   end
 end

metadata CHANGED

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-woothee
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 1.0.0
 platform: ruby
 authors:
 - TAGOMORI Satoshi
@@ -42,14 +42,14 @@ dependencies:
   name: fluentd
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "<"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 0.14.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "<"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 0.14.0
 - !ruby/object:Gem::Dependency
@@ -81,10 +81,8 @@ files:
 - Rakefile
 - fluent-plugin-woothee.gemspec
 - lib/fluent/plugin/filter_woothee.rb
-- lib/fluent/plugin/out_woothee.rb
 - test/helper.rb
 - test/plugin/test_filter_woothee.rb
-- test/plugin/test_out_woothee.rb
 homepage: https://github.com/woothee/fluent-plugin-woothee
 licenses:
 - Apache-2.0
@@ -113,4 +111,3 @@ summary: Fluentd plugin to parse UserAgent strings with woothee parser. It adds
 test_files:
 - test/helper.rb
 - test/plugin/test_filter_woothee.rb
-- test/plugin/test_out_woothee.rb

data/lib/fluent/plugin/out_woothee.rb DELETED

@@ -1,154 +0,0 @@
-class Fluent::WootheeOutput < Fluent::Output
-  Fluent::Plugin.register_output('woothee', self)
-  Fluent::Plugin.register_output('woothee_fast_crawler_filter', self)
-  # Define `router` method of v0.12 to support v0.10 or earlier
-  unless method_defined?(:router)
-    define_method("router") { Fluent::Engine }
-  end
-  # Define `log` method for v0.10.42 or earlier
-  unless method_defined?(:log)
-    define_method("log") { $log }
-  end
-  config_param :tag, :string, :default => nil
-  config_param :remove_prefix, :string, :default => nil
-  config_param :add_prefix, :string, :default => nil
-  config_param :fast_crawler_filter_mode, :bool, :default => false
-  config_param :key_name, :string
-  config_param :filter_categories, :default => [] do |val|
-    val.split(',').map(&:to_sym)
-  end
-  config_param :drop_categories, :default => [] do |val|
-    val.split(',').map(&:to_sym)
-  end
-  attr_accessor :mode
-  config_param :merge_agent_info, :bool, :default => false
-  config_param :out_key_name, :string, :default => 'agent_name'
-  config_param :out_key_category, :string, :default => 'agent_category'
-  config_param :out_key_os, :string, :default => 'agent_os'
-  config_param :out_key_os_version, :string, :default => nil # supress output
-  config_param :out_key_version, :string, :default => nil # supress output
-  config_param :out_key_vendor, :string, :default => nil # supress output
-  def initialize
-    super
-    require 'woothee'
-  end
-  def configure(conf)
-    super
-    # tag ->
-    if not @tag and not @remove_prefix and not @add_prefix
-      raise Fluent::ConfigError, "missing both of remove_prefix and add_prefix"
-    end
-    if @tag and (@remove_prefix or @add_prefix)
-      raise Fluent::ConfigError, "both of tag and remove_prefix/add_prefix must not be specified"
-    end
-    if @remove_prefix
-      @removed_prefix_string = @remove_prefix + '.'
-      @removed_length = @removed_prefix_string.length
-    end
-    if @add_prefix
-      @added_prefix_string = @add_prefix + '.'
-    end
-    # <- tag
-    if conf['type'] == 'woothee_fast_crawler_filter' or @fast_crawler_filter_mode
-      @fast_crawler_filter_mode = true
-      if @filter_categories.size > 0 or @drop_categories.size > 0 or @merge_agent_info
-        raise Fluent::ConfigError, "fast_crawler_filter cannot be specified with filter/drop/merge options"
-      end
-      return
-    end
-    if @filter_categories.size > 0 and @drop_categories.size > 0
-      raise Fluent::ConfigError, "both of 'filter' and 'drop' categories specified"
-    elsif @filter_categories.size > 0
-      unless @filter_categories.reduce(true){|r,i| r and Woothee::CATEGORY_LIST.include?(i)}
-        raise Fluent::ConfigError, "filter_categories has invalid category name"
-      end
-      @mode = :filter
-    elsif @drop_categories.size > 0
-      unless @drop_categories.reduce(true){|r,i| r and Woothee::CATEGORY_LIST.include?(i)}
-        raise Fluent::ConfigError, "drop_categories has invalid category name"
-      end
-      @mode = :drop
-    else
-      @mode = :through
-    end
-    if @mode == :through and not @merge_agent_info
-      raise Fluent::ConfigError, "configured not to do nothing (not to do either filter/drop nor addition of parser result)"
-    end
-  end
-  def tag_mangle(tag)
-    if @tag
-      @tag
-    else
-      if @remove_prefix and
-          ( (tag.start_with?(@removed_prefix_string) and tag.length > @removed_length) or tag == @remove_prefix)
-        tag = tag[@removed_length..-1]
-      end
-      if @add_prefix
-        tag = if tag and tag.length > 0
-                @added_prefix_string + tag
-              else
-                @add_prefix
-              end
-      end
-      tag
-    end
-  end
-  def fast_crawler_filter_emit(tag, es)
-    es.each do |time,record|
-      unless Woothee.is_crawler(record[@key_name] || '')
-        router.emit(tag, time, record)
-      end
-    end
-  end
-  def normal_emit(tag, es)
-    es.each do |time,record|
-      parsed = Woothee.parse(record[@key_name] || '')
-      category = parsed[Woothee::ATTRIBUTE_CATEGORY]
-      next if @mode == :filter and not @filter_categories.include?(category)
-      next if @mode == :drop and @drop_categories.include?(category)
-      if @merge_agent_info
-        record = record.merge({
-            @out_key_name => parsed[Woothee::ATTRIBUTE_NAME],
-            @out_key_category => parsed[Woothee::ATTRIBUTE_CATEGORY].to_s,
-            @out_key_os => parsed[Woothee::ATTRIBUTE_OS]
-          })
-        record[@out_key_os_version] = parsed[Woothee::ATTRIBUTE_OS_VERSION] if @out_key_os_version
-        record[@out_key_version] = parsed[Woothee::ATTRIBUTE_VERSION] if @out_key_version
-        record[@out_key_vendor] = parsed[Woothee::ATTRIBUTE_VENDOR] if @out_key_vendor
-      end
-      router.emit(tag, time, record)
-    end
-  end
-  def emit(tag, es, chain)
-    tag = tag_mangle(tag)
-    if @fast_crawler_filter_mode
-      fast_crawler_filter_emit(tag, es)
-    else
-      normal_emit(tag, es)
-    end
-    chain.next
-  end
-end

data/test/plugin/test_out_woothee.rb DELETED

@@ -1,364 +0,0 @@
-require 'helper'
-class Fluent::WootheeOutputTest < Test::Unit::TestCase
-  # fast crawler filter
-  CONFIG0 = %[
-type woothee_fast_crawler_filter
-key_name useragent
-tag filtered
-]
-  # through & merge
-  CONFIG1 = %[
-type woothee
-key_name agent
-remove_prefix test
-add_prefix merged
-merge_agent_info yes
-]
-  # filter & merge
-  CONFIG2 = %[
-type woothee
-key_name agent
-filter_categories pc,smartphone,mobilephone,appliance
-remove_prefix test
-add_prefix merged
-merge_agent_info yes
-out_key_name ua_name
-out_key_category ua_category
-out_key_os ua_os
-out_key_os_version ua_os_version
-out_key_version ua_version
-out_key_vendor ua_vendor
-]
-  # drop & non-merge
-  CONFIG3 = %[
-type woothee
-key_name user_agent
-drop_categories crawler,misc
-tag selected
-]
-  def setup
-    Fluent::Test.setup
-  end
-  def create_driver(conf=CONFIG1,tag='test')
-    Fluent::Test::OutputTestDriver.new(Fluent::WootheeOutput, tag).configure(conf)
-  end
-  def test_configure
-    # fast_crawler_filter
-    d = create_driver CONFIG0
-    assert_equal true, d.instance.fast_crawler_filter_mode
-    assert_equal 'useragent', d.instance.key_name
-    assert_equal 'filtered', d.instance.tag
-    # through & merge
-    d = create_driver CONFIG1
-    assert_equal false, d.instance.fast_crawler_filter_mode
-    assert_equal 'agent', d.instance.key_name
-    assert_equal 'test', d.instance.remove_prefix
-    assert_equal 'merged', d.instance.add_prefix
-    assert_equal 0, d.instance.filter_categories.size
-    assert_equal 0, d.instance.drop_categories.size
-    assert_equal :through, d.instance.mode
-    assert_equal true, d.instance.merge_agent_info
-    assert_equal 'agent_name', d.instance.out_key_name
-    assert_equal 'agent_category', d.instance.out_key_category
-    assert_equal 'agent_os', d.instance.out_key_os
-    assert_nil d.instance.out_key_version
-    assert_nil d.instance.out_key_vendor
-    # filter & merge
-    d = create_driver CONFIG2
-    assert_equal false, d.instance.fast_crawler_filter_mode
-    assert_equal 'agent', d.instance.key_name
-    assert_equal 'test', d.instance.remove_prefix
-    assert_equal 'merged', d.instance.add_prefix
-    assert_equal 4, d.instance.filter_categories.size
-    assert_equal [:pc,:smartphone,:mobilephone,:appliance], d.instance.filter_categories
-    assert_equal 0, d.instance.drop_categories.size
-    assert_equal :filter, d.instance.mode
-    assert_equal true, d.instance.merge_agent_info
-    assert_equal 'ua_name', d.instance.out_key_name
-    assert_equal 'ua_category', d.instance.out_key_category
-    assert_equal 'ua_os', d.instance.out_key_os
-    assert_equal 'ua_os_version', d.instance.out_key_os_version
-    assert_equal 'ua_version', d.instance.out_key_version
-    assert_equal 'ua_vendor', d.instance.out_key_vendor
-    # drop & non-merge
-    d = create_driver CONFIG3
-    assert_equal false, d.instance.fast_crawler_filter_mode
-    assert_equal 'user_agent', d.instance.key_name
-    assert_equal 'selected', d.instance.tag
-    assert_equal 0, d.instance.filter_categories.size
-    assert_equal 2, d.instance.drop_categories.size
-    assert_equal [:crawler,:misc], d.instance.drop_categories
-    assert_equal :drop, d.instance.mode
-    assert_equal false, d.instance.merge_agent_info
-  end
-  def test_tag_mangle
-    p = create_driver(CONFIG0).instance
-    assert_equal 'filtered', p.tag_mangle('data')
-    assert_equal 'filtered', p.tag_mangle('test.data')
-    assert_equal 'filtered', p.tag_mangle('test.test.data')
-    assert_equal 'filtered', p.tag_mangle('test')
-    p = create_driver(CONFIG1).instance
-    assert_equal 'merged.data', p.tag_mangle('data')
-    assert_equal 'merged.data', p.tag_mangle('test.data')
-    assert_equal 'merged.test.data', p.tag_mangle('test.test.data')
-    assert_equal 'merged', p.tag_mangle('test')
-    p = create_driver(CONFIG3).instance
-    assert_equal 'selected', p.tag_mangle('data')
-    assert_equal 'selected', p.tag_mangle('test.data')
-    assert_equal 'selected', p.tag_mangle('test.test.data')
-    assert_equal 'selected', p.tag_mangle('test')
-  end
-  def test_emit_fast_crawler_filter
-    d = create_driver CONFIG0
-    time = Time.parse('2012-07-20 16:19:00').to_i
-    d.run do
-      d.emit({'useragent' => 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)', 'value' => 1}, time)
-      d.emit({'useragent' => 'Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)', 'value' => 2}, time)
-      d.emit({'useragent' => 'Mozilla/5.0 (iPad; U; CPU OS 4_3_2 like Mac OS X; ja-jp) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5', 'value' => 3}, time)
-      d.emit({'useragent' => 'Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0)', 'value' => 4}, time)
-      d.emit({'useragent' => 'Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)', 'value' => 5}, time)
-      d.emit({'useragent' => 'Mozilla/5.0 (compatible; Rakutenbot/1.0; +http://dynamic.rakuten.co.jp/bot.html)', 'value' => 6}, time)
-      d.emit({'useragent' => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; ja-jp) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1', 'value' => 7}, time)
-      d.emit({'useragent' => 'Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)', 'value' => 8}, time)
-    end
-    emits = d.emits
-    assert_equal 4, emits.size
-    assert_equal 'filtered', emits[0][0]
-    assert_equal time, emits[0][1]
-    assert_equal 'Mozilla/5.0 (iPad; U; CPU OS 4_3_2 like Mac OS X; ja-jp) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8H7 Safari/6533.18.5', emits[0][2]['useragent']
-    assert_equal 3, emits[0][2]['value']
-    assert_equal 2, emits[0][2].keys.size
-    assert_equal 4, emits[1][2]['value']
-    assert_equal 6, emits[2][2]['value']
-    assert_equal 7, emits[3][2]['value']
-  end
-#   # through & merge
-  def test_emit_through
-    d = create_driver(CONFIG1, 'test.message')
-    time = Time.parse('2012-07-20 16:40:30').to_i
-    d.run do
-      d.emit({'value' => 0, 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'}, time)
-      d.emit({'value' => 1, 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.emit({'value' => 2, 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.emit({'value' => 3, 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'}, time)
-      d.emit({'value' => 4, 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'}, time)
-      d.emit({'value' => 5, 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'}, time)
-      d.emit({'value' => 6, 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'}, time)
-      d.emit({'value' => 7, 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'}, time)
-    end
-    emits = d.emits
-    assert_equal 8, emits.size
-    assert_equal 'merged.message', emits[0][0]
-    assert_equal time, emits[0][1]
-    # 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'
-    m = emits[0][2]
-    assert_equal 0, m['value']
-    assert_equal 'Internet Explorer', m['agent_name']
-    assert_equal 'pc', m['agent_category']
-    assert_equal 'Windows 8', m['agent_os']
-    assert_equal 5, m.keys.size
-    # 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = emits[1][2]
-    assert_equal 1, m['value']
-    assert_equal 'Firefox', m['agent_name']
-    assert_equal 'pc', m['agent_category']
-    assert_equal 'Windows Vista', m['agent_os']
-    # 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = emits[2][2]
-    assert_equal 2, m['value']
-    assert_equal 'Firefox', m['agent_name']
-    assert_equal 'pc', m['agent_category']
-    assert_equal 'Linux', m['agent_os']
-    # 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'
-    m = emits[3][2]
-    assert_equal 3, m['value']
-    assert_equal 'Safari', m['agent_name']
-    assert_equal 'smartphone', m['agent_category']
-    assert_equal 'Android', m['agent_os']
-    # 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'
-    m = emits[4][2]
-    assert_equal 4, m['value']
-    assert_equal 'docomo', m['agent_name']
-    assert_equal 'mobilephone', m['agent_category']
-    assert_equal 'docomo', m['agent_os']
-    # 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'
-    m = emits[5][2]
-    assert_equal 5, m['value']
-    assert_equal 'PlayStation Vita', m['agent_name']
-    assert_equal 'appliance', m['agent_category']
-    assert_equal 'PlayStation Vita', m['agent_os']
-    # 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'
-    m = emits[6][2]
-    assert_equal 6, m['value']
-    assert_equal 'Google Desktop', m['agent_name']
-    assert_equal 'misc', m['agent_category']
-    assert_equal 'UNKNOWN', m['agent_os']
-    # 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'
-    m = emits[7][2]
-    assert_equal 7, m['value']
-    assert_equal 'msnbot', m['agent_name']
-    assert_equal 'crawler', m['agent_category']
-    assert_equal 'UNKNOWN', m['agent_os']
-  end
-#   # filter & merge
-  def test_emit_filter
-    d = create_driver(CONFIG2, 'test.message')
-    time = Time.parse('2012-07-20 16:40:30').to_i
-    d.run do
-      d.emit({'value' => 0, 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'}, time)
-      d.emit({'value' => 1, 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.emit({'value' => 2, 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.emit({'value' => 3, 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'}, time)
-      d.emit({'value' => 4, 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'}, time)
-      d.emit({'value' => 5, 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'}, time)
-      d.emit({'value' => 6, 'agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'}, time)
-      d.emit({'value' => 7, 'agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'}, time)
-    end
-    emits = d.emits
-    assert_equal 6, emits.size
-    assert_equal 'merged.message', emits[0][0]
-    assert_equal time, emits[0][1]
-    # 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'
-    m = emits[0][2]
-    assert_equal 8, m.keys.size
-    assert_equal 0, m['value']
-    assert_equal 'Internet Explorer', m['ua_name']
-    assert_equal 'pc', m['ua_category']
-    assert_equal 'Windows 8', m['ua_os']
-    assert_equal 'NT 6.2', m['ua_os_version']
-    assert_equal 'Microsoft', m['ua_vendor']
-    assert_equal '10.0', m['ua_version']
-    # 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = emits[1][2]
-    assert_equal 1, m['value']
-    assert_equal 'Firefox', m['ua_name']
-    assert_equal 'pc', m['ua_category']
-    assert_equal 'Windows Vista', m['ua_os']
-    assert_equal 'NT 6.0', m['ua_os_version']
-    assert_equal 'Mozilla', m['ua_vendor']
-    assert_equal '9.0.1', m['ua_version']
-    # 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = emits[2][2]
-    assert_equal 2, m['value']
-    assert_equal 'Firefox', m['ua_name']
-    assert_equal 'pc', m['ua_category']
-    assert_equal 'Linux', m['ua_os']
-    assert_equal 'UNKNOWN', m['ua_os_version']
-    assert_equal 'Mozilla', m['ua_vendor']
-    assert_equal '9.0.1', m['ua_version']
-    # 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'
-    m = emits[3][2]
-    assert_equal 3, m['value']
-    assert_equal 'Safari', m['ua_name']
-    assert_equal 'smartphone', m['ua_category']
-    assert_equal 'Android', m['ua_os']
-    assert_equal '3.1', m['ua_os_version']
-    assert_equal 'Apple', m['ua_vendor']
-    assert_equal '4.0', m['ua_version']
-    # 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'
-    m = emits[4][2]
-    assert_equal 4, m['value']
-    assert_equal 'docomo', m['ua_name']
-    assert_equal 'mobilephone', m['ua_category']
-    assert_equal 'docomo', m['ua_os']
-    assert_equal 'UNKNOWN', m['ua_os_version']
-    assert_equal 'docomo', m['ua_vendor']
-    assert_equal 'N505i', m['ua_version']
-    # 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'
-    m = emits[5][2]
-    assert_equal 5, m['value']
-    assert_equal 'PlayStation Vita', m['ua_name']
-    assert_equal 'appliance', m['ua_category']
-    assert_equal 'PlayStation Vita', m['ua_os']
-    assert_equal '1.51', m['ua_os_version']
-    assert_equal 'Sony', m['ua_vendor']
-    assert_equal 'UNKNOWN', m['ua_version']
-  end
-#   # drop & non-merge
-  def test_emit_drop
-    d = create_driver(CONFIG3, 'test.message')
-    time = Time.parse('2012-07-20 16:40:30').to_i
-    d.run do
-      d.emit({'value' => 0, 'user_agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'}, time)
-      d.emit({'value' => 1, 'user_agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.emit({'value' => 2, 'user_agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'}, time)
-      d.emit({'value' => 3, 'user_agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'}, time)
-      d.emit({'value' => 4, 'user_agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'}, time)
-      d.emit({'value' => 5, 'user_agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'}, time)
-      d.emit({'value' => 6, 'user_agent' => 'Mozilla/5.0 (compatible; Google Desktop/5.9.1005.12335; http://desktop.google.com/)'}, time)
-      d.emit({'value' => 7, 'user_agent' => 'msnbot/1.1 (+http://search.msn.com/msnbot.htm)'}, time)
-    end
-    emits = d.emits
-    assert_equal 6, emits.size
-    assert_equal 'selected', emits[0][0]
-    assert_equal time, emits[0][1]
-    # 'agent' => 'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)'
-    m = emits[0][2]
-    assert_equal 0, m['value']
-    assert_equal 2, m.keys.size
-    # 'agent' => 'Mozilla/5.0 (Windows NT 6.0; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = emits[1][2]
-    assert_equal 1, m['value']
-    # 'agent' => 'Mozilla/5.0 (Ubuntu; X11; Linux i686; rv:9.0.1) Gecko/20100101 Firefox/9.0.1'
-    m = emits[2][2]
-    assert_equal 2, m['value']
-    # 'agent' => 'Mozilla/5.0 (Linux; U; Android 3.1; ja-jp; L-06C Build/HMJ37) AppleWebKit/534.13 (KHTML, like Gecko) Version/4.0 Safari/534.13'
-    m = emits[3][2]
-    assert_equal 3, m['value']
-    # 'agent' => 'DoCoMo/1.0/N505i/c20/TB/W24H12'
-    m = emits[4][2]
-    assert_equal 4, m['value']
-    # 'agent' => 'Mozilla/5.0 (PlayStation Vita 1.51) AppleWebKit/531.22.8 (KHTML, like Gecko) Silk/3.2'
-    m = emits[5][2]
-    assert_equal 5, m['value']
-  end
-end