fluent-plugin-rewrite-tag-filter 1.1.1 → 1.2.0

data/README.md

@@ -0,0 +1,162 @@
+# fluent-plugin-rewrite-tag-filter
+
+## Overview
+
+### RewriteTagFilterOutput
+
+Fluentd Output filter plugin. It has designed to rewrite tag like mod_rewrite.  
+Re-emmit a record with rewrited tag when a value matches with the regular expression.  
+Also you can change a tag from apache log by domain, status-code(ex. 500 error),  
+user-agent, request-uri, regex-backreference and so on with regular expression.
+
+## Installation
+
+```
+# native gem
+gem install fluent-plugin-rewrite-tag-filter
+
+# td-agent gem
+/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-rewrite-tag-filter
+```
+
+## Configuration
+
+### Syntax
+
+```
+rewruterule<num:1-200> <attribute> <regex_pattern> <new_tag>
+
+# Optional: Capitalize every matched regex backreference. (ex: $1, $2)
+capitalize_regex_backreference <yes/no> (default no)
+```
+
+### Usage
+
+It's a sample to exclude some static file log before split tag by domain.
+
+```
+<source>
+  type tail
+  path /var/log/httpd/access_log
+  format /^(?<domain>[^ ]*) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/
+  time_format %d/%b/%Y:%H:%M:%S %z
+  tag td.apache.access
+  pos_file /var/log/td-agent/apache_access.pos
+</source>
+
+# At rewriterule4, "site.$2$1" to be "site.ExampleMail" by capitalize_regex_backreference option.
+<match td.apache.access>
+  type rewrite_tag_filter
+  capitalize_regex_backreference yes
+  rewriterule1 path   \.(gif|jpe?g|png|pdf|zip)$  clear
+  rewriterule2 domain ^maps\.example\.com$        site.ExampleMaps
+  rewriterule3 domain ^news\.example\.com$        site.ExampleNews
+  rewriterule4 domain ^(mail)\.(example)\.com$    site.$2$1
+  rewriterule5 domain .+                          site.unmatched
+</match>
+
+<match site.*>
+  type mongo
+  host localhost
+  database apache_access
+  remove_tag_prefix site
+  tag_mapped
+  capped
+  capped_size 100m
+</match>
+
+<match clear>
+  type null
+</match>
+```
+
+### Result
+
+```
+$ mongo
+MongoDB shell version: 2.2.0
+> use apache_access
+switched to db apache_access
+> show collections
+ExampleMaps
+ExampleNews
+ExampleMail
+unmatched
+```
+
+### Debug
+
+On starting td-agent, Logging supported like below.
+
+```
+$ tailf /var/log/td-agent/td-agent.log
+2012-09-16 18:10:51 +0900: adding match pattern="td.apache.access" type="rewrite_tag_filter"
+2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [1, "path", /\.(gif|jpe?g|png|pdf|zip)$/, "clear"]
+2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [2, "domain", /^maps\.example\.com$/, "site.ExampleMaps"]
+2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [3, "domain", /^news\.example\.com$/, "site.ExampleNews"]
+2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [4, "domain", /^(mail)\.(example)\.com$/, "site.$2$1"]
+2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [5, "domain", /.+/, "site.unmatched"]
+```
+
+### Tag placeholder
+
+Supporing these placeholder for new_tag(rewrited tag).  
+It's available to use this placeholder with `remove_tag_prefix` option.  
+This feature adds removing tag prefix for `${tag}` or `__TAG__` in placeholder.
+
+- `${hostname}`
+- `__HOSTNAME__`
+- `${tag}`
+- `__TAG__`
+
+It's a sample to rewrite a tag with placeholder.
+
+```
+# It will get "rewrited.access.ExampleMail"
+<match apache.access>
+  type rewrite_tag_filter
+  rewriterule1  domain  ^(mail)\.(example)\.com$  rewrited.${tag}.$2$1
+  remove_tag_prefix apache
+</match>
+
+# It will get "rewrited.ExampleMail.app30-124.foo.com" when hostname is "app30-124.foo.com"
+<match apache.access>
+  type rewrite_tag_filter
+  rewriterule1  domain  ^(mail)\.(example)\.com$  rewrited.$2$1.${hostname}
+</match>
+```
+
+## Example
+
+- Example1: how to analyze response_time, response_code and user_agent for each virtual domain websites.  
+https://github.com/y-ken/fluent-plugin-rewrite-tag-filter/blob/master/example.conf
+
+- Example2: how to exclude specified patterns before analyze response_time for each virtual domain websites.  
+https://github.com/y-ken/fluent-plugin-rewrite-tag-filter/blob/master/example2.conf
+
+## Related Articles
+
+- 自在にタグを書き換える fluent-plugin-rewrite-tag-filter でログ解析が捗るお話 #fluentd  
+http://d.hatena.ne.jp/yoshi-ken/20120701/1341137269
+
+- Fluentd & TreasureDataで こっそり始めるログ集計 Fluentd Meetup #2 @mikeda  
+http://www.slideshare.net/baguzy/fluentd-meetup-2-14073930
+
+- 似てる #fluentd プラグインの比較  
+http://matsumana.wordpress.com/2012/11/15/%E4%BC%BC%E3%81%A6%E3%82%8B-fluentd-%E3%83%97%E3%83%A9%E3%82%B0%E3%82%A4%E3%83%B3%E3%81%AE%E6%AF%94%E8%BC%83/
+
+- Fluentdの集約サーバ用設定ファイル (fluent-plugin-rewrite-tag-filter版)  
+https://gist.github.com/matsumana/4078096
+
+- 稼働中のFluentdにflowcounter pluginを導入してみた  
+http://dayafterneet.blogspot.jp/2012/12/fluentdflowcounter-plugin.html
+
+## TODO
+
+Pull requests are very welcome!!
+
+## Copyright
+
+Copyright :  Copyright (c) 2012- Kentaro Yoshida (@yoshi_ken)  
+License   :  Apache License, Version 2.0
+

data/fluent-plugin-rewrite-tag-filter.gemspec

@@ -3,18 +3,17 @@ $:.push File.expand_path("../lib", __FILE__)
 
 Gem::Specification.new do |s|
   s.name        = "fluent-plugin-rewrite-tag-filter"
-  s.version     = "1.1.1"
+  s.version     = "1.2.0"
   s.authors     = ["Kentaro Yoshida"]
   s.email       = ["y.ken.studio@gmail.com"]
   s.homepage    = "https://github.com/y-ken/fluent-plugin-rewrite-tag-filter"
-  s.summary     = %q{Fluentd Output filter plugin. It has designed to rewrite tag like mod_rewrite. Re-emmit a record with rewrited tag when a value matches with the regular expression. Also you can change a tag with a values from apache log by domain, status-code(ex. 500 error), user-agent, request-uri, regex-backreference and so on with regular expression.}
+  s.summary     = %q{Fluentd Output filter plugin. It has designed to rewrite tag like mod_rewrite. Re-emmit a record with rewrited tag when a value matches with the regular expression. Also you can change a tag from apache log by domain, status-code(ex. 500 error), user-agent, request-uri, regex-backreference and so on with regular expression.}
 
   s.files         = `git ls-files`.split("\n")
   s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
   s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
   s.require_paths = ["lib"]
 
-  # specify any dependencies here; for example:
-  s.add_development_dependency "fluentd"
+  s.add_development_dependency "rake"
   s.add_runtime_dependency "fluentd"
 end

data/lib/fluent/plugin/out_rewrite_tag_filter.rb

@@ -8,6 +8,7 @@ class Fluent::RewriteTagFilterOutput < Fluent::Output
     config_param ('rewriterule' + i.to_s).to_sym, :string, :default => nil # NAME REGEXP
   end
   config_param :capitalize_regex_backreference, :bool, :default => false
+  config_param :remove_tag_prefix, :string, :default => nil
 
   def configure(conf)
     super
@@ -37,6 +38,10 @@ class Fluent::RewriteTagFilterOutput < Fluent::Output
     unless @rewriterules.length == rewriterule_names.uniq.length
       raise Fluent::ConfigError, "duplicated rewriterules found #{@rewriterules.inspect}"
     end
+
+    unless conf['remove_tag_prefix'].nil?
+      @remove_tag_prefix = Regexp.new("^#{Regexp.escape(remove_tag_prefix)}\.?")
+    end
   end
 
   def emit(tag, es, chain)
@@ -68,17 +73,15 @@ class Fluent::RewriteTagFilterOutput < Fluent::Output
   end
 
   def map_regex_table(elements)
-    # TODO: use each.with_index http://shirusu-ni-tarazu.hatenablog.jp/entry/2012/11/04/173513
     hash_table = Hash.new
-    index = 1
-    elements.each do |value|
+    elements.each.with_index(1) do |value, index|
       hash_table["$#{index}"] = @capitalize_regex_backreference ? value.capitalize : value
-      index += 1
     end
     return hash_table
   end
 
   def get_placeholder(tag)
+    tag = tag.sub(@remove_tag_prefix, '') if @remove_tag_prefix
     return {
       '__HOSTNAME__' => @hostname,
       '${hostname}' => @hostname,

data/test/plugin/test_out_rewrite_tag_filter.rb

@@ -24,6 +24,12 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
     rewriterule4 agent  "(Googlebot|CustomBot)-([a-zA-Z]+)" agent.$1-$2
   ]
 
+  # remove_tag_prefix test
+  CONFIG3 = %[
+    rewriterule1 domain ^www\.google\.com$ ${tag}
+    remove_tag_prefix input
+  ]
+
   def create_driver(conf=CONFIG,tag='test')
     Fluent::Test::OutputTestDriver.new(Fluent::RewriteTagFilterOutput, tag).configure(conf)
   end
@@ -43,7 +49,6 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
 
   def test_emit
     d1 = create_driver(CONFIG, 'input.access')
-    time = Time.parse("2012-01-02 13:14:15").to_i
     d1.run do
       d1.emit({'domain' => 'www.google.com', 'path' => '/foo/bar?key=value', 'agent' => 'Googlebot', 'response_time' => 1000000})
       d1.emit({'domain' => 'news.google.com', 'path' => '/', 'agent' => 'Googlebot-Mobile', 'response_time' => 900000})
@@ -68,7 +73,6 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
 
   def test_emit2
     d1 = create_driver(CONFIG2, 'input.access')
-    time = Time.parse("2012-01-02 13:14:15").to_i
     d1.run do
       d1.emit({'domain' => 'www.google.com', 'path' => '/foo/bar?key=value', 'agent' => 'Googlebot', 'response_time' => 1000000})
       d1.emit({'domain' => 'news.google.com', 'path' => '/', 'agent' => 'Googlebot-Mobile', 'response_time' => 900000})
@@ -87,5 +91,16 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
     p emits[3]
     assert_equal 'agent.Googlebot-Foobar', emits[3][0] #tag
   end
+
+  def test_emit3
+    d1 = create_driver(CONFIG3, 'input.access')
+    d1.run do
+      d1.emit({'domain' => 'www.google.com', 'path' => '/foo/bar?key=value', 'agent' => 'Googlebot', 'response_time' => 1000000})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    assert_equal 'access', emits[0][0] # tag
+  end
 end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-rewrite-tag-filter
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.2.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,10 +9,10 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-02-13 00:00:00.000000000 Z
+date: 2013-05-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: fluentd
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
@@ -53,7 +53,7 @@ files:
 - .gitignore
 - Gemfile
 - LICENSE.txt
-- README.rdoc
+- README.md
 - Rakefile
 - example.conf
 - example2.conf
@@ -86,9 +86,8 @@ signing_key:
 specification_version: 3
 summary: Fluentd Output filter plugin. It has designed to rewrite tag like mod_rewrite.
   Re-emmit a record with rewrited tag when a value matches with the regular expression.
-  Also you can change a tag with a values from apache log by domain, status-code(ex.
-  500 error), user-agent, request-uri, regex-backreference and so on with regular
-  expression.
+  Also you can change a tag from apache log by domain, status-code(ex. 500 error),
+  user-agent, request-uri, regex-backreference and so on with regular expression.
 test_files:
 - test/helper.rb
 - test/plugin/test_out_rewrite_tag_filter.rb

data/README.rdoc

@@ -1,140 +0,0 @@
-= fluent-plugin-rewrite-tag-filter
-
-== Overview
-
-=== RewriteTagFilterOutput
-
-It has designed to rewrite tags like mod_rewrite.
-Also you can change a tag from apache log by domain, status-code(ex. 500 error), 
-user-agent, request-uri, regex-backreference and so on with regular expression. 
-
-== Installation
-
-    # native gem
-    gem install fluent-plugin-rewrite-tag-filter
-    
-    # td-agent gem
-    /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-rewrite-tag-filter
-
-== Configuration
-
-=== Syntax
-
-    rewruterule<num:1-200> <attribute> <regex_pattern> <new_tag>
-    
-    # Optional: Capitalize every matched regex backreference. (ex: $1, $2)
-    capitalize_regex_backreference <yes/no> (default no)
-
-=== Usage
-
-It's a sample to exclude some static file log before split tag by domain.
-
-    <source>
-      type tail
-      path /var/log/httpd/access_log
-      format /^(?<domain>[^ ]*) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/
-      time_format %d/%b/%Y:%H:%M:%S %z
-      tag td.apache.access
-      pos_file /var/log/td-agent/apache_access.pos
-    </source>
-
-    # At rewriterule4, "site.$2$1" to be "site.ExampleMail" by capitalize_regex_backreference option.
-    <match td.apache.access>
-      type rewrite_tag_filter
-      capitalize_regex_backreference yes
-      rewriterule1 path   \.(gif|jpe?g|png|pdf|zip)$  clear
-      rewriterule2 domain ^maps\.example\.com$        site.ExampleMaps
-      rewriterule3 domain ^news\.example\.com$        site.ExampleNews
-      rewriterule4 domain ^(mail)\.(example)\.com$    site.$2$1
-      rewriterule5 domain .+                          site.unmatched
-    </match>
-
-    <match site.*>
-      type mongo
-      host localhost
-      database apache_access
-      remove_tag_prefix site
-      tag_mapped
-      capped
-      capped_size 100m
-    </match>
-
-    <match clear>
-      type null
-    </match>
-
-=== Result
-
-    $ mongo
-    MongoDB shell version: 2.2.0
-    > use apache_access
-    switched to db apache_access
-    > show collections
-    ExampleMaps
-    ExampleNews
-    ExampleMail
-    unmatched
-
-=== Debug
-
-On starting td-agent, Logging supported like below.
-
-    $ tailf /var/log/td-agent/td-agent.log
-    2012-09-16 18:10:51 +0900: adding match pattern="td.apache.access" type="rewrite_tag_filter"
-    2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [1, "path", /\.(gif|jpe?g|png|pdf|zip)$/, "clear"]
-    2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [2, "domain", /^maps\.example\.com$/, "site.ExampleMaps"]
-    2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [3, "domain", /^news\.example\.com$/, "site.ExampleNews"]
-    2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [4, "domain", /^(mail)\.(example)\.com$/, "site.$2$1"]
-    2012-09-16 18:10:51 +0900: adding rewrite_tag_filter rule: [5, "domain", /.+/, "site.unmatched"]
-
-=== Tag placeholder
-
-Supporing these placeholder for new_tag(rewrited tag).
-
-- ${hostname}
-- __HOSTNAME__
-- ${tag}
-- __TAG__
-
-It's a sample to rewrite a tag with placeholder.
-
-    # It will get "rewrited.apache.access.ExampleMail"
-    <match apache.access>
-      type rewrite_tag_filter
-      rewriterule1 domain ^(mail)\.(example)\.com$    rewrited.${tag}.$2$1
-    </match>
-
-    # It will get "rewrited.ExampleMail.app30-124.foo.com" when hostname is "app30-124.foo.com"
-    <match apache.access>
-      type rewrite_tag_filter
-      rewriterule1 domain ^(mail)\.(example)\.com$    rewrited.$2$1.${hostname}
-    </match>
-
-== Example
-
-- Example1: how to analyze response_time, response_code and user_agent for each virtual domain websites.
-  - https://github.com/y-ken/fluent-plugin-rewrite-tag-filter/blob/master/example.conf
-
-- Example2: how to exclude specified patterns before analyze response_time for each virtual domain websites.
-  - https://github.com/y-ken/fluent-plugin-rewrite-tag-filter/blob/master/example2.conf
-
-== Topic
-
-- 自在にタグを書き換える fluent-plugin-rewrite-tag-filter でログ解析が捗るお話 #fluentd
-  - http://d.hatena.ne.jp/yoshi-ken/20120701/1341137269
-
-- Fluentd & TreasureDataで こっそり始めるログ集計 Fluentd Meetup #2 @mikeda
-  - http://www.slideshare.net/baguzy/fluentd-meetup-2-14073930
-
-- 似てる #fluentd プラグインの比較
-  - http://matsumana.wordpress.com/2012/11/15/%E4%BC%BC%E3%81%A6%E3%82%8B-fluentd-%E3%83%97%E3%83%A9%E3%82%B0%E3%82%A4%E3%83%B3%E3%81%AE%E6%AF%94%E8%BC%83/
-
-== TODO
-
-- patches welcome!
-
-== Copyright
-
-Copyright:: Copyright (c) 2012- Kentaro Yoshida (@yoshi_ken)
-License::   Apache License, Version 2.0
-