fluent-mixin-rewrite-tag-name 0.0.1 → 0.0.2

data/README.md

@@ -2,14 +2,192 @@
 
 ## Overview
 
-Fluentd mixin plugin to rewrite tag like [rewrite-tag-filter](https://github.com/fluent/fluent-plugin-rewrite-tag-filter) for your any plugins.
+Fluentd mixin plugin to provides placeholder function for rewriting tag for your any plugins as like [fluent-plugin-rewrite-tag-filter](https://github.com/fluent/fluent-plugin-rewrite-tag-filter). It will let you get easy to implement tag placeholder for your own plugins.
+
+## Placeholders
+
+It supportes these placeholder for rewriting tag.
+
+- `${tag}`
+- `__TAG__`
+- `{$tag_parts[n]}`
+- `__TAG_PARTS[n]__`
+- `${hostname}`
+- `__HOSTNAME__`
+
+The placeholder of `{$tag_parts[n]}` and `__TAG_PARTS[n]__` acts accessing the index which split the tag with "." (dot).  
+For example with `td.apache.access` tag, it will get `td` by `${tag_parts[0]}` and `apache` by `${tag_parts[1]}`.
+
+**Note** 
+
+* To support upcase placeholder, set `enable_placeholder_upcase true` in configuration.
+* To support hostname placeholder, set `enable_placeholder_upcase true` in configuration.
+* Currently, range expression ```${tag_parts[0..2]}``` is not supported.
+
+#### Placeholder Option
+
+* `hostname_command` 
+
+By default, execute command as `hostname` to get full hostname.  
+On your needs, it could override hostname command using `hostname_command` option.  
+It comes short hostname with `hostname_command hostname -s` configuration specified.
+
+## Configuration
+
+Adding this mixin plugin, it will enabled to use these placeholder in your plugins.
+
+```xml
+# input plugin example
+<source>
+  type              foo_bar
+
+  # it will be rewrited to be 'customprefix.web10-222' when short hostname is 'web10-222'.
+  tag               customprefix.${hostname}
+  
+  # to use hostname placeholder, set 'enable_placeholder_hostname true'
+  enable_placeholder_hostname  true
+  hostname_command             hostname -s
+</source>
+```
+
+```xml
+# output plugin example
+<match test.foo>
+  type  foo_bar
+  
+  # it will be rewrited to be 'customprefix.test.foo'.
+  tag   customprefix.${tag}
+</match>
+```
+
+Another examples are written in [unit test](https://github.com/y-ken/fluent-mixin-rewrite-tag-name/blob/master/test/mixin/test_rewrite_tag_name.rb).
 
 ## Usage
 
-TODO
+#### 1. edit gemspec
+
+add dependency for .gemspec file like below. For more detail, see [gemspec example](https://github.com/y-ken/fluent-plugin-anonymizer/blob/master/fluent-plugin-anonymizer.gemspec)
+
+```ruby
+spec.add_runtime_dependency "fluent-mixin-rewrite-tag-name"
+```
+
+#### 2. activate fluent-mixin-rewrite-tag-name for your plugin
+
+It is the instruction in the case of adding `fluent-plugin-foobar`.
+
+```
+$ cd fluent-plugin-foobar
+$ vim fluent-plugin-foobar.gemspec # edit gemspec
+$ bundle install --path vendor/bundle # or just type `bundle install`
+```
+
+#### 3. edit your plugin to implement
+
+It is a quick guide to enable your plugin to use RewriteTagNameMixin.  
+The key points of implmentation is just four below.
+
+* add `require 'fluent/mixin/rewrite_tag_name'` at the top of source
+* in the case of output plugin, add `include Fluent::HandleTagNameMixin`  
+this is required if you will use 'remove_tag_prefix' option together
+* add `include Fluent::Mixin::RewriteTagName` in class after HandleTagNameMixin
+* add `emit_tag = tag.dup` and `filter_record(emit_tag, time, record)` before `Engine.emit`
+
+##### implement example for input plugin
+
+```ruby
+require 'fluent/mixin/rewrite_tag_name'
+
+module Fluent
+  class FooBarInput < Fluent::Input
+    Plugin.register_input('foo_bar', self)
+
+    # ...snip...
+
+    include Fluent::HandleTagNameMixin
+    include Fluent::Mixin::RewriteTagName    
+    config_param :hostname_command, :string, :default => 'hostname'
+
+    # ...snip...
+
+    def configure(conf)
+      super
+
+      # ...snip...
+
+      # add a error handling 
+      if ( !@tag && !@remove_tag_prefix && !@remove_tag_suffix && !@add_tag_prefix && !@add_tag_suffix )
+        raise Fluent::ConfigError, "foo_bar: missing remove_tag_prefix, remove_tag_suffix, add_tag_prefix or add_tag_suffix."
+      end      
+    end
+
+    # ...snip...
+
+    def emit_message(tag, message)
+      emit_tag = tag.dup
+      filter_record(emit_tag, time, message)
+      Engine.emit(emit_tag, Engine.now, message)
+    end
+
+    # ...snip...
+
+  end
+end
+```
+
+##### implement example for output plugin
+
+```ruby
+require 'fluent/mixin/rewrite_tag_name'
+
+class Fluent
+  class FooBarOutput < Fluent::Output
+    Fluent::Plugin.register_output('foo_bar', self)
+
+    include Fluent::Mixin::RewriteTagName
+    config_param :hostname_command, :string, :default => 'hostname'
+
+    # ...snip...
+
+    def configure(conf)
+      super
+
+      # ...snip...
+
+      # add a error handling 
+      if ( !@tag && !@remove_tag_prefix && !@remove_tag_suffix && !@add_tag_prefix && !@add_tag_suffix )
+        raise Fluent::ConfigError, "foo_bar: missing remove_tag_prefix, remove_tag_suffix, add_tag_prefix or add_tag_suffix."
+      end      
+    end
+
+    # ...snip...
+
+    def emit(tag, es, chain)
+      es.each do |time, record|
+        emit_tag = tag.dup
+        filter_record(emit_tag, time, record)
+        Fluent::Engine.emit(emit_tag, time, record)
+      end
+      chain.next
+    end
+
+    # ...snip...
+
+  end
+end
+```
+
+## Case Study
+
+These cool plugins are using this mixin!
+
+* [fluent-plugin-anonymizer](https://github.com/y-ken/fluent-plugin-anonymizer/)
 
 ## TODO
 
+* switchable tag template variable like 'tag', 'tag_format'
+* support range tag_parts like [fluent-plugin-forest](https://github.com/tagomoris/fluent-plugin-forest/compare/v0.2.2...master)
+* support tag_prefix and tag_suffix placeholder like [fluent-plugin-record-reformer](https://github.com/sonots/fluent-plugin-record-reformer)
 * merge into [fluentd/lib/fluent/mixin.rb](https://github.com/fluent/fluentd/blob/master/lib/fluent/mixin.rb) as RewriteTagNameMixin module.
 
 Pull requests are very welcome!!

data/fluent-mixin-rewrite-tag-name.gemspec

@@ -4,10 +4,10 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "fluent-mixin-rewrite-tag-name"
-  spec.version       = "0.0.1"
+  spec.version       = "0.0.2"
   spec.authors       = ["Kentaro Yoshida"]
   spec.email         = ["y.ken.studio@gmail.com"]
-  spec.summary       = %q{Fluentd mixin plugin to rewrite tag like rewrite-tag-filter for your any plugins.}
+  spec.summary       = %q{Fluentd mixin plugin to provides placeholder function for rewriting tag for your any plugins as like fluent-plugin-rewrite-tag-filter. It will let you get easy to implement tag placeholder for your own plugins.}
   spec.homepage      = "https://github.com/y-ken/fluent-mixin-rewrite-tag-name"
   spec.license       = "Apache License, Version 2.0"
 

data/lib/fluent/mixin/rewrite_tag_name.rb

@@ -2,7 +2,33 @@ module Fluent
   module Mixin
     module RewriteTagName
       include RecordFilterMixin
-      attr_accessor :tag
+      attr_accessor :tag, :hostname_command
+      attr_accessor :enable_placeholder_upcase, :enable_placeholder_hostname
+
+      DEFAULT_HOSTNAME_COMMAND = 'hostname'
+
+      def configure(conf)
+        super
+
+        @placeholder_expander = PlaceholderExpander.new
+
+        if enable_upcase = conf['enable_placeholder_upcase']
+          @enable_placeholder_upcase = enable_upcase
+        end
+        if @enable_placeholder_upcase
+          @placeholder_expander.enable_placeholder_upcase
+        end
+
+        if enable_hostname = conf['enable_placeholder_hostname']
+          @enable_placeholder_hostname = enable_hostname
+        end
+        if @enable_placeholder_hostname
+          hostname_command = @hostname_command || DEFAULT_HOSTNAME_COMMAND
+          hostname = `#{hostname_command}`.chomp
+          @placeholder_expander.enable_placeholder_hostname
+          @placeholder_expander.set_hostname(hostname)
+        end
+      end
 
       def filter_record(tag, time, record)
         super
@@ -12,16 +38,64 @@ module Fluent
       end
 
       def rewrite_tag!(tag)
-        placeholder = {
-          '${tag}' => tag,
-          '__TAG__' => tag
-        }
-        emit_tag = @tag.gsub(/(\${[a-z_]+(\[[0-9]+\])?}|__[A-Z_]+__)/) do
-          $log.warn "RewriteTagNameMixin: unknown placeholder found. :placeholder=>#{$1} :tag=>#{tag} :rewritetag=>#{rewritetag}" unless placeholder.include?($1)
-          placeholder[$1]
-        end
+
+        @placeholder_expander.set_tag(tag)
+        emit_tag = @placeholder_expander.expand(@tag)
         tag.gsub!(tag, emit_tag)
       end
+
+      class PlaceholderExpander
+        # referenced https://github.com/fluent/fluent-plugin-rewrite-tag-filter, thanks!
+        # referenced https://github.com/sonots/fluent-plugin-record-reformer, thanks!
+        attr_reader :placeholders
+
+        def initialize
+          @placeholders = {}
+          @enable_options = {
+            :hostname => false,
+            :upcase => false,
+          }
+        end
+
+        def expand(str)
+          str.gsub(/(\${[a-z_]+(\[-?[0-9]+\])?}|__[A-Z_]+(\[-?[0-9]+\])?__)/) {
+            $log.warn "RewriteTagNameMixin: unknown placeholder `#{$1}` found" unless @placeholders.include?($1)
+            @placeholders[$1]
+          }
+        end
+
+        def enable_placeholder_hostname
+          @enable_options[:hostname] = true
+        end
+
+        def enable_placeholder_upcase
+          @enable_options[:upcase] = true
+        end
+
+        def set_tag(value)
+          set_placeholder('tag', value)
+          set_tag_parts(value)
+        end
+
+        def set_hostname(value)
+          set_placeholder('hostname', value)
+        end
+
+        def set_tag_parts(tag)
+          tag_parts = tag.split('.') 
+          size = tag_parts.size
+          tag_parts.each_with_index { |t, idx|
+            set_placeholder("tag_parts[#{idx}]", t)
+            set_placeholder("tag_parts[#{idx-size}]", t) # support tag_parts[-1]
+          }
+        end
+
+        private
+        def set_placeholder(key, value)
+          @placeholders.store("${#{key.downcase}}", value)
+          @placeholders.store("__#{key.upcase}__", value) if @enable_options[:upcase]
+        end
+      end
     end
   end
 end

data/test/mixin/test_rewrite_tag_name.rb

@@ -37,6 +37,22 @@ class RewriteTagNameMixinTest < Test::Unit::TestCase
     assert_equal 'foo', emits[0][2]['message']
   end
 
+  def test_emit_upcase
+    d1 = create_driver(%[
+      tag                rewrited.__TAG__
+      remove_tag_prefix  input.
+      enable_placeholder_upcase true
+    ], 'input.access')
+    d1.run do
+      d1.emit({'message' => 'foo'})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    assert_equal 'rewrited.access', emits[0][0] # tag
+    assert_equal 'foo', emits[0][2]['message']
+  end
+
   def test_emit_with_HandleTagNameMixin
     d1 = create_driver(%[
       tag                rewrited.${tag}
@@ -51,4 +67,77 @@ class RewriteTagNameMixinTest < Test::Unit::TestCase
     assert_equal 'rewrited.access', emits[0][0] # tag
     assert_equal 'foo', emits[0][2]['message']
   end
+
+  def test_emit_tag_parts
+    d1 = create_driver(%[
+      tag                rewrited.${tag_parts[1]}
+    ], 'input.access.foo.bar')
+    d1.run do
+      d1.emit({'message' => 'foo'})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    assert_equal 'rewrited.access', emits[0][0] # tag
+  end
+
+  def test_emit_tag_parts_negative
+    d1 = create_driver(%[
+      tag                rewrited.${tag_parts[-1]}
+    ], 'input.access.foo.bar')
+    d1.run do
+      d1.emit({'message' => 'foo'})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    assert_equal 'rewrited.bar', emits[0][0] # tag
+    assert_equal 'foo', emits[0][2]['message']
+  end
+
+  def test_emit_tag_parts_negative2
+    d1 = create_driver(%[
+      tag                rewrited.${tag_parts[-2]}
+    ], 'input.access.foo.bar')
+    d1.run do
+      d1.emit({'message' => 'foo'})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    assert_equal 'rewrited.foo', emits[0][0] # tag
+    assert_equal 'foo', emits[0][2]['message']
+  end
+
+  def test_emit_hostname
+    d1 = create_driver(%[
+      tag                rewrited.${hostname}
+    ], 'input.access')
+    d1.run do
+      d1.emit({'message' => 'foo'})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    hostname = `hostname`.chomp
+    assert_equal "rewrited.#{hostname}", emits[0][0] # tag
+    assert_equal 'foo', emits[0][2]['message']
+  end
+
+  def test_emit_hostname_short
+    d1 = create_driver(%[
+      tag                rewrited.${hostname}
+      hostname_command   hostname -s
+    ], 'input.access')
+    d1.run do
+      d1.emit({'message' => 'foo'})
+    end
+    emits = d1.emits
+    assert_equal 1, emits.length
+    p emits[0]
+    hostname_command = d1.instance.config['hostname_command']
+    hostname = `#{hostname_command}`.chomp
+    assert_equal "rewrited.#{hostname}", emits[0][0] # tag
+    assert_equal 'foo', emits[0][2]['message']
+  end
 end

data/test/plugin.rb

@@ -2,17 +2,19 @@ class Fluent::RewriteTagNameMixinOutput < Fluent::Output
   Fluent::Plugin.register_output('rewrite_tag_name_mixin', self)
 
   config_param :tag, :string, :default => nil
+  config_param :hostname_command, :string, :default => 'hostname'
 
   include Fluent::HandleTagNameMixin
   include Fluent::Mixin::RewriteTagName
+  config_set_default :enable_placeholder_upcase, false
+  config_set_default :enable_placeholder_hostname, true
 
   def configure(conf)
     super
-    
-    if @tag.nil?
-      raise Fluent::ConfigError, "'tag' parameter is required."
-    end
 
+    if ( !@tag && !@remove_tag_prefix && !@remove_tag_suffix && !@add_tag_prefix && !@add_tag_suffix )
+      raise Fluent::ConfigError, "RewriteTagNameMixin: missing remove_tag_prefix, remove_tag_suffix, add_tag_prefix or add_tag_suffix."
+    end
   end
 
   def emit(tag, es, chain)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-mixin-rewrite-tag-name
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-01-20 00:00:00.000000000 Z
+date: 2014-01-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -101,8 +101,9 @@ rubyforge_project:
 rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
-summary: Fluentd mixin plugin to rewrite tag like rewrite-tag-filter for your any
-  plugins.
+summary: Fluentd mixin plugin to provides placeholder function for rewriting tag for
+  your any plugins as like fluent-plugin-rewrite-tag-filter. It will let you get easy
+  to implement tag placeholder for your own plugins.
 test_files:
 - test/helper.rb
 - test/mixin/test_rewrite_tag_name.rb