fluent-plugin-burrow 1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 39340b1b558d45c2bc90b371d73d33dd0fa4d26c
+  data.tar.gz: ac0f5ce4759eba67450f13bb07a023f304becd69
+SHA512:
+  metadata.gz: d547faabd129b91df6a46f5f9145933fd40f9c4cebdb460094992057368814ba3e5354e7c3a2132272bb043bad9ae7d5e61c1a128eff20dcd9ca7a70b346d5b7
+  data.tar.gz: cb4bdfff79da208322ffd2890ddb04c217ae2ddbd8821ad76959eb44d2ad71473834bc3acec18aaef088b11c1069add50ad7321ce59708e2d784060205042bf3

data/.gitignore

@@ -0,0 +1,5 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*
+vendor/*

data/.travis.yml

@@ -0,0 +1,6 @@
+language: ruby
+
+rvm:
+  - 2.1
+  - 2.0.0
+  - 1.9.3

data/Gemfile

@@ -0,0 +1,4 @@
+source "http://rubygems.org"
+
+# Specify your gem's dependencies in fluent-plugin-rewrite-tag-filter.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2014 Vanilla Forums
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,201 @@
+fluent-plugin-burrow
+====================
+
+This plugin for [Fluentd](http://fluentd.org) allows to extract a single key from an existing record and re-parse it with
+a supplied format. A new event is then emitted, with the record modified by the now-decoded key's value.
+
+## Motivation
+
+**out_burrow is designed to allow post-facto re-parsing of nested key elements.**
+
+For example, lets say your source application writes to syslog, but instead of plain string messages, it writes JSON
+encoded data. /var/log/syslog contains the following entry:
+
+    Jun 17 21:16:22 app1 5012162: {"event":"csrf_failure","msg":"Invalid transient key for System.","username":"System","userid":"1","ip":"192.34.93.74","method":"GET","domain":"http://timgunter.ca","path":"/dashboard/settings","tags":["csrf","failure"],"accountid":5009392,"siteid":5012162}
+
+In td-agent.conf, you might have something like this to read this event:
+
+```
+<source>
+    type syslog
+    port 5140
+    bind 127.0.0.1
+    tag raw.app.vanilla.events
+</source>
+```
+
+Unfortunately, in_syslog does not understand that the `message` field is encoded with JSON, so it escapes all the data
+and makes it unusable down the line. If we piped these events to a file, we would see something like this:
+
+```
+2014-06-17T21:16:22Z	raw.app.vanilla.events.local0.err	{"host":"app1","ident":"5012162","message":"{\"event\":\"csrf_failure\",\"msg\":\"Invalid transient key for System.\",\"username\":\"System\",\"userid\":\"1\",\"ip\":\"192.34.93.74\",\"method\":\"GET\",\"domain\":\"http://timgunter.ca\",\"path\":\"/dashboard/authentication\",\"tags\":[\"csrf\",\"failure\"],\"accountid\":5009392,\"siteid\":5012162}"}
+```
+
+Note how the `message` field has been escaped. This means that when this event eventually makes its way to a file, or
+another system (like elasticsearch for example), it will not be ready for consumption. That's where `out_burrow` comes in.
+
+Adding the following `match` block to td-agent.conf allows us to intercept the raw syslog events and re-parse the
+message field as JSON:
+
+```
+<match raw.app.vanilla.events.**>
+    type burrow
+    key_name message
+    action inplace
+    remove_prefix raw
+    format json
+</match>
+```
+
+There are several components to this rule, but for now lets look at the output:
+
+```
+2014-06-17T21:16:23Z	app.vanilla.events.local0.err	{"host":"app1","ident":"5012162","message":{"event":"csrf_failure","msg":"Invalid transient key for System.","username":"System","userid":"1","ip":"192.34.93.74","method":"GET","domain":"http://timgunter.ca","path":"/dashboard/settings/mobilethemes","tags":["csrf","failure"],"accountid":5009392,"siteid":5012162}}
+```
+
+Now the JSON is no longer escaped, and can be easily parsed by both fluentd and elasticsearch.
+
+## Settings
+
+### key_name
+
+`required`
+
+This is the name of the key we want to examine and re-parse, and is required.
+
+### format
+
+`required`
+
+This is format that Fluentd should expect the `key_name` field to be encoded with. out_burrow supports the same built-in
+format as Fluent::TextParser (and in_tail):
+
+- apache
+- apache2
+- nginx
+- syslog
+- json
+- csv
+- tsv
+- ltsv
+
+### tag
+
+`optional`
+
+When this event is re-emitted, change its tag to this setting's value.
+
+### remove_prefix
+
+`optional`
+
+When this event is re-emitted, remove this prefix from the source tag and use the resulting string as the new event's
+tag. This setting automatically adds a trailing period `.` to its value before stripping.
+
+### add_prefix
+
+`optional`
+
+When this event is re-emitted, prepend this prefix to the source tag and use the resulting string as the new event's tag.
+This setting automatically adds a trailing period `.` to its value before prepending.
+
+**One of the `tag`, `remove_prefix`, or `add_prefix` settings is required.**
+**`remove_prefix` and `add_prefix` can co-exist together.**
+
+### action
+
+`optional` and defaults to `inplace`
+
+The value of this setting determines how the new event will be constructed. There are three distinct options here:
+
+- inplace
+
+Perform decoding 'in place'. When the `key_name` field is successfully parsed, its contents will be written back to its
+original key in the original record, which will then be re-emitted.
+
+- overlay
+
+Overlay decoded data on top of original record, and re-emit. In our example above, if 'overlay' was used instead of
+'inplace', the resulting record would have been:
+
+```
+{
+    "host":"app1",
+    "ident":"5012162",
+    "event":"csrf_failure",
+    "msg":"Invalid transient key for System.",
+    "username":"System",
+    "userid":"1",
+    "ip":"192.34.93.74",
+    "method":"GET",
+    "domain":"http://timgunter.ca",
+    "path":"/dashboard/settings",
+    "tags":["csrf","failure"],
+    "accountid":5009392,
+    "siteid":5012162
+}
+```
+
+- replace
+
+Replace the original entirely with the contents of the decoded field. In our example above, if 'replace' was used
+instead of 'inplace', the resulting record would have been:
+
+```
+{
+    "event":"csrf_failure",
+    "msg":"Invalid transient key for System.",
+    "username":"System",
+    "userid":"1",
+    "ip":"192.34.93.74",
+    "method":"GET",
+    "domain":"http://timgunter.ca",
+    "path":"/dashboard/settings",
+    "tags":["csrf","failure"],
+    "accountid":5009392,
+    "siteid":5012162
+}
+```
+
+### keep_key
+
+`optional` and defaults to `false`
+
+Keep original source key (only valid with 'overlay' and 'replace' actions). When this is `true`, the original encoded
+source key is retained in the output.
+
+### keep_time
+
+`optional` and defaults to `false`
+
+Keep the original record's "time" key. If the original top level record contains a
+
+### record_time_key
+
+`optional` and defaults to `time`
+
+If `keep_time` is `true`, this field specifies the key that contains the original records's time. The value of this key
+will be copied into the new record after it has been parsed.
+
+### time_key
+
+`optional` and defaults to `time`
+
+When the `key_name` field's value is being parsed, look for this key and interpret it as the record's `time` key.
+
+### time_format
+
+`optional` and defaults to nil
+
+When parsing the `key_name` field's value and if `time_key` is set, this field denotes the format to expect the `time`
+to be in.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+If you have a question, [open an Issue](https://github.com/vanilla/fluent-plugin-burrow/issues).

data/Rakefile

@@ -0,0 +1,9 @@
+require "bundler/gem_tasks"
+require "rake/testtask"
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test

data/examples/syslog.json.txt

@@ -0,0 +1,5 @@
+#
+# This example demonstrates a method of extracting the "message" portion of an encoded syslog event and bringing the
+# data in that portion into the main record scope.
+# 
+#

data/fluent-plugin-burrow.gemspec

@@ -0,0 +1,21 @@
+# -*- encoding: utf-8 -*-
+$:.push File.expand_path("../lib", __FILE__)
+
+Gem::Specification.new do |s|
+  s.name        = "fluent-plugin-burrow"
+  s.description = "Extract a single key (in formats Fluent can natively understand) from an event and re-emit a new event that replaces the entire original record with that key's values."
+  s.version     = "1.0"
+  s.license     = "MIT"
+  s.authors     = ["Tim Gunter"]
+  s.email       = ["tim@vanillaforums.com"]
+  s.homepage    = "https://github.com/vanilla/fluent-plugin-burrow"
+  s.summary     = %q{Fluentd output filter plugin. Extract a single key (in formats Fluent can natively understand) from an event and re-emit a new event that replaces the entire original record with that key's values.}
+
+  s.files         = `git ls-files`.split("\n")
+  s.test_files    = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths = ["lib"]
+
+  s.add_development_dependency "rake"
+  s.add_runtime_dependency "fluentd"
+end

data/lib/fluent/plugin/out_burrow.rb

@@ -0,0 +1,143 @@
+# Burrow Output Plugin
+# @author Tim Gunter <tim@vanillaforums.com>
+#
+# This plugin allows to extract a single key from an existing event and re-parse it with a given
+# format, and then re-emit a new event with the key's value replaced, or with the whole record replaced.
+#
+
+class Fluent::BurrowPlugin < Fluent::Output
+  # Register type
+  Fluent::Plugin.register_output('burrow', self)
+
+  # Required
+  config_param :key_name, :string
+  config_param :format, :string
+
+  # Optional - tag format
+  config_param :tag, :string, :default => nil                 # Create a new tag for the re-emitted event
+  config_param :remove_prefix, :string, :default => nil       # Remove a prefix from the existing tag
+  config_param :add_prefix, :string, :default => nil          # Add a prefix to the existing tag
+
+  # Optional - record format
+  config_param :action, :string, :default => 'inplace'        # The action to take once key parsing is complete
+  config_param :keep_key, :bool, :default => false            # Keep original source key (only valid with 'overlay' and 'replace' actions)
+
+  # Optional - time format
+  config_param :keep_time, :bool, :default => false           # Keep the original event's "time" key
+  config_param :record_time_key, :string, :default => 'time'  # Allow a custom time field in the record
+  config_param :time_key, :string, :default => 'time'         # Allow a custom time field in the sub-record
+  config_param :time_format, :string, :default => nil         # Allow a custom time format for the new record
+
+  # Parse config hash
+  def configure(conf)
+    super
+
+    # One of 'tag', 'remove_prefix' or 'add_prefix' must be specified
+    if not @tag and not @remove_prefix and not @add_prefix
+      raise Fluent::ConfigError, "One of 'tag', 'remove_prefix' or 'add_prefix' must be specified"
+    end
+    if @tag and (@remove_prefix or @add_prefix)
+      raise Fluent::ConfigError, "Specifying both 'tag' and either 'remove_prefix' or 'add_prefix' is not supported"
+    end
+
+    # Prepare for tag modification if required
+    if @remove_prefix
+      @removed_prefix_string = @remove_prefix.chomp('.') + '.'
+      @removed_length = @removed_prefix_string.length
+    end
+    if @add_prefix
+      @added_prefix_string = @add_prefix.chomp('.') + '.'
+    end
+
+    # Validate action
+    actions = ['replace','overlay','inplace']
+    if not actions.include? @action
+      raise Fluent::ConfigError, "Invalid 'action', must be one of #{actions.join(',')}"
+    end
+
+    # Validate action-based restrictions
+    if @action == 'inplace' and @keep_key
+      raise Fluent::ConfigError, "Specifying 'keep_key' with action 'inplace' is not supported"
+    end
+
+    # Prepare fluent's built-in parser
+    @parser = Fluent::TextParser.new()
+    @parser.configure(conf)
+  end
+
+  # This method is called when starting.
+  def start
+    super
+  end
+
+  # This method is called when shutting down.
+  def shutdown
+  end
+
+  # This method is called when an event reaches Fluentd.
+  def emit(tag, es, chain)
+
+    # Figure out new event tag (either manually specified, or modified with add_prefix|remove_prefix)
+    if @tag
+      tag = @tag
+    else
+      if @remove_prefix and
+          ( (tag.start_with?(@removed_prefix_string) and tag.length > @removed_length) or tag == @remove_prefix)
+        tag = tag[@removed_length..-1]
+      end
+      if @add_prefix
+        if tag and tag.length > 0
+          tag = @added_prefix_string + tag
+        else
+          tag = @add_prefix
+        end
+      end
+    end
+
+    # Handle all currently available events in stream
+    es.each do |time,record|
+      # Extract raw key value
+      raw_value = record[@key_name]
+
+      # Remember original time key, or raw event time
+      raw_time = record[@record_time_key]
+
+      # Try to parse it according to 'format'
+      t,values = raw_value ? @parser.parse(raw_value) : [nil, nil]
+
+      # Set new event's time to current time unless new time key was found in the sub-event
+      t ||= raw_time
+
+      r = values;
+
+      # Overlay new record on top of original record?
+      case @action
+      when 'inplace'
+        r = record.merge({@key_name => r})
+      when 'overlay'
+        r = record.merge(r)
+      when 'replace'
+        # noop
+      end
+
+      if ['overlay','replace'].include? @action
+        if not @keep_key
+          r.delete(@key_name)
+        end
+      end
+
+      # Preserve 'time' key?
+      if @keep_time
+        r[@record_time_key] = raw_time
+      end
+
+      # Emit event back to Fluent
+      if r
+        Fluent::Engine.emit(tag, t, r)
+      end
+    end
+
+    chain.next
+  end
+
+end

data/test/helper.rb

@@ -0,0 +1,28 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'test/unit'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'fluent/test'
+unless ENV.has_key?('VERBOSE')
+  nulllogger = Object.new
+  nulllogger.instance_eval {|obj|
+    def method_missing(method, *args)
+      # pass
+    end
+  }
+  $log = nulllogger
+end
+
+require 'fluent/plugin/out_burrow'
+
+class Test::Unit::TestCase
+end

metadata

@@ -0,0 +1,87 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-burrow
+version: !ruby/object:Gem::Version
+  version: '1.0'
+platform: ruby
+authors:
+- Tim Gunter
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-03-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Extract a single key (in formats Fluent can natively understand) from
+  an event and re-emit a new event that replaces the entire original record with that
+  key's values.
+email:
+- tim@vanillaforums.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- .travis.yml
+- Gemfile
+- LICENSE
+- README.md
+- Rakefile
+- examples/syslog.json.txt
+- fluent-plugin-burrow.gemspec
+- lib/fluent/plugin/out_burrow.rb
+- test/helper.rb
+homepage: https://github.com/vanilla/fluent-plugin-burrow
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
+specification_version: 4
+summary: Fluentd output filter plugin. Extract a single key (in formats Fluent can
+  natively understand) from an event and re-emit a new event that replaces the entire
+  original record with that key's values.
+test_files:
+- test/helper.rb