fluent-plugin-event-tail 0.0.1

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# gem.dependencies
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Mario Freitas (imkira@gmail.com)
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,123 @@
+# fluent-plugin-event-tail
+
+event-tail is an input plugin for [fluentd](http://fluentd.org) based on
+[in_tail](http://docs.fluentd.org/articles/in_tail) but for reading
+[tag, time, record] JSON messages from a file.
+
+If you use ```fluent-logger-ruby```, ```fluent-logger-node```, and so on, you
+are probably connecting to 'localhost' (or worse, to a remote host) and sending
+messages via TCP sockets.
+
+That's all fine but what happens if fluentd is down for some reason?
+Well, most (if not all) logger implementations keep a list of pending failed
+messages and try sending them periodically. But what if your application dies
+before having the chance to flush everything to fluentd? Well, you will
+probably lose those pending messages.
+
+The reason I made this plugin is to allow me (and hopefully you too) to rather
+send those messages, not via TCP sockets, but directly to a file and have
+fluentd read them. The obvious advantage is that you always have backups of
+your logs in your hard disks in case fluentd is not running or the DB where you
+are aggregating them died for some reason. Another reason is, your logs are
+still human readable and easily transformable (they are just newline separated
+JSON strings).
+
+The format of the messages is based on
+[in_forward](http://docs.fluentd.org/articles/in_forward) plugin:
+
+```
+stream:
+  message...
+
+message:
+  [tag, time, record]
+  or
+  [tag, [[time,record], [time,record], ...]]
+
+example:
+  ["myapp.access", [1308466941, {"a"=>1}], [1308466942, {"b"=>2}]]
+```
+
+This plugin is mostly based on
+[in_tail](http://docs.fluentd.org/articles/in_tail),
+and therefore you are expected to append newline separated JSON strings of
+messages in the above format. Also note that this plugin supports formatted
+time strings via the ```time_format``` config parameter, not just numeric UNIX
+timestamps.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'fluent-plugin-event-tail'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install fluent-plugin-event-tail
+
+## Configuration
+
+This plugin has the same configuration as
+[in_tail](http://docs.fluentd.org/articles/in_tail),
+with the exception of ```tag``` and ```format``` that were removed.
+
+```
+# fluent.conf
+<match prefix.**>
+  type event_tail
+  path /var/log/your_app.log
+  pos_file /var/log/your_app.log.pos
+# disable next line to enable custom time formatting
+# time_format %d %b %Y %H:%M:%S
+</match>
+```
+
+## Usage
+
+After running fluentd, you can emit events by appending to the file:
+
+```
+# time 0 means "current time on server"
+echo '["prefix.debug",0,{"foo":"bar"}]' >> /var/log/your_app.log
+# time passed as UNIX timestamp (2013-02-20 01:18:31 +0900)
+echo '["prefix.debug1",1361290711,{"foo1":"bar1"}]' >> /var/log/your_app.log
+# same time but passed as string
+echo '["prefix.debug2","2009-03-26 22:33:12 +0900",{"foo2":"bar2"}]' >> /var/log/your_app.log
+# group multiple events in one line by tag
+echo '["prefix.debug3",[[1361290714,{"foo3":"bar3"}],[1361290716,{"foo4":"bar4"}]]]' >> /var/log/your_app.log
+```
+
+fluentd will report something like:
+```
+prefix.debug: {"foo":"bar"}
+prefix.debug1: {"foo1":"bar1"}
+prefix.debug2: {"foo2":"bar2"}
+prefix.debug3: {"foo3":"bar3"}
+prefix.debug3: {"foo4":"bar4"}
+````
+
+Please note that contrarily to ```in_forward``` that only accepts UNIX
+timestamps, fluent-plugin-event-tail supports the original in_tail
+```time_format``` parameter, so you can also pass strings as event times.
+The default value of ```time_format``` is ```'%Y-%m-%d %H:%M:%S %z```.
+If you pass a numeric UNIX timestamp then ```time_format``` will be ignored.
+
+## Contributing
+
+You are very welcome to submit patches or improve this plugin.
+Just make sure you send me a pull request.
+
+## License
+
+fluent-plugin-event-tail is licensed under the MIT license:
+
+www.opensource.org/licenses/MIT
+
+## Copyright
+
+Copyright (c) 2013 Mario Freitas. See
+[LICENSE.txt](http://github.com/imkira/fluent-plugin-event-tail/blob/master/LICENSE.txt) for further details.

data/Rakefile

@@ -0,0 +1,7 @@
+require 'bundler/gem_tasks'
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |t|
+  t.libs = ['lib']
+  t.test_files = FileList['test/**/*.rb']
+end

data/fluent-plugin-event-tail.gemspec

@@ -0,0 +1,20 @@
+# -*- encoding: utf-8 -*-
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+Gem::Specification.new do |gem|
+  gem.name          = "fluent-plugin-event-tail"
+  gem.version       = "0.0.1"
+  gem.authors       = ["Mario Freitas"]
+  gem.email         = ["imkira@gmail.com"]
+  gem.description   = %q{fluentd input plugin derived from in_tail and inspired by in_forward for reading [tag, time, record] messages from a file}
+  gem.summary       = %q{fluentd input plugin for reading [tag, time, record] messages from a file}
+  gem.homepage      = "https://github.com/imkira/fluent-plugin-event-tail"
+
+  gem.files         = `git ls-files`.split($/)
+  gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
+  gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
+  gem.require_paths = ["lib"]
+  gem.add_development_dependency "fluentd"
+  gem.add_runtime_dependency "fluentd"
+end

data/lib/fluent/plugin/in_event_tail.rb

@@ -0,0 +1,102 @@
+
+# -*- encoding: utf-8 -*-
+
+# Copyright (c) 2013 Mario Freitas (imkira@gmail.com)
+#
+# MIT License
+#
+# Permission is hereby granted, free of charge, to any person obtaining
+# a copy of this software and associated documentation files (the
+# "Software"), to deal in the Software without restriction, including
+# without limitation the rights to use, copy, modify, merge, publish,
+# distribute, sublicense, and/or sell copies of the Software, and to
+# permit persons to whom the Software is furnished to do so, subject to
+# the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+require 'fluent/plugin/in_tail'
+
+module Fluent
+  class EventTailInput < Fluent::TailInput
+    Fluent::Plugin.register_input('event_tail', self)
+
+    config_param :time_format, :string, :default => '%Y-%m-%d %H:%M:%S %z'
+
+    # don't need tag parameter
+    config_set_default(:tag, '')
+
+    # don't need format parameter
+    config_set_default(:format, '')
+
+    def configure_parser(conf)
+      # just disable the default parser
+    end
+
+    def receive_lines(lines)
+      es = MultiEventStream.new
+      tag = nil
+      lines.each do |line|
+        begin
+          line.chomp!
+          tag = parse_line(line) do |time, record|
+            es.add(time, record)
+          end
+        rescue
+          $log.warn line.dump, :error=>$!.to_s
+          $log.debug_backtrace
+        end
+      end
+
+      unless tag.nil? || es.empty?
+        begin
+          Engine.emit_stream(tag, es)
+        rescue => e
+          # ignore errors. Engine shows logs and backtraces.
+        end
+      end
+    end
+
+    def parse_line(line, &block)
+      msg = Yajl.load(line)
+      tag = msg[0].to_s
+      entries = msg[1]
+
+      # [tag, [[time,record], [time,record], ...]]
+      if entries.is_a? Array
+        entries.each do |e|
+          time = parse_time(e[0])
+          record = e[1]
+          block.call(time, record)
+        end
+
+        # [tag, time, record]
+      else
+        time = parse_time(msg[1])
+        record = msg[2]
+        block.call(time, record)
+      end
+
+      tag
+    end
+
+    def parse_time(time)
+      if !@time_format.nil? and time.is_a? String
+        Time.strptime(time, @time_format).to_i
+      else
+        time = time.to_i
+        time = Engine.now if time == 0
+        time
+      end
+    end
+  end
+end

data/test/fluent/plugin/in_event_tail.rb

@@ -0,0 +1,131 @@
+require 'fluent/test'
+require 'fluent/plugin/in_event_tail'
+
+class EventTailInputTest < Test::Unit::TestCase
+  def setup
+    Fluent::Test.setup
+    FileUtils.rm_rf(TMP_DIR)
+    FileUtils.mkdir_p(TMP_DIR)
+  end
+
+  TMP_DIR = File.dirname(__FILE__) + "/../tmp"
+
+  CONFIG = %[
+    path #{TMP_DIR}/tail.log
+    tag t1
+    rotate_wait 2s
+    pos_file #{TMP_DIR}/tail.pos
+  ]
+
+  def create_driver(conf = CONFIG)
+    Fluent::Test::InputTestDriver.new(Fluent::EventTailInput).configure(conf)
+  end
+
+  def test_configure
+    d = create_driver
+    assert_equal ["#{TMP_DIR}/tail.log"], d.instance.paths
+    assert_equal "t1", d.instance.tag
+    assert_equal 2, d.instance.rotate_wait
+    assert_equal "#{TMP_DIR}/tail.pos", d.instance.pos_file
+  end
+
+  def test_simple_emit
+    File.open("#{TMP_DIR}/tail.log", "w") {|f|
+      f.puts '["foo",123,{"bar":"hoge"}]'
+    }
+
+    d = create_driver
+
+    d.run do
+      sleep 1
+
+      File.open("#{TMP_DIR}/tail.log", "a") {|f|
+        f.puts '["foo3",789,{"bar3":"hoge3"}]'
+      }
+      sleep 1
+    end
+
+    emits = d.emits
+    assert_equal(emits.length, 1)
+    assert_equal("foo3", emits[0][0])
+    assert_equal(789, emits[0][1])
+    assert_equal({"bar3"=>"hoge3"}, emits[0][2])
+  end
+
+  def test_default_time_format
+    File.open("#{TMP_DIR}/tail.log", "w") {|f|
+      f.puts '["foo","10/Oct/2010:20:57:59 -0700",{"bar":"hoge"}]'
+    }
+
+    d = create_driver
+
+    d.run do
+      sleep 1
+
+      File.open("#{TMP_DIR}/tail.log", "a") {|f|
+        f.puts '["foo4","2012-10-22 11:57:59 -0100",{"bar4":"hoge4"}]'
+      }
+      sleep 1
+    end
+
+    emits = d.emits
+    assert_equal(emits.length, 1)
+    assert_equal("foo4", emits[0][0])
+    assert_equal(1350910679, emits[0][1])
+    assert_equal({"bar4"=>"hoge4"}, emits[0][2])
+  end
+
+  def test_composed_emit
+    File.open("#{TMP_DIR}/tail.log", "w") {|f|
+      f.puts '["foo",123,{"bar":"hoge"}]'
+    }
+
+    d = create_driver
+
+    d.run do
+      sleep 1
+
+      File.open("#{TMP_DIR}/tail.log", "a") {|f|
+        f.puts '["foo5",[[91011,{"bar5":"hoge5"}],' +
+          '["2011-10-24 12:30:20 -0400",{"bar6":"hoge6"}]]]'
+      }
+      sleep 1
+    end
+
+    emits = d.emits
+    assert_equal(emits.length, 2)
+    assert_equal("foo5", emits[0][0])
+    assert_equal(91011, emits[0][1])
+    assert_equal({"bar5"=>"hoge5"}, emits[0][2])
+    assert_equal("foo5", emits[1][0])
+    assert_equal(1319473820, emits[1][1])
+    assert_equal({"bar6"=>"hoge6"}, emits[1][2])
+  end
+
+  def test_custom_time_format
+    File.open("#{TMP_DIR}/tail.log", "w") {|f|
+      f.puts '["foo",123,{"bar":"hoge"}]'
+    }
+
+    conf = CONFIG + %[
+      time_format %d %b %Y %H:%M:%S
+    ]
+
+    d = create_driver(conf)
+
+    d.run do
+      sleep 1
+
+      File.open("#{TMP_DIR}/tail.log", "a") {|f|
+        f.puts '["foo7","6 Dec 2001 12:33:45",{"bar7":"hoge7"}]'
+      }
+      sleep 1
+    end
+
+    emits = d.emits
+    assert_equal(emits.length, 1)
+    assert_equal("foo7", emits[0][0])
+    assert_equal(1007609625, emits[0][1])
+    assert_equal({"bar7"=>"hoge7"}, emits[0][2])
+  end
+end

metadata

@@ -0,0 +1,87 @@
+--- !ruby/object:Gem::Specification
+name: fluent-plugin-event-tail
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+  prerelease: 
+platform: ruby
+authors:
+- Mario Freitas
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2013-02-19 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: fluentd
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: fluentd input plugin derived from in_tail and inspired by in_forward
+  for reading [tag, time, record] messages from a file
+email:
+- imkira@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- .gitignore
+- Gemfile
+- LICENSE.txt
+- README.md
+- Rakefile
+- fluent-plugin-event-tail.gemspec
+- lib/fluent/plugin/in_event_tail.rb
+- test/fluent/plugin/in_event_tail.rb
+homepage: https://github.com/imkira/fluent-plugin-event-tail
+licenses: []
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.24
+signing_key: 
+specification_version: 3
+summary: fluentd input plugin for reading [tag, time, record] messages from a file
+test_files:
+- test/fluent/plugin/in_event_tail.rb