logstash-output-webhdfs 0.0.1 → 0.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9f37f22c7256c852597fc24ff4578a11503f68b7
-  data.tar.gz: 968727c576295ddb49f09f8230418e14828da865
+  metadata.gz: b62674a3d35ba63e84ddb865b846c9b89c1bd08a
+  data.tar.gz: 3e35d2d01038c0c8e57f798b13aa31f9e46614d7
 SHA512:
-  metadata.gz: 7d9f851d5b01629602616fdf58ad009bf09e3a8bf5befe8d0313e61eddaf10ce9b5ae2b196f38e2119f2f6098387547eda3b6f9f6af96af7685b45369efadb89
-  data.tar.gz: ee608a1e5f9876d0aa0df0d4bbc11192d62668ac97ca91b8a17434fc744730301b69472b020d0085efc3dfcded5bff68facc4c7b2c525e7ea6cefd89017fea6c
+  metadata.gz: 4e1b78ca8e557ac1b76a4d4aaa79b137092f440ee5e183e2347be1a13b9c36dbb852349aa4e3be59641f89e4cc2167918704e75a383fc7de294fbe0ccf874412
+  data.tar.gz: 2db171e7d65de252e3eedf7b301a20e234a88b53b0c62c7b5bf80d25fbc44d951ea123274256dd312fd9710cd80c686d1ea4cf87206a184aa042905ea264581b

data/README.md

@@ -1,86 +1,41 @@
-# Logstash Plugin
+logstash-webhdfs
+================
 
-This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
+A logstash plugin to store events via webhdfs.
 
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
-
-## Documentation
-
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the logstash-users@googlegroups.com mailing list.
-
-## Developing
-
-### 1. Plugin Developement and Testing
-
-#### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
-
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
-
-- Install dependencies
-```sh
-bundle install
-```
-
-#### Test
-
-- Update your dependencies
+Tested with v1.3.3, v1.4.0 and 1.5.0.
 
-```sh
-bundle install
-```
-
-- Run tests
-
-```sh
-bundle exec rspec
-```
-
-### 2. Running your unpublished Plugin in Logstash
-
-#### 2.1 Run in a local Logstash clone
-
-- Edit Logstash `Gemfile` and add the local plugin path, for example:
-```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
-```
-- Install plugin
-```sh
-bin/plugin install --no-verify
-```
-- Run Logstash with your plugin
-```sh
-bin/logstash -e 'filter {awesome {}}'
-```
-At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
-#### 2.2 Run in an installed Logstash
+This plugin only has a mandatory dependency on the webhdfs gem from Kazuki Ohta and TAGOMORI Satoshi (@see: https://github.com/kzk/webhdfs). Optional dependencies are zlib and snappy gem.
 
-You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+No jars from hadoop are needed, thus reducing configuration and compatibility problems.
 
-- Build your plugin gem
-```sh
-gem build logstash-filter-awesome.gemspec
+## Installation
+Change into your logstash install directory and execute:
 ```
-- Install the plugin from the Logstash home
-```sh
-bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+bin/plugin install logstash-output-webhdfs
 ```
-- Start Logstash and proceed to test the plugin
-
-## Contributing
 
-All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
-
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
-
-It is more important to the community that you are able to contribute.
+## Documentation
 
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.
+Example configuration:
+
+    output {
+        webhdfs {
+            workers => 2
+            server => "your.nameno.de:14000"
+            user => "flume"
+            path => "/user/flume/logstash/dt=%{+Y}-%{+M}-%{+d}/logstash-%{+H}.log"
+            flush_size => 500
+            compression => "snappy"
+            idle_flush_time => 10
+            retry_interval => 0.5
+        }
+    }
+
+For a complete list of options, see config section in source code.
+
+This plugin has dependencies on:
+ * webhdfs module @<https://github.com/kzk/webhdfs>
+ * snappy module @<https://github.com/miyucy/snappy>

data/lib/logstash/outputs/webhdfs.rb

@@ -7,11 +7,18 @@ require "stud/buffer"
 # restapi.  
 #
 # This plugin only has a mandatory dependency on the webhdfs gem from 
-# Tagamori Satoshi (@see: https://github.com/kzk/webhdfs).
+# Kazuki Ohta and TAGOMORI Satoshi (@see: https://github.com/kzk/webhdfs).
 # Optional dependencies are zlib and snappy gem. 
 # No jars from hadoop are needed, thus reducing configuration and compatibility
 # problems.
 #
+# If you get an error like:
+#
+#     Max write retries reached. Exception: initialize: name or service not known {:level=>:error}
+#
+# make sure, that the hostname of your namenode is resolvable on the host running logstash. When creating/appending
+# to a file, webhdfs somtime sends a 307 TEMPORARY_REDIRECT with the HOSTNAME of the machine its running on.
+#
 # USAGE:
 # This is an example of logstash config:
 #
@@ -95,8 +102,8 @@ class LogStash::Outputs::WebHdfs < LogStash::Outputs::Base
   # How many times should we retry.
   config :retry_times, :validate => :number, :default => 5
 
-  # Compress output. One of [false, 'snappy', 'gzip']
-  config :compress, :validate => ["false", "snappy", "gzip"], :default => "false"
+  # Compress output. One of ['none', 'snappy', 'gzip']
+  config :compression, :validate => ["none", "snappy", "gzip"], :default => "none"
 
   # Set snappy chunksize. Only neccessary for stream format. Defaults to 32k. Max is 65536
   # @see http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt
@@ -116,13 +123,13 @@ class LogStash::Outputs::WebHdfs < LogStash::Outputs::Base
     rescue LoadError
       @logger.error("Module webhdfs could not be loaded.")
     end
-    if @compress == "gzip"
+    if @compression == "gzip"
       begin
         require "zlib"
       rescue LoadError
         @logger.error("Gzip compression selected but zlib module could not be loaded.")
       end
-    elsif @compress == "snappy"
+    elsif @compression == "snappy"
       begin
         require "snappy"
       rescue LoadError
@@ -185,15 +192,15 @@ class LogStash::Outputs::WebHdfs < LogStash::Outputs::Base
       output_files[path] << event_as_string
     end
     output_files.each do |path, output|
-      if @compress == "gzip"
+      if @compression == "gzip"
         path += ".gz"
         output = compress_gzip(output)
-      elsif @compress == "snappy"
+      elsif @compression == "snappy"
         path += ".snappy"
         if @snappy_format == "file"
           output = compress_snappy_file(output)
         elsif
-        output = compress_snappy_stream(output)
+          output = compress_snappy_stream(output)
         end
       end
       write_tries = 0
@@ -234,7 +241,7 @@ class LogStash::Outputs::WebHdfs < LogStash::Outputs::Base
     data= data.encode(Encoding::ASCII_8BIT, "binary", :undef => :replace)
     buffer = StringIO.new('', 'w')
     buffer.set_encoding Encoding::ASCII_8BIT unless RUBY_VERSION =~ /^1\.8/
-    compressed = Snappy.deflate(chunk)
+    compressed = Snappy.deflate(data)
     buffer << [compressed.size, compressed].pack("Na*")
     buffer.string
   end
@@ -261,7 +268,7 @@ class LogStash::Outputs::WebHdfs < LogStash::Outputs::Base
       @client.append(path, data)
     rescue WebHDFS::FileNotFoundError
       # Add snappy header if format is "file".
-      if @snappy_format == "file"
+      if @compression == "snappy" and @snappy_format == "file"
         @client.create(path, get_snappy_header! + data)
       elsif
         @client.create(path, data)

data/logstash-output-webhdfs.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-output-webhdfs'
-  s.version         = '0.0.1'
+  s.version         = '0.0.2'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "Plugin to write events to hdfs via webhdfs."
   s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
@@ -24,4 +24,4 @@ Gem::Specification.new do |s|
   s.add_runtime_dependency 'webhdfs'
   s.add_runtime_dependency 'snappy'
   s.add_development_dependency 'logstash-devutils'
-end
+end

data/spec/outputs/webhdfs_spec.rb

@@ -1,36 +1,45 @@
 # encoding: utf-8
 require 'logstash/devutils/rspec/spec_helper'
 require 'logstash/outputs/webhdfs'
+require 'webhdfs'
+require 'json'
 
 describe 'outputs/webhdfs' do
 
-  let(:event) do
-    LogStash::Event.new(
-        'message' => 'fanastic log entry',
-        'source' => 'someapp',
-        'type' => 'nginx',
+  webhdfs_server = 'localhost'
+  webhdfs_port = 50070
+  webhdfs_user = 'hadoop'
+  path_to_testlog = '/user/hadoop/test.log'
+  current_logfile_name = '/user/hadoop/test.log'
+  current_config = ""
+
+      event = LogStash::Event.new(
+        'message' => 'Hello world!',
+        'source' => 'out of the blue',
+        'type' => 'generator',
+        'host' => 'localhost',
         '@timestamp' => LogStash::Timestamp.now)
-  end
+
+  default_config =  { 'server' => webhdfs_server + ':' + webhdfs_port.to_s,
+                      'user' => webhdfs_user,
+                      'path' => path_to_testlog,
+                      'compression' => 'none' }
+
+  client = WebHDFS::Client.new(webhdfs_server, webhdfs_port, webhdfs_user)
 
   context 'when initializing' do
 
     it 'should fail to register without required values' do
-      configuration_error = false
-      begin
-        LogStash::Plugin.lookup("output", "webhdfs").new()
-      rescue LogStash::ConfigurationError
-        configuration_error = true
-      end
-      insist { configuration_error } == true
+      expect { LogStash::Plugin.lookup("output", "webhdfs").new() }.to raise_error(error=LogStash::ConfigurationError)
     end
 
-    subject = LogStash::Plugin.lookup("output", "webhdfs").new('server' => '127.0.0.1:50070', 'path' => '/path/to/webhdfs.file')
-
-    it 'should register' do
+    it 'should register with default values' do
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(default_config)
       expect { subject.register }.to_not raise_error
     end
 
     it 'should have default config values' do
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(default_config)
       insist { subject.idle_flush_time } == 1
       insist { subject.flush_size } == 500
       insist { subject.open_timeout } == 30
@@ -43,6 +52,78 @@ describe 'outputs/webhdfs' do
       insist { subject.snappy_format } == 'stream'
       insist { subject.remove_at_timestamp } == true
     end
+  end
+
+  context 'when writing messages' do
+
+    before :each do
+      current_logfile_name = path_to_testlog
+      current_config = default_config.clone
+    end
+
+    it 'should match the event data' do
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(current_config)
+      subject.register
+      subject.receive(event)
+      subject.teardown
+      insist { client.read(current_logfile_name).strip } == event.to_json
+    end
+
+    it 'should match the configured pattern' do
+      current_config['message_format'] = '%{message} came %{source}.'
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(current_config)
+      subject.register
+      subject.receive(event)
+      subject.teardown
+      insist { client.read(current_logfile_name).strip } == 'Hello world! came out of the blue.'
+    end
+
+    # Hive does not like a leading "@", but we need @timestamp for path calculation.
+    it 'should remove the @timestamp field if configured' do
+      current_config['remove_at_timestamp'] = true
+      current_config['message_format'] = '%{@timestamp} should be missing.'
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(current_config)
+      subject.register
+      subject.receive(event)
+      subject.teardown
+      insist { client.read(current_logfile_name).strip } == '%{@timestamp} should be missing.'
+    end
+
+    it 'should flush after configured idle time' do
+      current_config['idle_flush_time'] = 2
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(current_config)
+      subject.register
+      subject.receive(event)
+      expect { client.read(current_logfile_name) }.to raise_error(error=WebHDFS::FileNotFoundError)
+      sleep 3
+      insist { client.read(current_logfile_name).strip } == event.to_json
+    end
+
+    it 'should write some messages uncompressed' do
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(current_config)
+      subject.register
+      for _ in 0..499
+        subject.receive(event)
+      end
+      subject.teardown
+      insist { client.read(current_logfile_name).lines.count } == 500
+    end
+
+    it 'should write some messages gzip compressed' do
+      current_logfile_name = current_logfile_name + ".gz"
+      current_config['compression'] = 'gzip'
+      subject = LogStash::Plugin.lookup("output", "webhdfs").new(current_config)
+      subject.register
+      for _ in 0..499
+        subject.receive(event)
+      end
+      subject.teardown
+      insist { Zlib::Inflate.new(window_bits=47).inflate(client.read(current_logfile_name)).lines.count } == 500
+    end
+
+    after :each do
+      client.delete(current_logfile_name)
+    end
 
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-webhdfs
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
 platform: ruby
 authors:
 - Björn Puttmann, loshkovskyi
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2015-05-19 00:00:00.000000000 Z
+date: 2015-05-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core