logstash-output-rados 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 973fd1939d1d97304d5bc37141b68ecbd78e9e83
+  data.tar.gz: 8e0d1a2cdf98b8233e008d661eef46925ed8fd4f
+SHA512:
+  metadata.gz: 499d69316c3b4547281665a0c59d1f6d96ee472dce147ce42811e8aff979f8e14893a68697dbb1afdd0e450c85cb6f65bf366028e3f9f125dc95a244227619fa
+  data.tar.gz: 77fb5bfab3c8cfd1feb04ecf34bd251431f9fd3060ce1f900d7a8ea21ed326f626c887aafb6e816663a84f40da2c6f8e10869b439d286519c50f578f34dbe603

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+# 1.0.0
+- Initial release of the Rados output plugin

data/CONTRIBUTORS

@@ -0,0 +1,11 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Balint Csergo (deathowl)
+* Thanks for the developers of the S3 plugin.
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,16 @@
+[Missing the other part of the readme]
+
+## Running the tests
+
+```
+bundle install
+bundle rspec
+```
+
+If you want to run the integration test you have to have access to a ceph cluster, and a real bucket
+
+```
+RADOS_LOGSTASH_TEST_POOL=mytest bundle exec rspec spec/integration/rados_spec.rb --tag integration
+RADOS_LOGSTASH_TEST_POOL=mytest bundle exec rspec spec/outputs/rados_spec.rb
+
+```

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,88 @@
+# Logstash Plugin
+
+
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/rados.rb

@@ -0,0 +1,374 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "stud/temporary"
+require "stud/task"
+require "socket" # for Socket.gethostname
+require "thread"
+require "tmpdir"
+require "fileutils"
+
+
+# INFORMATION:
+#
+# This plugin sends logstash events to Ceph.
+# To use it you need to have a properly configured librados and a valid ceph cluster.
+# Make sure you have permissions to write files on Ceph.  Also be sure to run logstash as super user to establish a connection.
+#
+#
+# This plugin outputs temporary files to "/opt/logstash/rados_temp/". If you want, you can change the path at the start of register method.
+# These files have a special name, for example:
+#
+# ls.rados.ip-10-228-27-95.2013-04-18T10.00.tag_hello.part0.txt
+#
+# ls.rados : indicate logstash plugin rados
+#
+# "ip-10-228-27-95" : indicates the ip of your machine.
+# "2013-04-18T10.00" : represents the time whenever you specify time_file.
+# "tag_hello" : this indicates the event's tag.
+# "part0" : this means if you indicate size_file then it will generate more parts if you file.size > size_file.
+#           When a file is full it will be pushed to the pool and will be deleted from the temporary directory.
+#           If a file is empty is not pushed, it is not deleted.
+#
+# This plugin have a system to restore the previous temporary files if something crash.
+#
+##[Note] :
+#
+## If you specify size_file and time_file then it will create file for each tag (if specified), when time_file or
+## their size > size_file, it will be triggered then they will be pushed on Rados pool and will delete from local disk.
+## If you don't specify size_file, but time_file then it will create only one file for each tag (if specified).
+## When time_file it will be triggered then the files will be pushed on Rados and delete from local disk.
+#
+## If you don't specify time_file, but size_file  then it will create files for each tag (if specified),
+## that will be triggered when their size > size_file, then they will be pushed on Rados pool and will delete from local disk.
+#
+## If you don't specific size_file and time_file you have a curios mode. It will create only one file for each tag (if specified).
+## Then the file will be rest on temporary directory and don't will be pushed on pool until we will restart logstash.
+#
+#
+# #### Usage:
+# This is an example of logstash config:
+# [source,ruby]
+# output {
+#    rados{
+#      mypool => "mypool"             (required)
+#      size_file => 2048                        (optional)
+#      time_file => 5                           (optional)
+#    }
+#
+class LogStash::Outputs::Rados < LogStash::Outputs::Base
+
+  TEMPFILE_EXTENSION = "txt"
+  RADOS_INVALID_CHARACTERS = /[\^`><]/
+
+
+  config_name "rados"
+  default :codec, 'line'
+
+  # Rados pool
+  config :pool, :validate => :string, :default => 'logstash'
+
+  # Set the size of file in bytes, this means that files on pool when have dimension > file_size, they are stored in two or more file.
+  # If you have tags then it will generate a specific size file for every tags
+  ##NOTE: define size of file is the better thing, because generate a local temporary file on disk and then put it in pool.
+  config :size_file, :validate => :number, :default => 0
+
+  # Set the time, in minutes, to close the current sub_time_section of pool.
+  # If you define file_size you have a number of files in consideration of the section and the current tag.
+  # 0 stay all time on listerner, beware if you specific 0 and size_file 0, because you will not put the file on pool,
+  # for now the only thing this plugin can do is to put the file when logstash restart.
+  config :time_file, :validate => :number, :default => 0
+
+  # Set the directory where logstash will store the tmp files before sending it to Rados
+  # default to the current OS temporary directory in linux /tmp/logstash
+  config :temporary_directory, :validate => :string, :default => File.join(Dir.tmpdir, "logstash")
+
+  # Specify a prefix to the uploaded filename, this can simulate directories on rados
+  config :prefix, :validate => :string, :default => ''
+
+  # Specify how many workers to use to upload the files to Rados
+  config :upload_workers_count, :validate => :number, :default => 1
+
+  # Define tags to be appended to the file on the Rados pool.
+  #
+  # Example:
+  # tags => ["elasticsearch", "logstash", "kibana"]
+  #
+  # Will generate this file:
+  # "ls.rados.logstash.local.2015-01-01T00.00.tag_elasticsearch.logstash.kibana.part0.txt"
+  #
+  config :tags, :validate => :array, :default => []
+
+
+  config :use_ssl, :validate => :boolean, :default => true
+
+  # Exposed attributes for testing purpose.
+  attr_accessor :tempfile
+  attr_reader :page_counter
+  # Exposed attributes for testing purpose.
+  attr_reader :cluster
+
+
+  public
+  def write_on_pool(file)
+    rados_pool = @cluster.pool(@pool)
+    rados_pool.open
+    remote_filename = "#{@prefix}#{File.basename(file)}"
+
+    @logger.debug("RADOS: ready to write file in pool", :remote_filename => remote_filename, :pool => @pool)
+
+    File.open(file, 'r') do |fileIO|
+      begin
+        # prepare for write the file
+        object = rados_pool.rados_object(remote_filename)
+        object.write(0, fileIO.read)
+      rescue SystemCallError => error
+        @logger.error("RADOS: CEPH error", :error => error)
+        raise LogStash::Error, "CEPH Configuration Error, #{error}"
+      ensure
+        rados_pool.close
+      end
+    end
+
+    @logger.debug("RADOS: has written remote file in pool", :remote_filename => remote_filename, :pool  => @pool)
+  end
+
+  # This method is used for create new empty temporary files for use. Flag is needed for indicate new subsection time_file.
+  public
+  def create_temporary_file
+    filename = File.join(@temporary_directory, get_temporary_filename(@page_counter))
+
+    @logger.debug("RADOS: Creating a new temporary file", :filename => filename)
+
+    @file_rotation_lock.synchronize do
+      unless @tempfile.nil?
+        @tempfile.close
+      end
+
+      @tempfile = File.open(filename, "a")
+    end
+  end
+
+  public
+  def register
+    require "ceph-ruby"
+    # required if using ruby version < 2.0
+    # http://ruby.awsblog.com/post/Tx16QY1CI5GVBFT/Threading-with-the-AWS-SDK-for-Ruby
+    workers_not_supported
+
+    @cluster = CephRuby::Cluster.new
+    @upload_queue = Queue.new
+    @file_rotation_lock = Mutex.new
+
+    if @prefix && @prefix =~ RADOS_INVALID_CHARACTERS
+      @logger.error("RADOS: prefix contains invalid characters", :prefix => @prefix, :contains => RADOS_INVALID_CHARACTERS)
+      raise LogStash::ConfigurationError, "RADOS: prefix contains invalid characters"
+    end
+
+    if !Dir.exist?(@temporary_directory)
+      FileUtils.mkdir_p(@temporary_directory)
+    end
+    restore_from_crashes if @restore == true
+    reset_page_counter
+    create_temporary_file
+    configure_periodic_rotation if time_file != 0
+    configure_upload_workers
+
+    @codec.on_event do |event, encoded_event|
+      handle_event(encoded_event)
+    end
+  end
+
+  public
+  def restore_from_crashes
+    @logger.debug("RADOS: is attempting to verify previous crashes...")
+
+    Dir[File.join(@temporary_directory, "*.#{TEMPFILE_EXTENSION}")].each do |file|
+      name_file = File.basename(file)
+      @logger.warn("RADOS: have found temporary file the upload process crashed, uploading file to Rados.", :filename => name_file)
+      move_file_to_pool_async(file)
+    end
+  end
+
+  public
+  def move_file_to_pool(file)
+    if !File.zero?(file)
+      write_on_pool(file)
+      @logger.debug("RADOS: file was put on the upload thread", :filename => File.basename(file), :pool => @pool)
+    end
+
+    begin
+      File.delete(file)
+    rescue Errno::ENOENT
+      # Something else deleted the file, logging but not raising the issue
+      @logger.warn("RADOS: Cannot delete the temporary file since it doesn't exist on disk", :filename => File.basename(file))
+    rescue Errno::EACCES
+      @logger.error("RADOS: Logstash doesnt have the permission to delete the file in the temporary directory.", :filename => File.basename(file), :temporary_directory => @temporary_directory)
+    end
+  end
+
+  public
+  def periodic_interval
+    @time_file * 60
+  end
+
+  public
+  def get_temporary_filename(page_counter = 0)
+    current_time = Time.now
+    filename = "ls.rados.#{Socket.gethostname}.#{current_time.strftime("%Y-%m-%dT%H.%M")}"
+
+    if @tags.size > 0
+      return "#{filename}.tag_#{@tags.join('.')}.part#{page_counter}.#{TEMPFILE_EXTENSION}"
+    else
+      return "#{filename}.part#{page_counter}.#{TEMPFILE_EXTENSION}"
+    end
+  end
+
+  public
+  def receive(event)
+
+    @codec.encode(event)
+  end
+
+  public
+  def rotate_events_log?
+    @file_rotation_lock.synchronize do
+      @tempfile.size > @size_file
+    end
+  end
+
+  public
+  def write_events_to_multiple_files?
+    @size_file > 0
+  end
+
+  public
+  def write_to_tempfile(event)
+    begin
+      @logger.debug("RADOS: put event into tempfile ", :tempfile => File.basename(@tempfile))
+
+      @file_rotation_lock.synchronize do
+        @tempfile.syswrite(event)
+      end
+    rescue Errno::ENOSPC
+      @logger.error("RADOS: No space left in temporary directory", :temporary_directory => @temporary_directory)
+      close
+    end
+  end
+
+  public  # Specify how many workers to use to upload the files to Rados
+  config :upload_workers_count, :validate => :number, :default => 1
+  def close
+    shutdown_upload_workers
+    @periodic_rotation_thread.stop! if @periodic_rotation_thread
+
+    @file_rotation_lock.synchronize do
+      @tempfile.close unless @tempfile.nil? && @tempfile.closed?
+    end
+    @cluster.shutdown
+  end
+
+  private
+  def shutdown_upload_workers
+    @logger.debug("RADOS: Gracefully shutdown the upload workers")
+    @upload_queue << LogStash::ShutdownEvent
+  end
+
+  private
+  def handle_event(encoded_event)
+    if write_events_to_multiple_files?
+      if rotate_events_log?
+        @logger.debug("RADOS: tempfile is too large, let's upload it and create new file", :tempfile => File.basename(@tempfile))
+
+        move_file_to_pool_async(@tempfile.path)
+        next_page
+        create_temporary_file
+      else
+        @logger.debug("RADOS: tempfile file size report.", :tempfile_size => @tempfile.size, :size_file => @size_file)
+      end
+    end
+
+    write_to_tempfile(encoded_event)
+  end
+
+  private
+  def configure_periodic_rotation
+    @periodic_rotation_thread = Stud::Task.new do
+      LogStash::Util::set_thread_name("<RADOS periodic uploader")
+
+      Stud.interval(periodic_interval, :sleep_then_run => true) do
+        @logger.debug("RADOS: time_file triggered, uploading the file", :filename => @tempfile.path)
+
+        move_file_to_pool_async(@tempfile.path)
+        next_page
+        create_temporary_file
+      end
+    end
+  end
+
+  private
+  def configure_upload_workers
+    @logger.debug("RADOS: Configure upload workers")
+
+    @upload_workers = @upload_workers_count.times.map do |worker_id|
+      Stud::Task.new do
+        LogStash::Util::set_thread_name("<RADOS upload worker #{worker_id}")
+
+        while true do
+          @logger.debug("RADOS: upload worker is waiting for a new file to upload.", :worker_id => worker_id)
+
+          upload_worker
+        end
+      end
+    end
+  end
+
+  private
+  def upload_worker
+    file = @upload_queue.deq
+
+    case file
+      when LogStash::ShutdownEvent
+        @logger.debug("RADOS: upload worker is shutting down gracefuly")
+        @upload_queue.enq(LogStash::ShutdownEvent)
+      else
+        @logger.debug("RADOS: upload worker is uploading a new file", :filename => File.basename(file))
+        move_file_to_pool(file)
+    end
+  end
+
+  private
+  def next_page
+    @page_counter += 1
+  end
+
+  private
+  def reset_page_counter
+    @page_counter = 0
+  end
+
+  private
+  def delete_on_pool(filename)
+    rados_pool = @cluster.pool(@pool)
+    rados_pool.open
+    remote_filename = "#{@prefix}#{File.basename(filename)}"
+
+    @logger.debug("RADOS: delete file from pool", :remote_filename => remote_filename, :pool => @pool)
+
+    begin
+        object = rados_pool.rados_object(remote_filename)
+        object.destroy
+    rescue SystemCallError => error
+        @logger.error("RADOS: CEPH error", :error => error)
+        raise LogStash::Error, "CEPH Configuration Error, #{error}"
+    ensure
+        rados_pool.close
+    end
+  end
+
+  private
+  def move_file_to_pool_async(file)
+    @logger.debug("RADOS: Sending the file to the upload queue.", :filename => File.basename(file))
+    @upload_queue.enq(file)
+  end
+end

data/logstash-output-rados.gemspec

@@ -0,0 +1,29 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'logstash-output-rados'
+  s.version         = '1.0.0'
+  s.licenses        = ['Apache License (2.0)']
+  s.summary         = "This plugin was created for store the logstash's events into  Ceph's Rados"
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Deathowl"]
+  s.email           = 'bagoly@endticket.com'
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
+  s.add_runtime_dependency 'stud', '~> 0.0.22'
+  s.add_runtime_dependency 'ceph-ruby', '~> 1.1'
+  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'logstash-input-generator'
+  s.add_development_dependency 'logstash-codec-line'
+end

data/spec/integration/rados_spec.rb

@@ -0,0 +1,87 @@
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/rados"
+require 'socket'
+require "fileutils"
+require "stud/temporary"
+require_relative "../supports/helpers"
+
+describe LogStash::Outputs::Rados, :integration => true, :rados => true do
+  before do
+    Thread.abort_on_exception = true
+  end
+
+  let!(:minimal_settings)  {  { "pool" => ENV['RADOS_LOGSTASH_TEST_POOL'],
+                                "temporary_directory" => Stud::Temporary.pathname('temporary_directory') }}
+
+  let!(:rados_object) do
+      radosoutput = LogStash::Outputs::Rados.new(minimal_settings)
+      radosoutput.register
+      radosoutput.cluster
+  end
+
+  describe "#register" do
+    it "write a file on the pool to check permissions" do
+      rados = LogStash::Outputs::Rados.new(minimal_settings)
+      expect(rados.register).not_to raise_error
+    end
+  end
+
+  describe "#write_on_pool" do
+    after(:each) do
+      File.unlink(fake_data.path)
+    end
+
+    let!(:fake_data) { Stud::Temporary.file }
+
+    it "should prefix the file on the pool if a prefix is specified" do
+      prefix = "my-prefix"
+
+      config = minimal_settings.merge({
+        "prefix" => prefix,
+      })
+
+      rados = LogStash::Outputs::Rados.new(config)
+      rados.register
+      rados.write_on_pool(fake_data)
+
+      expect(key_exists_on_pool?("#{prefix}#{File.basename(fake_data.path)}")).to eq(true)
+    end
+
+    it 'should use the same local filename if no prefix is specified' do
+      rados = LogStash::Outputs::Rados.new(minimal_settings)
+      rados.register
+      rados.write_on_pool(fake_data)
+
+      expect(key_exists_on_pool?(File.basename(fake_data.path))).to eq(true)
+    end
+  end
+
+  describe "#move_file_to_pool" do
+    let!(:rados) { LogStash::Outputs::Rados.new(minimal_settings) }
+
+    before do
+      rados.register
+    end
+
+    it "should upload the file if the size > 0" do
+      tmp = Stud::Temporary.file
+      allow(File).to receive(:zero?).and_return(false)
+      rados.move_file_to_pool(tmp)
+      expect(key_exists_on_pool?(File.basename(tmp.path))).to eq(true)
+    end
+  end
+
+  describe "#restore_from_crashes" do
+    it "read the temp directory and upload the matching file to rados" do
+      Stud::Temporary.pathname do |temp_path|
+        tempfile = File.open(File.join(temp_path, 'A'), 'w+') { |f| f.write('test')}
+
+        rados = LogStash::Outputs::Rados.new(minimal_settings.merge({ "temporary_directory" => temp_path }))
+        rados.restore_from_crashes
+
+        expect(File.exist?(tempfile.path)).to eq(false)
+        expect(key_exists_on_pool?(File.basename(tempfile.path))).to eq(true)
+      end
+    end
+  end
+end

data/spec/outputs/rados_spec.rb

@@ -0,0 +1,279 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/rados"
+require "logstash/codecs/line"
+require "logstash/pipeline"
+require "fileutils"
+require_relative "../supports/helpers"
+
+describe LogStash::Outputs::Rados do
+  before do
+    # We stub all the calls from Rad, for more information see:
+    # http://ruby.awsblog.com/post/Tx2SU6TYJWQQLC3/Stubbing-AWS-Responses
+    Thread.abort_on_exception = true
+  end
+
+  let(:minimal_settings)  {  {"pool" => ENV['RADOS_LOGSTASH_TEST_POOL'] }  }
+
+  describe "#register" do
+    it "should create the tmp directory if it doesn't exist" do
+      temporary_directory = Stud::Temporary.pathname("temporary_directory")
+
+      config = {
+        "pool" => "logstash",
+        "size_file" => 10,
+        "temporary_directory" => temporary_directory
+      }
+
+      rados = LogStash::Outputs::Rados.new(config)
+      rados.register
+
+      expect(Dir.exist?(temporary_directory)).to eq(true)
+      rados.close
+      FileUtils.rm_r(temporary_directory)
+    end
+
+    it "should raise a ConfigurationError if the prefix contains one or more '\^`><' characters" do
+      config = {
+        "prefix" => "`no\><^"
+      }
+
+      rados = LogStash::Outputs::Rados.new(config)
+
+      expect {
+        rados.register
+      }.to raise_error(LogStash::ConfigurationError)
+    end
+  end
+
+  describe "#generate_temporary_filename" do
+    before do
+      allow(Socket).to receive(:gethostname) { "logstash.local" }
+    end
+
+    it "should add tags to the filename if present" do
+      config = minimal_settings.merge({ "tags" => ["elasticsearch", "logstash", "kibana"], "temporary_directory" => "/tmp/logstash"})
+      rados = LogStash::Outputs::Rados.new(config)
+      expect(rados.get_temporary_filename).to match(/^ls\.rados\.logstash\.local\.\d{4}-\d{2}\-\d{2}T\d{2}\.\d{2}\.tag_#{config["tags"].join("\.")}\.part0\.txt\Z/)
+    end
+
+    it "should not add the tags to the filename" do
+      config = minimal_settings.merge({ "tags" => [], "temporary_directory" => "/tmp/logstash" })
+      rados = LogStash::Outputs::Rados.new(config)
+      expect(rados.get_temporary_filename(3)).to match(/^ls\.rados\.logstash\.local\.\d{4}-\d{2}\-\d{2}T\d{2}\.\d{2}\.part3\.txt\Z/)
+    end
+
+    it "normalized the temp directory to include the trailing slash if missing" do
+      rados = LogStash::Outputs::Rados.new(minimal_settings.merge({ "temporary_directory" => "/tmp/logstash" }))
+      expect(rados.get_temporary_filename).to match(/^ls\.rados\.logstash\.local\.\d{4}-\d{2}\-\d{2}T\d{2}\.\d{2}\.part0\.txt\Z/)
+    end
+  end
+
+
+  describe "#write_events_to_multiple_files?" do
+    it 'returns true if the size_file is != 0 ' do
+      rados = LogStash::Outputs::Rados.new(minimal_settings.merge({ "size_file" => 200 }))
+      expect(rados.write_events_to_multiple_files?).to eq(true)
+    end
+
+    it 'returns false if size_file is zero or not set' do
+      rados = LogStash::Outputs::Rados.new(minimal_settings)
+      expect(rados.write_events_to_multiple_files?).to eq(false)
+    end
+  end
+
+  describe "#write_to_tempfile" do
+    it "should append the event to a file" do
+      Stud::Temporary.file("logstash", "a+") do |tmp|
+        rados = LogStash::Outputs::Rados.new(minimal_settings)
+        rados.register
+        rados.tempfile = tmp
+        rados.write_to_tempfile("test-write")
+        tmp.rewind
+        expect(tmp.read).to eq("test-write")
+      end
+    end
+  end
+
+  describe "#rotate_events_log" do
+
+    context "having a single worker" do
+      let(:rados) { LogStash::Outputs::Rados.new(minimal_settings.merge({ "size_file" => 1024 })) }
+
+      before(:each) do
+        rados.register
+      end
+
+      it "returns true if the tempfile is over the file_size limit" do
+        Stud::Temporary.file do |tmp|
+          allow(tmp).to receive(:size) { 2024001 }
+
+          rados.tempfile = tmp
+          expect(rados.rotate_events_log?).to be(true)
+        end
+      end
+
+      it "returns false if the tempfile is under the file_size limit" do
+        Stud::Temporary.file do |tmp|
+          allow(tmp).to receive(:size) { 100 }
+
+          rados.tempfile = tmp
+          expect(rados.rotate_events_log?).to eq(false)
+        end
+      end
+    end
+
+    context "having periodic rotations" do
+      let(:rados)  { LogStash::Outputs::Rados.new(minimal_settings.merge({ "size_file" => 1024, "time_file" => 6e-10 })) }
+      let(:tmp) { Tempfile.new('rados_rotation_temp_file') }
+
+      before(:each) do
+        rados.tempfile = tmp
+        rados.register
+      end
+
+      after(:each) do
+        rados.close
+        tmp.close 
+        tmp.unlink
+      end
+
+      it "raises no error when periodic rotation happen" do
+        1000.times do
+          expect { rados.rotate_events_log? }.not_to raise_error
+        end
+      end
+    end
+  end
+
+  describe "#move_file_to_pool" do
+    subject { LogStash::Outputs::Rados.new(minimal_settings) }
+
+    it "should always delete the source file" do
+      tmp = Stud::Temporary.file
+
+      allow(File).to receive(:zero?).and_return(true)
+      expect(File).to receive(:delete).with(tmp)
+
+      subject.move_file_to_pool(tmp)
+    end
+
+    it 'should not upload the file if the size of the file is zero' do
+      temp_file = Stud::Temporary.file
+      allow(temp_file).to receive(:zero?).and_return(true)
+
+      expect(subject).not_to receive(:write_on_pool)
+      subject.move_file_to_pool(temp_file)
+    end
+
+    it "should upload the file if the size > 0" do
+      tmp = Stud::Temporary.file
+
+      allow(File).to receive(:zero?).and_return(false)
+      expect(subject).to receive(:write_on_pool)
+
+      subject.move_file_to_pool(tmp)
+    end
+  end
+
+  describe "#restore_from_crashes" do
+    it "read the temp directory and upload the matching file to rados" do
+      subject = LogStash::Outputs::Rados.new(minimal_settings.merge({ "temporary_directory" => "/tmp/logstash/" }))
+
+      expect(Dir).to receive(:[]).with("/tmp/logstash/*.txt").and_return(["/tmp/logstash/01.txt"])
+      expect(subject).to receive(:move_file_to_pool_async).with("/tmp/logstash/01.txt")
+
+
+      subject.restore_from_crashes
+    end
+  end
+
+  describe "#receive" do
+    it "should send the event through the codecs" do
+      data = {"foo" => "bar", "baz" => {"bah" => ["a","b","c"]}, "@timestamp" => "2014-05-30T02:52:17.929Z"}
+      event = LogStash::Event.new(data)
+
+      expect_any_instance_of(LogStash::Codecs::Line).to receive(:encode).with(event)
+
+      subject = LogStash::Outputs::Rados.new(minimal_settings)
+      subject.register
+
+      subject.receive(event)
+    end
+  end
+
+  describe "when rotating the temporary file" do
+    before { allow(File).to receive(:delete) }
+
+    it "doesn't skip events if using the size_file option" do
+      Stud::Temporary.directory do |temporary_directory|
+        size_file = rand(200..20000)
+        event_count = rand(300..15000)
+
+        config = %Q[
+        input {
+          generator {
+            count => #{event_count}
+          }
+        }
+        output {
+          rados {
+            size_file => #{size_file}
+            codec => line
+            temporary_directory => '#{temporary_directory}'
+            pool =>  '#{ENV['RADOS_LOGSTASH_TEST_POOL']}'
+          }
+        }
+        ]
+
+        pipeline = LogStash::Pipeline.new(config)
+
+        pipeline_thread = Thread.new { pipeline.run }
+        sleep 0.1 while !pipeline.ready?
+        pipeline_thread.join
+
+        events_written_count = events_in_files(Dir[File.join(temporary_directory, 'ls.*.txt')])
+        expect(events_written_count).to eq(event_count)
+      end
+    end
+
+    it "doesn't skip events if using the time_file option", :tag => :slow do
+      Stud::Temporary.directory do |temporary_directory|
+        time_file = rand(1..2)
+        number_of_rotation = rand(2..5)
+
+        config = {
+          "time_file" => time_file,
+          "codec" => "line",
+          "temporary_directory" => temporary_directory,
+          "pool" => "#{ENV['RADOS_LOGSTASH_TEST_POOL']}"
+        }
+
+        rados = LogStash::Outputs::Rados.new(minimal_settings.merge(config))
+        # Make the test run in seconds intead of minutes..
+        expect(rados).to receive(:periodic_interval).and_return(time_file)
+        rados.register
+
+        # Force to have a few files rotation
+        stop_time = Time.now + (number_of_rotation * time_file)
+        event_count = 0
+
+        event = LogStash::Event.new("message" => "Hello World")
+
+        until Time.now > stop_time do
+          rados.receive(event)
+          event_count += 1
+        end
+        rados.close
+
+        generated_files = Dir[File.join(temporary_directory, 'ls.*.txt')]
+
+        events_written_count = events_in_files(generated_files)
+
+        # Skew times can affect the number of rotation..
+        expect(generated_files.count).to be_within(number_of_rotation).of(number_of_rotation + 1)
+        expect(events_written_count).to eq(event_count)
+      end
+    end
+  end
+end

data/spec/supports/helpers.rb

@@ -0,0 +1,11 @@
+def key_exists_on_pool?(key)
+   pool = rados_object.pool(minimal_settings["pool"])
+   pool.open
+   obj = pool.rados_object(key)
+   obj.exists?
+end
+
+def events_in_files(files)
+  files.collect { |file| File.foreach(file).count }.inject(&:+)
+end
+

metadata

@@ -0,0 +1,150 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-rados
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Deathowl
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2016-01-29 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  name: stud
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.1'
+  name: ceph-ruby
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.1'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-input-generator
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-line
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email: bagoly@endticket.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/logstash/outputs/rados.rb
+- logstash-output-rados.gemspec
+- spec/integration/rados_spec.rb
+- spec/outputs/rados_spec.rb
+- spec/supports/helpers.rb
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.5
+signing_key:
+specification_version: 4
+summary: This plugin was created for store the logstash's events into  Ceph's Rados
+test_files:
+- spec/integration/rados_spec.rb
+- spec/outputs/rados_spec.rb
+- spec/supports/helpers.rb