logstash-input-rabbitmq 0.1.0 → 3.0.0

checksums.yaml

@@ -1,15 +1,7 @@
 ---
-!binary "U0hBMQ==":
-  metadata.gz: !binary |-
-    ODdmYmIwZDg5ZGFiZmMxZGExNGE3MmM2M2RmODRhODg3MTJkNDhkMQ==
-  data.tar.gz: !binary |-
-    ZDlmNDIzZGY3Y2NmOWFlMTlhMzQ5ZDk5MjQxMTM0NTJmMDA2MDBlYw==
+SHA1:
+  metadata.gz: 63863bf744b2904baccd696f14968c741d858e30
+  data.tar.gz: 630d4a3870e93baa41363675c2f79a3a7467a866
 SHA512:
-  metadata.gz: !binary |-
-    OTU1NGViZTA0Y2NhNmVkNGZhZTE3ZDVkZmVkN2RlZmVhYzMyMjBkMjNlMzUy
-    OWVkZTAwMWMxZDdiNjAxZjJmMGNhNGMzN2E3MzQyZTdhNzhkYjhkMDFkZTlj
-    ZWM3MTQyOTU5NWZjYjE2M2JkZGIyMTBjNDBjYTM5Y2M5NGViOTY=
-  data.tar.gz: !binary |-
-    YzllMjI4NzNlZDJmMmI4MTk1OTM1N2UxNzVmNjhjYjNiNzQwNWE4ZmMwMDZk
-    YmM4NjUxN2MyMmExYWUwOWY1MDY3Yjk2MDllZWE3MjA3MzE5OGZhMTlmN2E2
-    NjRhNTdmYmYwOWJhYzZjMDM1MTkyNzVhMDQwNjliMTRjYWU3MWU=
+  metadata.gz: 502452f178e270279701933abb79859fcb65c28b49c6ff919104a3d7d549e4d0a4355f127b06d9f9c33fdf19c60621b45911d161390958a382a69b61e209b565
+  data.tar.gz: 6e9d1da9c818443681c60ea116bf1fbe6d20cbc7015ae7425935a8d9fac1006e0ede54ed40e49a1770b486ec1a7fb438b08636e704b92cd0849d718b77523b78

data/CHANGELOG.md

@@ -0,0 +1,9 @@
+* 2.0.0
+  - Massive refactor
+  - Implement Logstash 2.x stop behavior
+  - Fix reconnect issues
+  - Depend on rabbitmq_connection mixin for most connection functionality
+* 1.1.1
+  - Bump march hare to 2.12.0 which fixes jar perms on unices
+* 1.1.0
+  - Bump march hare version to 2.11.0

data/CONTRIBUTORS

@@ -0,0 +1,22 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Avishai Ish-Shalom (avishai-ish-shalom)
+* Jonathan Van Eenwyk (jdve)
+* Jordan Sissel (jordansissel)
+* João Duarte (jsvd)
+* Mathieu MILLET (htam-net)
+* Michael Klishin (michaelklishin)
+* Michael Zaccari (mzaccari)
+* Nick Ethier (nickethier)
+* Pier-Hugues Pellerin (ph)
+* Richard Pijnenburg (electrical)
+* Suyog Rao (suyograo)
+* Tim Potter (tpot)
+* avleen
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -1,4 +1,2 @@
-source 'http://rubygems.org'
-gem 'rake'
-gem 'gem_publisher'
-gem 'archive-tar-minitar'
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -1,4 +1,4 @@
-Copyright (c) 2012-2014 Elasticsearch <http://www.elasticsearch.org>
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/rabbitmq.rb

@@ -1,128 +1,111 @@
 # encoding: utf-8
-require "logstash/inputs/threadable"
-require "logstash/namespace"
-
-# Pull events from a RabbitMQ exchange.
-#
-# The default settings will create an entirely transient queue and listen for all messages by default.
-# If you need durability or any other advanced settings, please set the appropriate options
-#
-# This has been tested with Bunny 0.9.x, which supports RabbitMQ 2.x and 3.x. You can
-# find links to both here:
-#
-# * RabbitMQ - <http://www.rabbitmq.com/>
-# * March Hare: <http://rubymarchhare.info>
-# * Bunny - <https://github.com/ruby-amqp/bunny>
-class LogStash::Inputs::RabbitMQ < LogStash::Inputs::Threadable
-
-  config_name "rabbitmq"
-  milestone 1
-
-  #
-  # Connection
-  #
-
-  # RabbitMQ server address
-  config :host, :validate => :string, :required => true
-
-  # RabbitMQ port to connect on
-  config :port, :validate => :number, :default => 5672
-
-  # RabbitMQ username
-  config :user, :validate => :string, :default => "guest"
-
-  # RabbitMQ password
-  config :password, :validate => :password, :default => "guest"
-
-  # The vhost to use. If you don't know what this is, leave the default.
-  config :vhost, :validate => :string, :default => "/"
-
-  # Enable or disable SSL
-  config :ssl, :validate => :boolean, :default => false
-
-  # Validate SSL certificate
-  config :verify_ssl, :validate => :boolean, :default => false
-
-  # Enable or disable logging
-  config :debug, :validate => :boolean, :default => false, :deprecated => "Use the logstash --debug flag for this instead."
-
-
-
-  #
-  # Queue & Consumer
-  #
-
-  # The name of the queue Logstash will consume events from.
-  config :queue, :validate => :string, :default => ""
-
-  # Is this queue durable? (aka; Should it survive a broker restart?)
-  config :durable, :validate => :boolean, :default => false
-
-  # Should the queue be deleted on the broker when the last consumer
-  # disconnects? Set this option to 'false' if you want the queue to remain
-  # on the broker, queueing up messages until a consumer comes along to
-  # consume them.
-  config :auto_delete, :validate => :boolean, :default => false
-
-  # Is the queue exclusive? Exclusive queues can only be used by the connection
-  # that declared them and will be deleted when it is closed (e.g. due to a Logstash
-  # restart).
-  config :exclusive, :validate => :boolean, :default => false
-
-  # Extra queue arguments as an array.
-  # To make a RabbitMQ queue mirrored, use: {"x-ha-policy" => "all"}
-  config :arguments, :validate => :array, :default => {}
-
-  # Prefetch count. Number of messages to prefetch
-  config :prefetch_count, :validate => :number, :default => 256
-
-  # Enable message acknowledgement
-  config :ack, :validate => :boolean, :default => true
-
-  # Passive queue creation? Useful for checking queue existance without modifying server state
-  config :passive, :validate => :boolean, :default => false
-
-
-
-  #
-  # (Optional) Exchange binding
-  #
-
-  # Optional.
-  #
-  # The name of the exchange to bind the queue to.
-  config :exchange, :validate => :string
-
-  # Optional.
-  #
-  # The routing key to use when binding a queue to the exchange.
-  # This is only relevant for direct or topic exchanges.
-  #
-  # * Routing keys are ignored on fanout exchanges.
-  # * Wildcards are not valid on direct exchanges.
-  config :key, :validate => :string, :default => "logstash"
-
-
-  def initialize(params)
-    params["codec"] = "json" if !params["codec"]
-
-    super
-  end
-
-  # Use March Hare on JRuby to avoid IO#select CPU spikes
-  # (see github.com/ruby-amqp/bunny/issues/95).
-  #
-  # On MRI, use Bunny.
-  #
-  # See http://rubybunny.info and http://rubymarchhare.info
-  # for the docs.
-  if RUBY_ENGINE == "jruby"
-    require "logstash/inputs/rabbitmq/march_hare"
-
-    include MarchHareImpl
-  else
-    require "logstash/inputs/rabbitmq/bunny"
-
-    include BunnyImpl
+require 'logstash/plugin_mixins/rabbitmq_connection'
+require 'logstash/inputs/threadable'
+
+module LogStash
+  module Inputs
+    class RabbitMQ < LogStash::Inputs::Threadable
+      include ::LogStash::PluginMixins::RabbitMQConnection
+
+      config_name("rabbitmq")
+
+      # The name of the queue Logstash will consume events from.
+      config :queue, :validate => :string, :default => ""
+
+      # Is this queue durable? (aka; Should it survive a broker restart?)
+      config :durable, :validate => :boolean, :default => false
+
+      # Should the queue be deleted on the broker when the last consumer
+      # disconnects? Set this option to `false` if you want the queue to remain
+      # on the broker, queueing up messages until a consumer comes along to
+      # consume them.
+      config :auto_delete, :validate => :boolean, :default => false
+
+      # Is the queue exclusive? Exclusive queues can only be used by the connection
+      # that declared them and will be deleted when it is closed (e.g. due to a Logstash
+      # restart).
+      config :exclusive, :validate => :boolean, :default => false
+
+      # Extra queue arguments as an array.
+      # To make a RabbitMQ queue mirrored, use: `{"x-ha-policy" => "all"}`
+      config :arguments, :validate => :array, :default => {}
+
+      # Prefetch count. Number of messages to prefetch
+      config :prefetch_count, :validate => :number, :default => 256
+
+      # Enable message acknowledgement
+      config :ack, :validate => :boolean, :default => true
+
+      # Passive queue creation? Useful for checking queue existance without modifying server state
+      config :passive, :validate => :boolean, :default => false
+
+      # The name of the exchange to bind the queue to
+      config :exchange, :validate => :string
+
+      # The routing key to use when binding a queue to the exchange.
+      # This is only relevant for direct or topic exchanges.
+      #
+      # * Routing keys are ignored on fanout exchanges.
+      # * Wildcards are not valid on direct exchanges.
+      config :key, :validate => :string, :default => "logstash"
+
+
+      def register
+        connect!
+
+        declare_queue!
+        bind_exchange!
+      end
+
+      def run(output_queue)
+        @output_queue = output_queue
+        consume!
+      end
+
+      def bind_exchange!
+        if @exchange
+          @hare_info.queue.bind(@exchange, :routing_key => @key)
+        end
+      end
+
+      def declare_queue!
+        @hare_info.queue = declare_queue()
+      end
+
+      def declare_queue
+        @hare_info.channel.queue(@queue,
+                                 :durable     => @durable,
+                                 :auto_delete => @auto_delete,
+                                 :exclusive   => @exclusive,
+                                 :passive     => @passive,
+                                 :arguments   => @arguments)
+      end
+
+      def consume!
+        # we manually build a consumer here to be able to keep a reference to it
+        # in an @ivar even though we use a blocking version of HB::Queue#subscribe
+        @consumer = @hare_info.queue.build_consumer(:block => true) do |metadata, data|
+          @codec.decode(data) do |event|
+            decorate(event)
+            @output_queue << event if event
+          end
+          @hare_info.channel.ack(metadata.delivery_tag) if @ack
+        end
+
+        @hare_info.queue.subscribe_with(@consumer, :manual_ack => @ack, :block => true)
+      end
+
+      def stop
+        super
+        shutdown_consumer
+        close_connection
+      end
+
+      def shutdown_consumer
+        return unless @consumer
+        @consumer.gracefully_shut_down
+      end
+
+    end
   end
-end # class LogStash::Inputs::RabbitMQ
+end

data/logstash-input-rabbitmq.gemspec

@@ -1,34 +1,30 @@
 Gem::Specification.new do |s|
-
   s.name            = 'logstash-input-rabbitmq'
-  s.version         = '0.1.0'
+  s.version         = '3.0.0'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "Pull events from a RabbitMQ exchange."
-  s.description     = "Pull events from a RabbitMQ exchange."
-  s.authors         = ["Elasticsearch"]
-  s.email           = 'richard.pijnenburg@elasticsearch.com'
-  s.homepage        = "http://logstash.net/"
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Elastic"]
+  s.email           = 'info@elastic.co'
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
   s.require_paths = ["lib"]
 
   # Files
-  s.files = `git ls-files`.split($\)+::Dir.glob('vendor/*')
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
 
   # Tests
   s.test_files = s.files.grep(%r{^(test|spec|features)/})
 
   # Special flag to let us know this is actually a logstash plugin
-  s.metadata = { "logstash_plugin" => "true", "group" => "input" }
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
 
   # Gem dependencies
-  s.add_runtime_dependency 'logstash', '>= 1.4.0', '< 2.0.0'
+  s.add_runtime_dependency "logstash-core", "~> 2.0.0.snapshot"
+  s.add_runtime_dependency "logstash-mixin-rabbitmq_connection", '>= 1.0.0', '< 2.0.0'
 
   s.add_runtime_dependency 'logstash-codec-json'
 
-  if RUBY_PLATFORM == 'java'
-    s.add_runtime_dependency 'march_hare', ['~> 2.5.1']
-  else
-    s.add_runtime_dependency 'bunny', ['>= 1.6.0']
-  end
-
+  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'logstash-codec-plain'
 end
 

data/spec/inputs/rabbitmq_spec.rb

@@ -1 +1,132 @@
-require 'spec_helper'
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/rabbitmq"
+require "thread"
+
+Thread.abort_on_exception = true
+
+describe LogStash::Inputs::RabbitMQ do
+  let(:klass) { LogStash::Inputs::RabbitMQ }
+  let(:host) { "localhost" }
+  let(:port) { 5672 }
+  let(:exchange_type) { "topic" }
+  let(:exchange) { "myexchange" }
+  let(:queue) { "myqueue" }
+  let(:rabbitmq_settings) {
+    {
+      "host" => host,
+      "port" => port,
+      "queue" => queue
+    }
+  }
+  let(:instance) { klass.new(rabbitmq_settings) }
+  let(:hare_info) { instance.instance_variable_get(:@hare_info) }
+
+  context "when connected" do
+    let(:connection) { double("MarchHare Connection") }
+    let(:channel) { double("Channel") }
+    let(:exchange) { double("Exchange") }
+    let(:channel) { double("Channel") }
+    let(:queue) { double("queue") }
+
+    before do
+      allow(instance).to receive(:connect!).and_call_original
+      allow(::MarchHare).to receive(:connect).and_return(connection)
+      allow(connection).to receive(:create_channel).and_return(channel)
+      allow(connection).to receive(:on_blocked)
+      allow(connection).to receive(:on_unblocked)
+      allow(channel).to receive(:exchange).and_return(exchange)
+      allow(channel).to receive(:queue).and_return(queue)
+
+      allow(queue).to receive(:build_consumer).with(:block => true)
+      allow(queue).to receive(:subscribe_with).with(any_args)
+
+      instance.register
+    end
+
+    describe "#connect!" do
+      subject { hare_info }
+
+      it "should set the queue correctly" do
+        expect(subject.queue).to eql(queue)
+      end
+
+      context "with an exchange declared" do
+        let(:instance) { rabbitmq_settings.merge("exchange" => "myexchange", "exchange_type" => "fanout") }
+      end
+    end
+  end
+end
+
+describe "with a live server", :integration => true do
+  let(:klass) { LogStash::Inputs::RabbitMQ }
+  let(:config) { {"host" => "127.0.0.1"} }
+  let(:instance) { klass.new(config) }
+  let(:hare_info) { instance.instance_variable_get(:@hare_info) }
+  let(:output_queue) { Queue.new }
+
+  # Spawn a connection in the bg and wait up (n) seconds
+  def spawn_and_wait(instance)
+    instance.register
+
+    output_queue # materialize in this thread
+
+    Thread.new {
+      instance.run(output_queue)
+    }
+
+    20.times do
+      instance.connected? ? break : sleep(0.1)
+    end
+
+    # Extra time to make sure the consumer can attach
+    # Without this there's a chance the shutdown code will execute
+    # before consumption begins. This is tricky to do more elegantly
+    sleep 1
+  end
+
+  let(:test_connection) { MarchHare.connect(instance.send(:rabbitmq_settings)) }
+  let(:test_channel) { test_connection.create_channel }
+
+  before do
+    # Materialize the instance in the current thread to prevent dupes
+    # If you use multiple threads with lazy evaluation weird stuff happens
+    instance
+    spawn_and_wait(instance)
+
+    test_channel # Start up the test client as well
+  end
+
+  after do
+    instance.stop()
+    test_channel.close
+    test_connection.close
+  end
+
+  context "using defaults" do
+    it "should start, connect, and stop cleanly" do
+      expect(instance.connected?).to be_truthy
+    end
+  end
+
+  describe "receiving a message with a queue specified" do
+    let(:queue_name) { "foo_queue" }
+    let(:config) { super.merge("queue" => queue_name) }
+
+    it "should process the message" do
+      message = "Foo Message"
+      q = test_channel.queue(queue_name)
+      q.publish(message)
+
+      event = output_queue.pop
+      expect(event["message"]).to eql(message)
+    end
+  end
+
+  describe LogStash::Inputs::RabbitMQ do
+    let(:config) { super.merge("queue" => "foo_queue") }
+    it_behaves_like "an interruptible input plugin" do
+
+    end
+  end
+end

metadata

@@ -1,105 +1,130 @@
 --- !ruby/object:Gem::Specification
 name: logstash-input-rabbitmq
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 3.0.0
 platform: ruby
 authors:
-- Elasticsearch
-autorequire: 
+- Elastic
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2014-11-05 00:00:00.000000000 Z
+date: 2015-09-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: logstash
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 1.4.0
+        version: 2.0.0.snapshot
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 2.0.0.snapshot
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 1.0.0
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
-  type: :runtime
+  name: logstash-mixin-rabbitmq_connection
   prerelease: false
+  type: :runtime
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 1.4.0
+        version: 1.0.0
     - - <
       - !ruby/object:Gem::Version
         version: 2.0.0
 - !ruby/object:Gem::Dependency
-  name: logstash-codec-json
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
+  name: logstash-codec-json
+  prerelease: false
   type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
   prerelease: false
+  type: :development
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
-  name: bunny
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 1.6.0
-  type: :runtime
+        version: '0'
+  name: logstash-codec-plain
   prerelease: false
+  type: :development
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ! '>='
+    - - '>='
       - !ruby/object:Gem::Version
-        version: 1.6.0
-description: Pull events from a RabbitMQ exchange.
-email: richard.pijnenburg@elasticsearch.com
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email: info@elastic.co
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
+- CHANGELOG.md
+- CONTRIBUTORS
 - Gemfile
 - LICENSE
-- Rakefile
+- NOTICE.TXT
+- README.md
 - lib/logstash/inputs/rabbitmq.rb
-- lib/logstash/inputs/rabbitmq/bunny.rb
-- lib/logstash/inputs/rabbitmq/hot_bunnies.rb
-- lib/logstash/inputs/rabbitmq/march_hare.rb
 - logstash-input-rabbitmq.gemspec
-- rakelib/publish.rake
-- rakelib/vendor.rake
 - spec/inputs/rabbitmq_spec.rb
-homepage: http://logstash.net/
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
 licenses:
 - Apache License (2.0)
 metadata:
   logstash_plugin: 'true'
-  group: input
-post_install_message: 
+  logstash_group: input
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ! '>='
+  - - '>='
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.4.1
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
 specification_version: 4
 summary: Pull events from a RabbitMQ exchange.
 test_files:

data/.gitignore

@@ -1,4 +0,0 @@
-*.gem
-Gemfile.lock
-.bundle
-vendor

data/Rakefile

@@ -1,6 +0,0 @@
-@files=[]
-
-task :default do
-  system("rake -T")
-end
-

data/lib/logstash/inputs/rabbitmq/bunny.rb

@@ -1,118 +0,0 @@
-# encoding: utf-8
-class LogStash::Inputs::RabbitMQ
-  module BunnyImpl
-    def register
-      require "bunny"
-
-      @vhost       ||= Bunny::DEFAULT_HOST
-      # 5672. Will be switched to 5671 by Bunny if TLS is enabled.
-      @port        ||= AMQ::Protocol::DEFAULT_PORT
-      @routing_key ||= "#"
-
-      @settings = {
-        :vhost => @vhost,
-        :host  => @host,
-        :port  => @port,
-        :automatically_recover => false
-      }
-      @settings[:user]      = @user || Bunny::DEFAULT_USER
-      @settings[:pass]      = if @password
-                                @password.value
-                              else
-                                Bunny::DEFAULT_PASSWORD
-                              end
-
-      @settings[:log_level] = if @debug || @logger.debug?
-                                :debug
-                              else
-                                :error
-                              end
-
-      @settings[:tls]        = @ssl if @ssl
-      @settings[:verify_ssl] = @verify_ssl if @verify_ssl
-
-      proto                  = if @ssl
-                                 "amqps"
-                               else
-                                 "amqp"
-                               end
-      @connection_url        = "#{proto}://#{@user}@#{@host}:#{@port}#{vhost}/#{@queue}"
-
-      @logger.info("Registering input #{@connection_url}")
-    end
-
-    def run(output_queue)
-      @output_queue = output_queue
-
-      begin
-        setup
-        consume
-      rescue Bunny::NetworkFailure, Bunny::ConnectionClosedError, Bunny::ConnectionLevelException, Bunny::TCPConnectionFailed => e
-        n = Bunny::Session::DEFAULT_NETWORK_RECOVERY_INTERVAL * 2
-
-        # Because we manually reconnect instead of letting Bunny
-        # handle failures,
-        # make sure we don't leave any consumer work pool
-        # threads behind. MK.
-        @ch.maybe_kill_consumer_work_pool!
-        @logger.error("RabbitMQ connection error: #{e.message}. Will attempt to reconnect in #{n} seconds...")
-
-        sleep n
-        retry
-      end
-    end
-
-    def teardown
-      @consumer.cancel
-
-      @ch.close   if @ch && @ch.open?
-      @conn.close if @conn && @conn.open?
-
-      finished
-    end
-
-    def setup
-      @conn = Bunny.new(@settings)
-
-      @logger.debug("Connecting to RabbitMQ. Settings: #{@settings.inspect}, queue: #{@queue.inspect}")
-      return if terminating?
-      @conn.start
-
-      @ch = @conn.create_channel.tap do |ch|
-        ch.prefetch(@prefetch_count)
-      end
-      @logger.info("Connected to RabbitMQ at #{@settings[:host]}")
-
-      @arguments_hash = Hash[*@arguments]
-
-      @q = @ch.queue(@queue,
-                     :durable     => @durable,
-                     :auto_delete => @auto_delete,
-                     :exclusive   => @exclusive,
-                     :passive     => @passive,
-                     :arguments   => @arguments)
-
-      # exchange binding is optional for the input
-      if @exchange
-        @q.bind(@exchange, :routing_key => @key)
-      end
-    end
-
-    def consume
-      @logger.info("Will consume events from queue #{@q.name}")
-
-      # we both need to block the caller in Bunny::Queue#subscribe and have
-      # a reference to the consumer so that we can cancel it, so
-      # a consumer manually. MK.
-      @consumer = Bunny::Consumer.new(@ch, @q)
-      @q.subscribe(:manual_ack => @ack, :block => true) do |delivery_info, properties, data|
-        @codec.decode(data) do |event|
-          decorate(event)
-          @output_queue << event
-        end
-
-        @ch.acknowledge(delivery_info.delivery_tag) if @ack
-      end
-    end
-  end # BunnyImpl
-end

data/lib/logstash/inputs/rabbitmq/hot_bunnies.rb

@@ -1 +0,0 @@
-require "logstash/inputs/rabbitmq/march_hare"

data/lib/logstash/inputs/rabbitmq/march_hare.rb

@@ -1,129 +0,0 @@
-# encoding: utf-8
-class LogStash::Inputs::RabbitMQ
-  # MarchHare-based implementation for JRuby
-  module MarchHareImpl
-    def register
-      require "hot_bunnies"
-      require "java"
-
-      @vhost       ||= "127.0.0.1"
-      # 5672. Will be switched to 5671 by Bunny if TLS is enabled.
-      @port        ||= 5672
-      @key         ||= "#"
-
-      @settings = {
-        :vhost => @vhost,
-        :host  => @host,
-        :port  => @port,
-        :user  => @user,
-        :automatic_recovery => false
-      }
-      @settings[:pass]      = @password.value if @password
-      @settings[:tls]       = @ssl if @ssl
-
-      proto                 = if @ssl
-                                "amqps"
-                              else
-                                "amqp"
-                              end
-      @connection_url       = "#{proto}://#{@user}@#{@host}:#{@port}#{vhost}/#{@queue}"
-
-      @logger.info("Registering input #{@connection_url}")
-    end
-
-    def run(output_queue)
-      @output_queue          = output_queue
-      @break_out_of_the_loop = java.util.concurrent.atomic.AtomicBoolean.new(false)
-
-      # MarchHare does not raise exceptions when connection goes down with a blocking
-      # consumer running (it uses callbacks, as the RabbitMQ Java client does).
-      #
-      # However, MarchHare::Channel will make sure to unblock all blocking consumers
-      # on any internal shutdown, so #consume will return and another loop iteration
-      # will run.
-      #
-      # This is very similar to how the Bunny implementation works and is sufficient
-      # for our needs: it recovers successfully after RabbitMQ is kill -9ed, the
-      # network device is shut down, etc. MK.
-      until @break_out_of_the_loop.get do
-        begin
-          setup
-          consume
-        rescue MarchHare::Exception, java.lang.Throwable, com.rabbitmq.client.AlreadyClosedException => e
-          n = 10
-          @logger.error("RabbitMQ connection error: #{e}. Will reconnect in #{n} seconds...")
-
-          sleep n
-          retry
-        rescue LogStash::ShutdownSignal => ss
-          shutdown_consumer
-        end
-
-        n = 10
-        @logger.error("RabbitMQ connection error: #{e}. Will reconnect in #{n} seconds...")
-      end
-    end
-
-    def teardown
-      shutdown_consumer
-
-      @ch.close         if @ch && @ch.open?
-      @connection.close if @connection && @connection.open?
-
-      finished
-    end
-
-    #
-    # Implementation
-    #
-
-    protected
-
-    def setup
-      return if terminating?
-
-      @conn = MarchHare.connect(@settings)
-      @logger.info("Connected to RabbitMQ #{@connection_url}")
-
-      @ch          = @conn.create_channel.tap do |ch|
-        ch.prefetch = @prefetch_count
-      end
-
-      @arguments_hash = Hash[*@arguments]
-
-      @q = @ch.queue(@queue,
-        :durable     => @durable,
-        :auto_delete => @auto_delete,
-        :exclusive   => @exclusive,
-        :passive     => @passive,
-        :arguments   => @arguments)
-
-      # exchange binding is optional for the input
-      if @exchange
-        @q.bind(@exchange, :routing_key => @key)
-      end
-    end
-
-    def consume
-      return if terminating?
-
-      # we manually build a consumer here to be able to keep a reference to it
-      # in an @ivar even though we use a blocking version of HB::Queue#subscribe
-      @consumer = @q.build_consumer(:block => true) do |metadata, data|
-        @codec.decode(data) do |event|
-          decorate(event)
-          @output_queue << event if event
-        end
-        @ch.ack(metadata.delivery_tag) if @ack
-      end
-      @q.subscribe_with(@consumer, :manual_ack => @ack, :block => true)
-    end
-
-    def shutdown_consumer
-      @break_out_of_the_loop.set(true)
-
-      @consumer.cancel
-      @consumer.gracefully_shut_down
-    end
-  end # MarchHareImpl
-end

data/rakelib/publish.rake

@@ -1,9 +0,0 @@
-require "gem_publisher"
-
-desc "Publish gem to RubyGems.org"
-task :publish_gem do |t|
-  gem_file = Dir.glob(File.expand_path('../*.gemspec',File.dirname(__FILE__))).first
-  gem = GemPublisher.publish_if_updated(gem_file, :rubygems)
-  puts "Published #{gem}" if gem
-end
-

data/rakelib/vendor.rake

@@ -1,169 +0,0 @@
-require "net/http"
-require "uri"
-require "digest/sha1"
-
-def vendor(*args)
-  return File.join("vendor", *args)
-end
-
-directory "vendor/" => ["vendor"] do |task, args|
-  mkdir task.name
-end
-
-def fetch(url, sha1, output)
-
-  puts "Downloading #{url}"
-  actual_sha1 = download(url, output)
-
-  if actual_sha1 != sha1
-    fail "SHA1 does not match (expected '#{sha1}' but got '#{actual_sha1}')"
-  end
-end # def fetch
-
-def file_fetch(url, sha1)
-  filename = File.basename( URI(url).path )
-  output = "vendor/#{filename}"
-  task output => [ "vendor/" ] do
-    begin
-      actual_sha1 = file_sha1(output)
-      if actual_sha1 != sha1
-        fetch(url, sha1, output)
-      end
-    rescue Errno::ENOENT
-      fetch(url, sha1, output)
-    end
-  end.invoke
-
-  return output
-end
-
-def file_sha1(path)
-  digest = Digest::SHA1.new
-  fd = File.new(path, "r")
-  while true
-    begin
-      digest << fd.sysread(16384)
-    rescue EOFError
-      break
-    end
-  end
-  return digest.hexdigest
-ensure
-  fd.close if fd
-end
-
-def download(url, output)
-  uri = URI(url)
-  digest = Digest::SHA1.new
-  tmp = "#{output}.tmp"
-  Net::HTTP.start(uri.host, uri.port, :use_ssl => (uri.scheme == "https")) do |http|
-    request = Net::HTTP::Get.new(uri.path)
-    http.request(request) do |response|
-      fail "HTTP fetch failed for #{url}. #{response}" if [200, 301].include?(response.code)
-      size = (response["content-length"].to_i || -1).to_f
-      count = 0
-      File.open(tmp, "w") do |fd|
-        response.read_body do |chunk|
-          fd.write(chunk)
-          digest << chunk
-          if size > 0 && $stdout.tty?
-            count += chunk.bytesize
-            $stdout.write(sprintf("\r%0.2f%%", count/size * 100))
-          end
-        end
-      end
-      $stdout.write("\r      \r") if $stdout.tty?
-    end
-  end
-
-  File.rename(tmp, output)
-
-  return digest.hexdigest
-rescue SocketError => e
-  puts "Failure while downloading #{url}: #{e}"
-  raise
-ensure
-  File.unlink(tmp) if File.exist?(tmp)
-end # def download
-
-def untar(tarball, &block)
-  require "archive/tar/minitar"
-  tgz = Zlib::GzipReader.new(File.open(tarball))
-  # Pull out typesdb
-  tar = Archive::Tar::Minitar::Input.open(tgz)
-  tar.each do |entry|
-    path = block.call(entry)
-    next if path.nil?
-    parent = File.dirname(path)
-    
-    mkdir_p parent unless File.directory?(parent)
-
-    # Skip this file if the output file is the same size
-    if entry.directory?
-      mkdir path unless File.directory?(path)
-    else
-      entry_mode = entry.instance_eval { @mode } & 0777
-      if File.exists?(path)
-        stat = File.stat(path)
-        # TODO(sissel): Submit a patch to archive-tar-minitar upstream to
-        # expose headers in the entry.
-        entry_size = entry.instance_eval { @size }
-        # If file sizes are same, skip writing.
-        next if stat.size == entry_size && (stat.mode & 0777) == entry_mode
-      end
-      puts "Extracting #{entry.full_name} from #{tarball} #{entry_mode.to_s(8)}"
-      File.open(path, "w") do |fd|
-        # eof? check lets us skip empty files. Necessary because the API provided by
-        # Archive::Tar::Minitar::Reader::EntryStream only mostly acts like an
-        # IO object. Something about empty files in this EntryStream causes
-        # IO.copy_stream to throw "can't convert nil into String" on JRuby
-        # TODO(sissel): File a bug about this.
-        while !entry.eof?
-          chunk = entry.read(16384)
-          fd.write(chunk)
-        end
-          #IO.copy_stream(entry, fd)
-      end
-      File.chmod(entry_mode, path)
-    end
-  end
-  tar.close
-  File.unlink(tarball) if File.file?(tarball)
-end # def untar
-
-def ungz(file)
-
-  outpath = file.gsub('.gz', '')
-  tgz = Zlib::GzipReader.new(File.open(file))
-  begin
-    File.open(outpath, "w") do |out|
-      IO::copy_stream(tgz, out)
-    end
-    File.unlink(file)
-  rescue
-    File.unlink(outpath) if File.file?(outpath)
-   raise
-  end
-  tgz.close
-end
-
-desc "Process any vendor files required for this plugin"
-task "vendor" do |task, args|
-
-  @files.each do |file| 
-    download = file_fetch(file['url'], file['sha1'])
-    if download =~ /.tar.gz/
-      prefix = download.gsub('.tar.gz', '').gsub('vendor/', '')
-      untar(download) do |entry|
-        if !file['files'].nil?
-          next unless file['files'].include?(entry.full_name.gsub(prefix, ''))
-          out = entry.full_name.split("/").last
-        end
-        File.join('vendor', out)
-      end
-    elsif download =~ /.gz/
-      ungz(download)
-    end
-  end
-
-end