logstash-integration-zeromq 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b71c26b1bd4ecbc2b7cfc0d6eb5ad13fb23adf09
+  data.tar.gz: 2faecee1a1c09daadf0bfb069497d3f1bcc9ddf6
+SHA512:
+  metadata.gz: 7457d6155c7defe32bb244ecd03f51d3ac325929fec0d2d57b0f2136db040755d45b8c7320d85b7613e843c257ccbbfeec2b9cbf05810333a3fb6a05f223059d
+  data.tar.gz: d5f1a035835c50e363a26a8e83955ada5271e7650f22615afe490180c5d2510548d7430d68bd31386d0e4df0e5a9928dfac18e7dae388e80cefbbe3539f6d8f9

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Joao Duarte - jsvduarte@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-filter-zeromq
+Example filter plugin. This should help bootstrap your effort to write your own filter plugin!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,93 @@
+NOTE: all code for the individual plugins was taken from:
+
+* input - https://github.com/logstash-plugins/logstash-input-zeromq
+* filter - https://github.com/logstash-plugins/logstash-filter-zeromq
+* output - https://github.com/logstash-plugins/logstash-output-zeromq
+* mixin - https://github.com/logstash-plugins/logstash-mixin-zeromq
+
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/filters/zeromq.rb

@@ -0,0 +1,217 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "logstash/json"
+
+# ZeroMQ filter. This is the best way to send an event externally for filtering
+# It works much like an exec filter would by sending the event "offsite"
+# for processing and waiting for a response
+#
+# The protocol here is:
+#   * REQ sent with JSON-serialized logstash event
+#   * REP read expected to be the full JSON 'filtered' event
+#   * - if reply read is an empty string, it will cancel the event.
+#
+# Note that this is a limited subset of the zeromq functionality in
+# inputs and outputs. The only topology that makes sense here is:
+# REQ/REP. bunde
+class LogStash::Filters::ZeroMQ < LogStash::Filters::Base
+
+  config_name "zeromq"
+
+  # 0mq socket address to connect or bind
+  # Please note that inproc:// will not work with logstash
+  # as we use a context per thread
+  # By default, filters connect
+  config :address, :validate => :string, :default => "tcp://127.0.0.1:2121"
+
+  # The field to send off-site for processing
+  # If this is unset, the whole event will be sent
+  config :field, :validate => :string
+
+  # A sentinel value to signal the filter to cancel the event
+  # If the peer returns the sentinel value, the event will be cancelled
+  config :sentinel, :validate => :string, :default => ""
+
+  # 0mq mode
+  # server mode binds/listens
+  # client mode connects
+  config :mode, :validate => ["server", "client"], :default => "client"
+
+  # timeout in milliseconds on which to wait for a reply.
+  config :timeout, :validate => :number, :default => 500
+
+  # number of retries, used for both sending and receiving messages.
+  # for sending, retries should return instantly.
+  # for receiving, the total blocking time is up to retries X timeout,
+  # which by default is 3 X 500 = 1500ms
+  config :retries, :validate => :number, :default => 3
+
+  # tag to add if zeromq timeout expires before getting back an answer.
+  # If set to "" then no tag will be added.
+  config :add_tag_on_timeout, :validate => :string, :default => "zeromqtimeout"
+
+  # 0mq socket options
+  # This exposes zmq_setsockopt
+  # for advanced tuning
+  # see http://api.zeromq.org/2-1:zmq-setsockopt for details
+  #
+  # This is where you would set values like:
+  # ZMQ::HWM - high water mark
+  # ZMQ::IDENTITY - named queues
+  # ZMQ::SWAP_SIZE - space for disk overflow
+  # ZMQ::SUBSCRIBE - topic filters for pubsub
+  #
+  # example: sockopt => ["ZMQ::HWM", 50, "ZMQ::IDENTITY", "my_named_queue"]
+  config :sockopt, :validate => :hash
+
+  public
+  def initialize(params)
+    super(params)
+
+    @threadsafe = false
+  end
+
+  public
+  def register
+    require "ffi-rzmq"
+    require "logstash/zeromq_mixin"
+    self.class.send(:include, LogStash::PluginMixins::ZeroMQ)
+    connect
+  end #def register
+
+  public
+  def close
+    close
+    super()
+  end #def close
+
+  private
+  def close
+    @logger.debug("0mq: closing socket.")
+    @poller.deregister(@zsocket, ZMQ::POLLIN)
+    @zsocket.close
+  end #def close
+
+  private
+  def connect
+    @logger.debug("0mq: connecting socket")
+    @zsocket = context.socket(ZMQ::REQ)
+    error_check(@zsocket.setsockopt(ZMQ::LINGER, 0),
+                "while setting ZMQ::LINGER == 0)")
+    @poller = ZMQ::Poller.new
+    @poller.register(@zsocket, ZMQ::POLLIN)
+
+    if @sockopt
+      #TODO: should make sure that ZMQ::LINGER and ZMQ::POLLIN are not changed
+      setopts(@zsocket, @sockopt)
+    end
+
+    setup(@zsocket, @address)
+  end #def connect
+
+  private
+  def reconnect
+    close
+    connect
+  end #def reconnect
+
+  #send and receive data. message is assumed to be json
+  #will return a boolean for success, and a string containing one of several things:
+  #  - empty string: response from server
+  #  - updated string: response from server
+  #  - original message: could not send request or get response from server in time
+  private
+  def send_recv(message)
+    success = false
+    @retries.times do
+      @logger.debug? && @logger.debug("0mq: sending", :request => message)
+      rc = @zsocket.send_string(message)
+      if ZMQ::Util.resultcode_ok?(rc)
+        success = true
+        break
+      else
+        @logger.debug? && @logger.debug("0mq: error sending message (zmq_errno = #{ZMQ::Util.errno}, zmq_error_string = '#{ZMQ::Util.error_string}'")
+        reconnect
+      end #if resultcode
+    end #retries.times
+
+    #if we did not succeed log it and fail here.
+    if not success
+      @logger.warn("0mq: error sending message (zmq_errno = #{ZMQ::Util.errno}, zmq_error_string = '#{ZMQ::Util.error_string}'")
+      return success, message
+    end
+
+    #now get reply
+    reply = ''
+    success = false
+    @retries.times do
+      @logger.debug? && @logger.debug("0mq: polling for reply for #{@timeout}ms.")
+      #poll the socket. If > 0, something to read. If < 0, error. If zero, loop
+      num_readable = @poller.poll(@timeout)
+      if num_readable > 0
+        #something to read, do it.
+        rc = @zsocket.recv_string(reply)
+        @logger.debug? && @logger.debug("0mq: message received, checking error")
+        error_check(rc, "in recv_string")
+        success = true
+        break
+      elsif num_readable < 0
+        #error, reconnect
+        close
+        connect
+      end
+    end # @retries.times
+
+    #if we maxed out on number of retries, then set reply to message so that
+    #the event isn't cancelled. we want to carry on if the server is down.
+    if not success
+      @logger.warn("0mq: did not receive reply (zmq_errno = #{ZMQ::Util.errno}, zmq_error_string = '#{ZMQ::Util.error_string}'")
+      return success, message
+    end
+
+    return success, reply
+  end #def send_recv
+
+  public
+  def filter(event)
+    begin
+      # TODO (lusis): Allow filtering multiple fields
+      if @field
+        success, reply = send_recv(event.get(@field))
+      else
+        success, reply = send_recv(event.to_json)
+      end
+      # If we receive an sentinel value as reply, this is an indication that the filter
+      # wishes to cancel this event.
+      if success && reply == @sentinel
+        @logger.debug? && @logger.debug("0mq: recieved sentinel value, cancelling event.")
+        event.cancel
+        return
+      end
+      @logger.debug? && @logger.debug("0mq: receiving", :reply => reply)
+      if @field
+        event.set(@field, event.sprintf(reply))
+        filter_matched(event)
+      else
+        reply = LogStash::Event.new(LogStash::Json.load(reply))
+        event.overwrite(reply)
+      end
+      filter_matched(event)
+      #if message send/recv was not successful add the timeout
+      if not success
+        tags = event.get("tags") || []
+        tags << @add_tag_on_timeout
+        event.set("tags", tags)
+      end
+    rescue => e
+      @logger.warn("0mq filter exception", :address => @address, :exception => e, :backtrace => e.backtrace)
+    end
+  end # def filter
+
+  private
+  def server?
+    @mode == "server"
+  end # def server?
+
+end

data/lib/logstash/inputs/zeromq.rb

@@ -0,0 +1,186 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+require "socket"
+
+# Read events over a 0MQ SUB socket.
+#
+# You need to have the 0mq 2.1.x library installed to be able to use
+# this input plugin.
+#
+# The default settings will create a subscriber binding to `tcp://127.0.0.1:2120`
+# waiting for connecting publishers.
+#
+class LogStash::Inputs::ZeroMQ < LogStash::Inputs::Base
+
+  config_name "zeromq"
+
+  default :codec, "json"
+
+  # 0mq socket address to connect or bind
+  # Please note that `inproc://` will not work with logstash
+  # as each we use a context per thread.
+  # By default, inputs bind/listen
+  # and outputs connect
+  config :address, :validate => :array, :default => ["tcp://*:2120"]
+
+  # 0mq topology
+  # The default logstash topologies work as follows:
+  #
+  # * pushpull - inputs are pull, outputs are push
+  # * pubsub - inputs are subscribers, outputs are publishers
+  # * pair - inputs are clients, inputs are servers
+  #
+  # If the predefined topology flows don't work for you,
+  # you can change the `mode` setting
+  # TODO (lusis) add req/rep MAYBE
+  # TODO (lusis) add router/dealer
+  config :topology, :validate => ["pushpull", "pubsub", "pair"], :required => true
+
+  # 0mq topic
+  # This is used for the `pubsub` topology only
+  # On inputs, this allows you to filter messages by topic
+  # On outputs, this allows you to tag a message for routing
+  # NOTE: ZeroMQ does subscriber side filtering.
+  # NOTE: All topics have an implicit wildcard at the end
+  # You can specify multiple topics here
+  config :topic, :validate => :array
+
+  # Event topic field
+  # This is used for the `pubsub` topology only
+  # When a message is received on a topic, the topic name on which
+  # the message was received will saved in this field.
+  config :topic_field, :validate => :string, :default => "topic"
+
+  # mode
+  # server mode binds/listens
+  # client mode connects
+  config :mode, :validate => ["server", "client"], :default => "server"
+
+  # sender
+  # overrides the sender to
+  # set the source of the event
+  # default is `zmq+topology://type/`
+  config :sender, :validate => :string
+
+  # 0mq socket options
+  # This exposes `zmq_setsockopt`
+  # for advanced tuning
+  # see http://api.zeromq.org/2-1:zmq-setsockopt for details
+  #
+  # This is where you would set values like:
+  #
+  #  * `ZMQ::HWM` - high water mark
+  #  * `ZMQ::IDENTITY` - named queues
+  #  * `ZMQ::SWAP_SIZE` - space for disk overflow
+  #
+  # Example:
+  # [source,ruby]
+  #     sockopt => {
+  #        "ZMQ::HWM" => 50
+  #        "ZMQ::IDENTITY"  => "my_named_queue"
+  #     }
+  #
+  # defaults to: `sockopt => { "ZMQ::RCVTIMEO" => "1000" }`, which has the effect of "interrupting"
+  # the recv operation at least once every second to allow for properly shutdown handling.
+  config :sockopt, :validate => :hash, :default => { "ZMQ::RCVTIMEO" => "1000" }
+
+  public
+  def register
+    require "ffi-rzmq"
+    require "logstash/zeromq_mixin"
+    self.class.send(:include, LogStash::PluginMixins::ZeroMQ)
+    @host = Socket.gethostname
+    init_socket
+  end # def register
+
+  def init_socket
+    case @topology
+    when "pair"
+      zmq_const = ZMQ::PAIR
+    when "pushpull"
+      zmq_const = ZMQ::PULL
+    when "pubsub"
+      zmq_const = ZMQ::SUB
+    end # case socket_type
+    @zsocket = context.socket(zmq_const)
+    error_check(@zsocket.setsockopt(ZMQ::LINGER, 1),
+                "while setting ZMQ::LINGER == 1)")
+
+    if @sockopt
+      setopts(@zsocket, @sockopt)
+    end
+
+    @address.each do |addr|
+      setup(@zsocket, addr)
+    end
+
+    if @topology == "pubsub"
+      if @topic.nil?
+        @logger.debug("ZMQ - No topic provided. Subscribing to all messages")
+        error_check(@zsocket.setsockopt(ZMQ::SUBSCRIBE, ""),
+      "while setting ZMQ::SUBSCRIBE")
+      else
+        @topic.each do |t|
+          @logger.debug("ZMQ subscribing to topic: #{t}")
+          error_check(@zsocket.setsockopt(ZMQ::SUBSCRIBE, t),
+        "while setting ZMQ::SUBSCRIBE == #{t}")
+        end
+      end
+    end
+  end
+
+  def close
+    begin
+      error_check(@zsocket.close, "while closing the zmq socket")
+      context.terminate
+    rescue RuntimeError => e
+      @logger.error("Failed to properly teardown ZeroMQ")
+    end
+  end # def close
+
+  def server?
+    @mode == "server"
+  end # def server?
+
+  def run(output_queue)
+    begin
+      while !stop?
+        handle_message(output_queue)
+      end
+    rescue => e
+      @logger.debug? && @logger.debug("ZMQ Error", :subscriber => @zsocket,
+                    :exception => e)
+      retry
+    end # begin
+  end # def run
+
+  private
+  def build_source_string
+    id = @address.first.clone
+  end
+
+  def handle_message(output_queue)
+    # Here's the unified receiver
+    more = true
+    parts = []
+    rc = @zsocket.recv_strings(parts)
+    error_check(rc, "in recv_strings", true)
+    return unless ZMQ::Util.resultcode_ok?(rc)
+
+    if @topology == "pubsub" && parts.length > 1
+      # assume topic is a simple string
+      topic, *parts = parts
+    else
+      topic = nil
+    end
+    parts.each do |msg|
+      @codec.decode(msg) do |event|
+        event.set("host", event.get("host") || @host)
+        event.set(@topic_field, topic.force_encoding('UTF-8')) unless topic.nil?
+        decorate(event)
+        output_queue << event
+      end
+    end
+  end
+end

data/lib/logstash/outputs/zeromq.rb

@@ -0,0 +1,135 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+
+# Write events to a 0MQ PUB socket.
+#
+# You need to have the 0mq 2.1.x library installed to be able to use
+# this output plugin.
+#
+# The default settings will create a publisher connecting to a subscriber
+# bound to tcp://127.0.0.1:2120
+#
+class LogStash::Outputs::ZeroMQ < LogStash::Outputs::Base
+
+  config_name "zeromq"
+
+  default :codec, "json"
+
+  # 0mq socket address to connect or bind.
+  # Please note that `inproc://` will not work with logstashi.
+  # For each we use a context per thread.
+  # By default, inputs bind/listen and outputs connect.
+  config :address, :validate => :array, :default => ["tcp://127.0.0.1:2120"]
+
+  # The default logstash topologies work as follows:
+  #
+  # * pushpull - inputs are pull, outputs are push
+  # * pubsub - inputs are subscribers, outputs are publishers
+  # * pair - inputs are clients, outputs are servers
+  #
+  # If the predefined topology flows don't work for you,
+  # you can change the 'mode' setting
+  config :topology, :validate => ["pushpull", "pubsub", "pair"], :required => true
+
+  # This is used for the 'pubsub' topology only.
+  # On inputs, this allows you to filter messages by topic.
+  # On outputs, this allows you to tag a message for routing.
+  # NOTE: ZeroMQ does subscriber-side filtering
+  # NOTE: Topic is evaluated with `event.sprintf` so macros are valid here.
+  config :topic, :validate => :string, :default => ""
+
+  # Server mode binds/listens. Client mode connects.
+  config :mode, :validate => ["server", "client"], :default => "client"
+
+  # This exposes zmq_setsockopt for advanced tuning.
+  # See http://api.zeromq.org/2-1:zmq-setsockopt for details.
+  #
+  # This is where you would set values like:
+  #
+  # * ZMQ::HWM - high water mark
+  # * ZMQ::IDENTITY - named queues
+  # * ZMQ::SWAP_SIZE - space for disk overflow
+  #
+  # Example:
+  # [source,ruby]
+  #     sockopt => {
+  #        "ZMQ::HWM" => 50
+  #        "ZMQ::IDENTITY"  => "my_named_queue"
+  #     }
+  config :sockopt, :validate => :hash
+
+  public
+  def register
+    load_zmq
+
+    if @mode == "server"
+      workers_not_supported("With 'mode => server', only one zeromq socket may bind to a port and may not be shared among threads. Going to single-worker mode for this plugin!")
+    end
+
+    connect
+
+    @codec.on_event(&method(:publish))
+  end # def register
+
+  public
+  def close
+    begin
+      error_check(@zsocket.close, "while closing the socket")
+    rescue RuntimeError => e
+      @logger.error("Failed to properly teardown ZeroMQ")
+    end
+  end # def close
+
+  def receive(event)
+    @codec.encode(event)
+  end
+
+  private
+  def server?
+    @mode == "server"
+  end # def server?
+
+  def publish(event, payload)
+    if @topology == "pubsub"
+      topic = event.sprintf(@topic)
+      error_check(@zsocket.send_string(topic, ZMQ::SNDMORE), "in topic send_string")
+    end
+    @logger.debug? && @logger.debug("0mq: sending", :event => payload)
+    error_check(@zsocket.send_string(payload), "in send_string")
+  rescue => e
+    warn e.inspect
+    @logger.warn("0mq output exception", :address => @address, :exception => e)
+  end
+
+  def load_zmq
+    require "ffi-rzmq"
+    require "logstash/zeromq_mixin"
+    self.class.send(:include, LogStash::PluginMixins::ZeroMQ)
+  end
+
+  def connect
+    # Translate topology shorthand to socket types
+    case @topology
+    when "pair"
+      zmq_const = ZMQ::PAIR
+    when "pushpull"
+      zmq_const = ZMQ::PUSH
+    when "pubsub"
+      zmq_const = ZMQ::PUB
+    end # case socket_type
+
+    @zsocket = context.socket(zmq_const)
+
+    error_check(@zsocket.setsockopt(ZMQ::LINGER, 1),
+                "while setting ZMQ::LINGER == 1)")
+
+    if @sockopt
+      setopts(@zsocket, @sockopt)
+    end
+
+    @address.each do |addr|
+      setup(@zsocket, addr)
+    end
+  end
+end

data/lib/logstash/zeromq_mixin.rb

@@ -0,0 +1,58 @@
+# encoding: utf-8
+require 'ffi-rzmq'
+require "logstash/namespace"
+
+class LogStash::PluginMixins::ZeroMQContext
+  class << self
+    def context
+      @context ||= ZMQ::Context.new
+    end
+  end
+end
+
+module LogStash::PluginMixins::ZeroMQ
+  # LOGSTASH-400
+  # see https://github.com/chuckremes/ffi-rzmq/blob/master/lib/ffi-rzmq/socket.rb#L93-117
+  STRING_OPTS = %w{IDENTITY SUBSCRIBE UNSUBSCRIBE}
+
+  def context
+    LogStash::PluginMixins::ZeroMQContext.context
+  end
+
+  def terminate_context
+    context.terminate
+  end
+
+  def setup(socket, address)
+    if server?
+      error_check(socket.bind(address), "binding to #{address}")
+    else
+      error_check(socket.connect(address), "connecting to #{address}")
+    end
+    @logger.info("0mq: #{server? ? 'bound' : 'connected'}", :address => address)
+  end
+
+  def error_check(rc, doing, eagain_not_error=false)
+    unless ZMQ::Util.resultcode_ok?(rc) || (ZMQ::Util.errno == ZMQ::EAGAIN && eagain_not_error)
+      @logger.error("ZeroMQ error while #{doing}", { :error_code => rc })
+      raise "ZeroMQ Error while #{doing}"
+    end
+  end # def error_check
+
+  def setopts(socket, options)
+    options.each do |opt,value|
+      sockopt = opt.split('::')[1]
+      option = ZMQ.const_defined?(sockopt) ? ZMQ.const_get(sockopt) : ZMQ.const_missing(sockopt)
+      unless STRING_OPTS.include?(sockopt)
+        begin
+          Float(value)
+          value = value.to_i
+        rescue ArgumentError
+          raise "#{sockopt} requires a numeric value. #{value} is not numeric"
+        end
+      end # end unless
+      error_check(socket.setsockopt(option, value),
+              "while setting #{opt} == #{value}")
+    end # end each
+  end # end setopts
+end # module LogStash::PluginMixins::ZeroMQ

data/lib/logstash_registry.rb

@@ -0,0 +1,9 @@
+# encoding: utf-8
+require "logstash/plugins/registry"
+require "logstash/inputs/zeromq"
+require "logstash/filters/zeromq"
+require "logstash/outputs/zeromq"
+
+LogStash::PLUGIN_REGISTRY.add(:input, "zeromq", LogStash::Inputs::ZeroMQ)
+LogStash::PLUGIN_REGISTRY.add(:filter, "zeromq", LogStash::Filters::ZeroMQ)
+LogStash::PLUGIN_REGISTRY.add(:output, "zeromq", LogStash::Outputs::ZeroMQ)

data/logstash-integration-zeromq.gemspec

@@ -0,0 +1,24 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-integration-zeromq'
+  s.version         = '0.1.1'
+  s.licenses      = ['Apache License (2.0)']
+  s.summary       = 'Logstash integration with ZeroMQ with input filter and output plugins'
+  s.homepage      = 'https://elastic.co'
+  s.authors       = ['Joao Duarte']
+  s.email         = 'jsvduarte@gmail.com'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "pack" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
+  s.add_runtime_dependency "ffi-rzmq", "~> 2.0"
+  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'logstash-codec-json'
+end

data/spec/filters/zeromq_spec.rb

@@ -0,0 +1,56 @@
+require "logstash/devutils/rspec/spec_helper"
+require 'logstash/filters/zeromq'
+require 'json'
+require 'securerandom'
+
+describe LogStash::Filters::ZeroMQ do
+  let(:sentinel) { SecureRandom.hex[1..2] }
+  let(:filter) { LogStash::Plugin.lookup("filter", "zeromq").new({"sentinel" => sentinel}) }
+  let(:event) do
+    LogStash::Event.new({
+      "message" => "some message",
+      "seq" => rand(1000),
+      "field1" => "some field value",
+      "@version" => "1",
+      "@timestamp" => Time.now.utc.iso8601(3)
+    })
+  end
+
+  let(:event_text) { event.to_json }
+
+  before do
+    allow(filter).to receive(:connect)
+  end
+
+  it "should send entire event as json if field not specified" do
+    expect(filter).to receive(:send_recv).with(event_text)
+    filter.filter(event)
+  end
+
+  it "should cancel event if peer returns an answer==sentinel" do
+    expect(filter).to receive(:send_recv).and_return([true, sentinel])
+    expect(event).to receive(:cancel)
+    filter.filter(event)
+  end
+
+  it "should not cancel event if send_recv failed" do
+    expect(filter).to receive(:send_recv).and_return([false, event_text])
+    expect(event).to_not receive(:cancel)
+    filter.filter(event)
+  end
+
+  context "Filter specific field" do
+    let(:filter) { LogStash::Plugin.lookup("filter", "zeromq").new({"field" => "field1"}) }
+
+    it "should send only a single field" do
+      expect(filter).to receive(:send_recv).with(event.get("field1"))
+      filter.filter(event)
+    end
+
+    it "should not cancel event if send_recv failed and event field==sentinel" do
+      expect(filter).to receive(:send_recv).and_return([false, sentinel])
+      expect(event).to_not receive(:cancel)
+      filter.filter(event)
+    end
+  end
+end

data/spec/inputs/zeromq_spec.rb

@@ -0,0 +1,82 @@
+# encoding: utf-8
+require_relative "../spec_helper"
+require "logstash/plugin"
+require "logstash/event"
+require "securerandom"
+
+def send_mock_messages(messages, &block)
+  socket = double("socket")
+  expect(socket).to receive(:recv_strings) do |arr|
+    messages.each do |msg|
+      msg.each do |frame|
+        arr << frame
+      end
+    end
+    0
+  end
+  plugin.instance_variable_set(:@zsocket, socket)
+  q = []
+  plugin.send(:handle_message, q)
+  q
+end
+
+describe LogStash::Inputs::ZeroMQ, :zeromq => true do
+
+  context "when register and close" do
+
+    let(:plugin) { LogStash::Plugin.lookup("input", "zeromq").new({ "topology" => "pushpull" }) }
+
+    it "should register and close without errors" do
+      expect { plugin.register }.to_not raise_error
+      expect { plugin.close }.to_not raise_error
+    end
+
+    context "when interrupting the plugin" do
+      it_behaves_like "an interruptible input plugin" do
+        let(:config) { { "topology" => "pushpull" } }
+        before do
+          subject.register
+        end
+        after do
+          subject.close
+        end
+      end
+    end
+
+  end
+
+  context "pubsub" do
+    topic_field = SecureRandom.hex
+    let(:plugin) { LogStash::Plugin.lookup("input", "zeromq").new({"topology" => "pubsub", "topic_field" => topic_field}) }
+
+    before do
+      allow(plugin).to receive(:init_socket)
+      plugin.register
+    end
+
+    it "should set the topic field with multiple message frames" do
+      events = send_mock_messages([["topic", '{"message": "message"}', '{"message": "message2"}']])
+      expect(events.first.get(topic_field)).to eq("topic")
+      expect(events.first.get("message")).to eq("message")
+      expect(events[1].get("message")).to eq("message2")
+      expect(events[1].get(topic_field)).to eq("topic")
+      expect(events.length).to eq(2)
+    end
+  end
+
+  context "pushpull" do
+    let(:plugin) { LogStash::Plugin.lookup("input", "zeromq").new({ "topology" => "pushpull" }) }
+
+    before do
+      allow(plugin).to receive(:init_socket)
+      plugin.register
+    end
+
+    it "should receive multiple frames" do
+      events = send_mock_messages([['{"message": "message"}', '{"message": "message2"}']])
+      expect(events.first.get("message")).to eq("message")
+      expect(events[1].get("message")).to eq("message2")
+      expect(events.length).to eq(2)
+    end
+  end
+end

data/spec/integration/input_spec.rb

@@ -0,0 +1,37 @@
+# encoding: utf-8
+require_relative "../spec_helper"
+require "logstash/plugin"
+require "logstash/event"
+require "json"
+
+describe LogStash::Inputs::ZeroMQ, :integration => true do
+
+  let(:helpers) { ZeroMQHelpers.new }
+
+  describe "receive events" do
+
+    let(:nevents)  { 10 }
+    let(:port)     { rand(1000)+1025 }
+
+    let(:conf) do
+      {  "address" => ["tcp://127.0.0.1:#{port}"],
+         "topology" => "pubsub" }
+    end
+
+    let(:events) do
+      helpers.input(conf, nevents) do
+        client = ZeroMQClient.new("127.0.0.1", port)
+        nevents.times do |value|
+          client.send("TOPIC", ZMQ::SNDMORE)
+          client.send({"message" => "data #{value}"}.to_json)
+        end
+        client.close
+      end
+    end
+
+    it "should receive the events" do
+      expect(events.count).to be(nevents)
+      expect(events.map(&:to_hash)).to all(include("topic" => "TOPIC"))
+    end
+  end
+end

data/spec/integration/output_spec.rb

@@ -0,0 +1,34 @@
+# encoding: utf-8
+require_relative "../spec_helper"
+require "logstash/plugin"
+require "logstash/event"
+
+describe LogStash::Outputs::ZeroMQ, :integration => true do
+
+  describe "send events" do
+
+    let(:nevents)  { 10 }
+    let(:port)     { rand(1000)+1025 }
+
+    let(:conf) do
+      {  "address" => ["tcp://127.0.0.1:#{port}"],
+         "topology" => "pushpull" }
+    end
+
+    let(:events) do
+      output(conf, nevents) do
+        res = []
+        client = ZeroMQServer.new("127.0.0.1", port)
+        nevents.times do
+          res << client.recv
+        end
+        client.close
+        res
+      end
+    end
+
+    it "should receive the events" do
+      expect(events.count).to be(nevents)
+    end
+  end
+end

data/spec/outputs/zeromq_spec.rb

@@ -0,0 +1,34 @@
+# encoding: utf-8
+require_relative "../spec_helper"
+require "logstash/plugin"
+require "logstash/event"
+
+describe LogStash::Outputs::ZeroMQ, :zeromq => true do
+
+  context "when register and close" do
+
+    let(:plugin) { LogStash::Plugin.lookup("output", "zeromq").new({ "topology" => "pushpull" }) }
+
+    it "should register and close without errors" do
+      expect { plugin.register }.to_not raise_error
+      expect { plugin.do_close }.to_not raise_error
+    end
+
+  end
+
+  context "pubsub" do
+    let(:plugin) { LogStash::Plugin.lookup("output", "zeromq").new({"topology" => "pubsub", "topic" => "%{topic}"})}
+
+    before do
+      plugin.send(:load_zmq)
+    end
+
+    it "should use topic field as topic" do
+      mock_socket = instance_spy("ZMQ::Socket", :send_string => 0)
+      plugin.instance_variable_set(:@zsocket, mock_socket)
+      plugin.send(:publish, LogStash::Event.new("topic" => "test-topic", "message" => "text"), "payload")
+      expect(mock_socket).to have_received(:send_string).with("test-topic", ZMQ::SNDMORE).once.ordered
+      expect(mock_socket).to have_received(:send_string).with("payload").ordered
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,32 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/zeromq"
+require "logstash/filters/zeromq"
+require "logstash/outputs/zeromq"
+require_relative "support/client"
+
+class ZeroMQHelpers
+
+  def input(config, size, &block)
+    plugin = LogStash::Plugin.lookup("input", "zeromq").new(config)
+    plugin.register
+    queue  = Queue.new
+
+    pipeline_thread = Thread.new { plugin.run(queue) }
+    sleep 0.3
+    block.call
+    sleep 0.1 while queue.size != size
+    result = size.times.inject([]) do |acc|
+      acc << queue.pop
+    end
+    plugin.do_stop
+    pipeline_thread.join
+    result
+  end # def input
+
+end
+
+RSpec.configure do |config|
+  # config.filter_run_excluding({ :zeromq => true, :integration => true })
+  config.order = :random
+end

data/spec/support/client.rb

@@ -0,0 +1,48 @@
+# encoding: utf-8
+require 'ffi-rzmq'
+
+class ZeroMQClient
+
+  attr_reader :addr, :port, :context, :requester
+
+  def initialize(addr, port)
+    @addr   = addr
+    @port   = port
+    @context = ZMQ::Context.new(1)
+    @requester = context.socket(ZMQ::PUB)
+    @requester.connect("tcp://#{addr}:#{port}")
+  end
+
+  def send(data, flags=0)
+    @requester.send_string(data, flags)
+  end
+
+  def close
+    @requester.close
+    @context.terminate
+  end
+end
+
+class ZeroMQServer
+
+  attr_reader :addr, :port, :context, :requester
+
+  def initialize(addr, port)
+    @addr   = addr
+    @port   = port
+    @context = ZMQ::Context.new(1)
+    @requester = context.socket(ZMQ::PULL)
+    @requester.bind("tcp://#{addr}:#{port}")
+  end
+
+  def recv
+    s = ''
+    @requester.recv_string s
+    s
+  end
+
+  def close
+    @requester.close
+    @context.terminate
+  end
+end

data/spec/zeromq_spec.rb

@@ -0,0 +1,37 @@
+# encoding: utf-8
+require_relative "spec_helper"
+require "logstash/plugin"
+require "logstash/event"
+require "json"
+
+describe LogStash::Inputs::ZeroMQ, :integration => true do
+
+  let(:helpers) { ZeroMQHelpers.new }
+
+  describe "receive events" do
+
+    let(:nevents)  { 10 }
+    let(:port)     { rand(1000)+1025 }
+
+    let(:conf) do
+      {  "address" => ["tcp://127.0.0.1:#{port}"],
+         "topology" => "pubsub" }
+    end
+
+    let(:events) do
+      helpers.input(conf, nevents) do
+        client = ZeroMQClient.new("127.0.0.1", port)
+        nevents.times do |value|
+          client.send("TOPIC", ZMQ::SNDMORE)
+          client.send({"message" => "data #{value}"}.to_json)
+        end
+        client.close
+      end
+    end
+
+    it "should receive the events" do
+      expect(events.count).to be(nevents)
+      expect(events.map(&:to_hash)).to all(include("topic" => "TOPIC"))
+    end
+  end
+end

metadata

@@ -0,0 +1,129 @@
+--- !ruby/object:Gem::Specification
+name: logstash-integration-zeromq
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Joao Duarte
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-06-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  name: ffi-rzmq
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-json
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description:
+email: jsvduarte@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/filters/zeromq.rb
+- lib/logstash/inputs/zeromq.rb
+- lib/logstash/outputs/zeromq.rb
+- lib/logstash/zeromq_mixin.rb
+- lib/logstash_registry.rb
+- logstash-integration-zeromq.gemspec
+- spec/filters/zeromq_spec.rb
+- spec/inputs/zeromq_spec.rb
+- spec/integration/input_spec.rb
+- spec/integration/output_spec.rb
+- spec/outputs/zeromq_spec.rb
+- spec/spec_helper.rb
+- spec/support/client.rb
+- spec/zeromq_spec.rb
+homepage: https://elastic.co
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: pack
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.8
+signing_key:
+specification_version: 4
+summary: Logstash integration with ZeroMQ with input filter and output plugins
+test_files:
+- spec/filters/zeromq_spec.rb
+- spec/inputs/zeromq_spec.rb
+- spec/integration/input_spec.rb
+- spec/integration/output_spec.rb
+- spec/outputs/zeromq_spec.rb
+- spec/spec_helper.rb
+- spec/support/client.rb
+- spec/zeromq_spec.rb