logstash-output-zeromq 2.0.4 → 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 71feb458cc271212d00f8269ca2891dfca2d64d2
4
- data.tar.gz: 5af956b08a2e26c7cbdbf0c60fe5c71c543afee0
3
+ metadata.gz: f1408116f878e88c8710d7c0d2a9f3aed0da4995
4
+ data.tar.gz: 019a6cf56a05c3b58cabee42ca7dc2077f86f4c2
5
5
  SHA512:
6
- metadata.gz: b77a42adddf951abc6adf74cdc36d5cd98ed703d8715c79f043c3552e562dcfa12ba3101f49448deb8bffa14a639a4a50cd54848cf206cecffb38207b3794ebd
7
- data.tar.gz: f735b020d4ed6a225a57b4181d2b57a9bdccd37016ce5c49ab1261952278e8753c0827f4fafaf34169a33d7e5007504ca9dcc97a2f79e784706da20aa6bb5bcb
6
+ metadata.gz: 69e182757ddd7af8eca29da2e23b09d5438086917f0e3b8863c518d2c1991352c5d2fc13c053e8024d90387d4021eafd0c859fdbe3eafa4ce89cb8fd1ac9337f
7
+ data.tar.gz: 8faff7f09233a20b0b17a7b4d7b70db0e511a27d452431a61f6d4ef18961fef2fb0767ac2ed9556f67cea9881b7c1045cc97af4382e9541cd9fec47f66cb2296
@@ -1,9 +1,11 @@
1
+ # 2.1.0
2
+ - Depend on logstash-mixin-zeromq, remove lib/logstash/util/zeromq
3
+ - Use new plugin API
1
4
  # 2.0.4
2
5
  - Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
3
6
  # 2.0.3
4
7
  - New dependency requirements for logstash-core for the 5.0 release
5
8
  ## 2.0.0
6
- - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
9
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
7
10
  instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
8
11
  - Dependency on logstash-core update to 2.0
9
-
@@ -10,6 +10,7 @@ Contributors:
10
10
  * Nick Padilla (NickPadilla)
11
11
  * Pier-Hugues Pellerin (ph)
12
12
  * Richard Pijnenburg (electrical)
13
+ * Avishai Ish-Shalom (avishai-ish-shalom)
13
14
 
14
15
  Note: If you've sent us patches, bug reports, or otherwise contributed to
15
16
  Logstash, and you aren't on the list above and want to be, please let us know
data/LICENSE CHANGED
@@ -1,4 +1,4 @@
1
- Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
1
+ Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
2
2
 
3
3
  Licensed under the Apache License, Version 2.0 (the "License");
4
4
  you may not use this file except in compliance with the License.
data/README.md CHANGED
@@ -1,7 +1,6 @@
1
1
  # Logstash Plugin
2
2
 
3
- [![Build
4
- Status](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Outputs/job/logstash-plugin-output-zeromq-unit/badge/icon)](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Outputs/job/logstash-plugin-output-zeromq-unit/)
3
+ [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-output-zeromq.svg)](https://travis-ci.org/logstash-plugins/logstash-output-zeromq)
5
4
 
6
5
  This is a plugin for [Logstash](https://github.com/elastic/logstash).
7
6
 
@@ -56,7 +55,12 @@ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
56
55
  ```
57
56
  - Install plugin
58
57
  ```sh
58
+ # Logstash 2.3 and higher
59
+ bin/logstash-plugin install --no-verify
60
+
61
+ # Prior to Logstash 2.3
59
62
  bin/plugin install --no-verify
63
+
60
64
  ```
61
65
  - Run Logstash with your plugin
62
66
  ```sh
@@ -74,7 +78,12 @@ gem build logstash-filter-awesome.gemspec
74
78
  ```
75
79
  - Install the plugin from the Logstash home
76
80
  ```sh
77
- bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
81
+ # Logstash 2.3 and higher
82
+ bin/logstash-plugin install --no-verify
83
+
84
+ # Prior to Logstash 2.3
85
+ bin/plugin install --no-verify
86
+
78
87
  ```
79
88
  - Start Logstash and proceed to test the plugin
80
89
 
@@ -26,7 +26,7 @@ class LogStash::Outputs::ZeroMQ < LogStash::Outputs::Base
26
26
  #
27
27
  # * pushpull - inputs are pull, outputs are push
28
28
  # * pubsub - inputs are subscribers, outputs are publishers
29
- # * pair - inputs are clients, inputs are servers
29
+ # * pair - inputs are clients, outputs are servers
30
30
  #
31
31
  # If the predefined topology flows don't work for you,
32
32
  # you can change the 'mode' setting
@@ -61,36 +61,13 @@ class LogStash::Outputs::ZeroMQ < LogStash::Outputs::Base
61
61
 
62
62
  public
63
63
  def register
64
- require "ffi-rzmq"
65
- require "logstash/util/zeromq"
66
- self.class.send(:include, LogStash::Util::ZeroMQ)
64
+ load_zmq
67
65
 
68
66
  if @mode == "server"
69
67
  workers_not_supported("With 'mode => server', only one zeromq socket may bind to a port and may not be shared among threads. Going to single-worker mode for this plugin!")
70
68
  end
71
69
 
72
- # Translate topology shorthand to socket types
73
- case @topology
74
- when "pair"
75
- zmq_const = ZMQ::PAIR
76
- when "pushpull"
77
- zmq_const = ZMQ::PUSH
78
- when "pubsub"
79
- zmq_const = ZMQ::PUB
80
- end # case socket_type
81
-
82
- @zsocket = context.socket(zmq_const)
83
-
84
- error_check(@zsocket.setsockopt(ZMQ::LINGER, 1),
85
- "while setting ZMQ::LINGER == 1)")
86
-
87
- if @sockopt
88
- setopts(@zsocket, @sockopt)
89
- end
90
-
91
- @address.each do |addr|
92
- setup(@zsocket, addr)
93
- end
70
+ connect
94
71
 
95
72
  @codec.on_event(&method(:publish))
96
73
  end # def register
@@ -104,28 +81,55 @@ class LogStash::Outputs::ZeroMQ < LogStash::Outputs::Base
104
81
  end
105
82
  end # def close
106
83
 
84
+ def receive(event)
85
+ @codec.encode(event)
86
+ end
87
+
107
88
  private
108
89
  def server?
109
90
  @mode == "server"
110
91
  end # def server?
111
92
 
112
- public
113
- def receive(event)
114
-
115
-
116
- @codec.encode(event)
117
- end # def receive
118
-
119
93
  def publish(event, payload)
120
- @logger.debug? && @logger.debug("0mq: sending", :event => payload)
121
94
  if @topology == "pubsub"
122
- # TODO(sissel): Need to figure out how to fit this into the codecs system.
123
- #@logger.debug("0mq output: setting topic to: #{event.sprintf(@topic)}")
124
- #error_check(@zsocket.send_string(event.sprintf(@topic), ZMQ::SNDMORE),
125
- #"in topic send_string")
95
+ topic = event.sprintf(@topic)
96
+ error_check(@zsocket.send_string(topic, ZMQ::SNDMORE), "in topic send_string")
126
97
  end
98
+ @logger.debug? && @logger.debug("0mq: sending", :event => payload)
127
99
  error_check(@zsocket.send_string(payload), "in send_string")
128
100
  rescue => e
101
+ warn e.inspect
129
102
  @logger.warn("0mq output exception", :address => @address, :exception => e)
130
103
  end
104
+
105
+ def load_zmq
106
+ require "ffi-rzmq"
107
+ require "logstash/plugin_mixins/zeromq"
108
+ self.class.send(:include, LogStash::PluginMixins::ZeroMQ)
109
+ end
110
+
111
+ def connect
112
+ # Translate topology shorthand to socket types
113
+ case @topology
114
+ when "pair"
115
+ zmq_const = ZMQ::PAIR
116
+ when "pushpull"
117
+ zmq_const = ZMQ::PUSH
118
+ when "pubsub"
119
+ zmq_const = ZMQ::PUB
120
+ end # case socket_type
121
+
122
+ @zsocket = context.socket(zmq_const)
123
+
124
+ error_check(@zsocket.setsockopt(ZMQ::LINGER, 1),
125
+ "while setting ZMQ::LINGER == 1)")
126
+
127
+ if @sockopt
128
+ setopts(@zsocket, @sockopt)
129
+ end
130
+
131
+ @address.each do |addr|
132
+ setup(@zsocket, addr)
133
+ end
134
+ end
131
135
  end # class LogStash::Outputs::ZeroMQ
@@ -1,10 +1,10 @@
1
1
  Gem::Specification.new do |s|
2
2
 
3
3
  s.name = 'logstash-output-zeromq'
4
- s.version = '2.0.4'
4
+ s.version = '2.1.0'
5
5
  s.licenses = ['Apache License (2.0)']
6
- s.summary = "Write events to a 0MQ PUB socket."
7
- s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
6
+ s.summary = "Write events to a 0MQ socket."
7
+ s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
8
8
  s.authors = ["Elastic"]
9
9
  s.email = 'info@elastic.co'
10
10
  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
@@ -21,9 +21,10 @@ Gem::Specification.new do |s|
21
21
 
22
22
  # Gem dependencies
23
23
  s.add_runtime_dependency "logstash-core-plugin-api", "~> 1.0"
24
+ s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
24
25
 
25
26
  s.add_runtime_dependency 'logstash-codec-json'
26
- s.add_runtime_dependency 'ffi-rzmq', '~> 2.0.4'
27
+ s.add_runtime_dependency 'logstash-mixin-zeromq', '>= 2.0.0'
27
28
 
28
29
  s.add_development_dependency 'logstash-devutils'
29
30
  end
@@ -0,0 +1,34 @@
1
+ # encoding: utf-8
2
+ require_relative "../spec_helper"
3
+ require "logstash/plugin"
4
+ require "logstash/event"
5
+
6
+ describe LogStash::Outputs::ZeroMQ, :integration => true do
7
+
8
+ describe "send events" do
9
+
10
+ let(:nevents) { 10 }
11
+ let(:port) { rand(1000)+1025 }
12
+
13
+ let(:conf) do
14
+ { "address" => ["tcp://127.0.0.1:#{port}"],
15
+ "topology" => "pushpull" }
16
+ end
17
+
18
+ let(:events) do
19
+ output(conf, nevents) do
20
+ res = []
21
+ client = ZeroMQClient.new("127.0.0.1", port)
22
+ nevents.times do
23
+ res << client.recv
24
+ end
25
+ client.close
26
+ res
27
+ end
28
+ end
29
+
30
+ it "should receive the events" do
31
+ expect(events.count).to be(nevents)
32
+ end
33
+ end
34
+ end
@@ -1 +1,34 @@
1
- require "logstash/devutils/rspec/spec_helper"
1
+ # encoding: utf-8
2
+ require_relative "../spec_helper"
3
+ require "logstash/plugin"
4
+ require "logstash/event"
5
+
6
+ describe LogStash::Outputs::ZeroMQ, :zeromq => true do
7
+
8
+ context "when register and close" do
9
+
10
+ let(:plugin) { LogStash::Plugin.lookup("output", "zeromq").new({ "topology" => "pushpull" }) }
11
+
12
+ it "should register and close without errors" do
13
+ expect { plugin.register }.to_not raise_error
14
+ expect { plugin.do_close }.to_not raise_error
15
+ end
16
+
17
+ end
18
+
19
+ context "pubsub" do
20
+ let(:plugin) { LogStash::Plugin.lookup("output", "zeromq").new({"topology" => "pubsub", "topic" => "%{topic}"})}
21
+
22
+ before do
23
+ plugin.send(:load_zmq)
24
+ end
25
+
26
+ it "should use topic field as topic" do
27
+ mock_socket = instance_spy("ZMQ::Socket", :send_string => 0)
28
+ plugin.instance_variable_set(:@zsocket, mock_socket)
29
+ plugin.send(:publish, LogStash::Event.new("topic" => "test-topic", "message" => "text"), "payload")
30
+ expect(mock_socket).to have_received(:send_string).with("test-topic", ZMQ::SNDMORE).once.ordered
31
+ expect(mock_socket).to have_received(:send_string).with("payload").ordered
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,35 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/outputs/zeromq"
4
+ require_relative "support/client"
5
+
6
+ module ZeroMQHelpers
7
+
8
+ def setup_clients(number_of_clients, port)
9
+ number_of_clients.times.inject([]) do |clients|
10
+ clients << ZeroMQClient.new(localhost, port)
11
+ end
12
+ end
13
+
14
+ def output(config, size, &block)
15
+ plugin = LogStash::Plugin.lookup("output", "zeromq").new(config)
16
+ plugin.register
17
+ events = []
18
+ size.times do |n|
19
+ events << LogStash::Event.new({"message" => "data #{n}"})
20
+ end
21
+ pipeline_thread = Thread.new { plugin.multi_receive(events) }
22
+ sleep 0.3
23
+ result = block.call
24
+ plugin.close
25
+ pipeline_thread.join
26
+ result
27
+ end # def input
28
+
29
+ end
30
+
31
+ RSpec.configure do |config|
32
+ config.include ZeroMQHelpers
33
+ # config.filter_run_excluding({ :zeromq => true, :integration => true })
34
+ config.order = :random
35
+ end
@@ -0,0 +1,26 @@
1
+ # encoding: utf-8
2
+ require 'ffi-rzmq'
3
+
4
+ class ZeroMQClient
5
+
6
+ attr_reader :addr, :port, :context, :requester
7
+
8
+ def initialize(addr, port)
9
+ @addr = addr
10
+ @port = port
11
+ @context = ZMQ::Context.new(1)
12
+ @requester = context.socket(ZMQ::PULL)
13
+ @requester.bind("tcp://#{addr}:#{port}")
14
+ end
15
+
16
+ def recv
17
+ s = ''
18
+ @requester.recv_string s
19
+ s
20
+ end
21
+
22
+ def close
23
+ @requester.close
24
+ @context.terminate
25
+ end
26
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-zeromq
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.0.4
4
+ version: 2.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2016-03-24 00:00:00.000000000 Z
11
+ date: 2016-05-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -24,6 +24,26 @@ dependencies:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
26
  version: '1.0'
27
+ - !ruby/object:Gem::Dependency
28
+ requirement: !ruby/object:Gem::Requirement
29
+ requirements:
30
+ - - ">="
31
+ - !ruby/object:Gem::Version
32
+ version: 2.0.0.beta2
33
+ - - "<"
34
+ - !ruby/object:Gem::Version
35
+ version: 3.0.0
36
+ name: logstash-core
37
+ prerelease: false
38
+ type: :runtime
39
+ version_requirements: !ruby/object:Gem::Requirement
40
+ requirements:
41
+ - - ">="
42
+ - !ruby/object:Gem::Version
43
+ version: 2.0.0.beta2
44
+ - - "<"
45
+ - !ruby/object:Gem::Version
46
+ version: 3.0.0
27
47
  - !ruby/object:Gem::Dependency
28
48
  requirement: !ruby/object:Gem::Requirement
29
49
  requirements:
@@ -41,17 +61,17 @@ dependencies:
41
61
  - !ruby/object:Gem::Dependency
42
62
  requirement: !ruby/object:Gem::Requirement
43
63
  requirements:
44
- - - "~>"
64
+ - - ">="
45
65
  - !ruby/object:Gem::Version
46
- version: 2.0.4
47
- name: ffi-rzmq
66
+ version: 2.0.0
67
+ name: logstash-mixin-zeromq
48
68
  prerelease: false
49
69
  type: :runtime
50
70
  version_requirements: !ruby/object:Gem::Requirement
51
71
  requirements:
52
- - - "~>"
72
+ - - ">="
53
73
  - !ruby/object:Gem::Version
54
- version: 2.0.4
74
+ version: 2.0.0
55
75
  - !ruby/object:Gem::Dependency
56
76
  requirement: !ruby/object:Gem::Requirement
57
77
  requirements:
@@ -66,7 +86,7 @@ dependencies:
66
86
  - - ">="
67
87
  - !ruby/object:Gem::Version
68
88
  version: '0'
69
- description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
89
+ description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
70
90
  email: info@elastic.co
71
91
  executables: []
72
92
  extensions: []
@@ -79,9 +99,11 @@ files:
79
99
  - NOTICE.TXT
80
100
  - README.md
81
101
  - lib/logstash/outputs/zeromq.rb
82
- - lib/logstash/util/zeromq.rb
83
102
  - logstash-output-zeromq.gemspec
103
+ - spec/integration/zeromq_spec.rb
84
104
  - spec/outputs/zeromq_spec.rb
105
+ - spec/spec_helper.rb
106
+ - spec/support/client.rb
85
107
  homepage: http://www.elastic.co/guide/en/logstash/current/index.html
86
108
  licenses:
87
109
  - Apache License (2.0)
@@ -107,6 +129,9 @@ rubyforge_project:
107
129
  rubygems_version: 2.4.8
108
130
  signing_key:
109
131
  specification_version: 4
110
- summary: Write events to a 0MQ PUB socket.
132
+ summary: Write events to a 0MQ socket.
111
133
  test_files:
134
+ - spec/integration/zeromq_spec.rb
112
135
  - spec/outputs/zeromq_spec.rb
136
+ - spec/spec_helper.rb
137
+ - spec/support/client.rb
@@ -1,47 +0,0 @@
1
- # encoding: utf-8
2
- require 'ffi-rzmq'
3
- require "logstash/namespace"
4
-
5
- module LogStash::Util::ZeroMQ
6
- CONTEXT = ZMQ::Context.new
7
- # LOGSTASH-400
8
- # see https://github.com/chuckremes/ffi-rzmq/blob/master/lib/ffi-rzmq/socket.rb#L93-117
9
- STRING_OPTS = %w{IDENTITY SUBSCRIBE UNSUBSCRIBE}
10
-
11
- def context
12
- CONTEXT
13
- end
14
-
15
- def setup(socket, address)
16
- if server?
17
- error_check(socket.bind(address), "binding to #{address}")
18
- else
19
- error_check(socket.connect(address), "connecting to #{address}")
20
- end
21
- @logger.info("0mq: #{server? ? 'connected' : 'bound'}", :address => address)
22
- end
23
-
24
- def error_check(rc, doing)
25
- unless ZMQ::Util.resultcode_ok?(rc)
26
- @logger.error("ZeroMQ error while #{doing}", { :error_code => rc })
27
- raise "ZeroMQ Error while #{doing}"
28
- end
29
- end # def error_check
30
-
31
- def setopts(socket, options)
32
- options.each do |opt,value|
33
- sockopt = opt.split('::')[1]
34
- option = ZMQ.const_defined?(sockopt) ? ZMQ.const_get(sockopt) : ZMQ.const_missing(sockopt)
35
- unless STRING_OPTS.include?(sockopt)
36
- begin
37
- Float(value)
38
- value = value.to_i
39
- rescue ArgumentError
40
- raise "#{sockopt} requires a numeric value. #{value} is not numeric"
41
- end
42
- end # end unless
43
- error_check(socket.setsockopt(option, value),
44
- "while setting #{opt} == #{value}")
45
- end # end each
46
- end # end setopts
47
- end # module LogStash::Util::ZeroMQ