logstash-input-redis-cluster 2.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 1c753cc2def1e0233efa5a49a580159d4ba2aa9f
4
+ data.tar.gz: e47850e128dee3d7f6a379bd2aaf25cafdeed534
5
+ SHA512:
6
+ metadata.gz: f9a25fffed1c8b9f601e967546e17f81cb1e36d00869aa3ccc3fc1d4cc618cc749abd1eb88f97c95f1941893032b9139af36ad33feb611e12566d3579dcfc580
7
+ data.tar.gz: b0e9d2d6c56d4b80625d922b87140fb195ae8d5f4a7c1f5ff0537e31e420487d65475637859c8015a6ec67b990647d4ecc6cb513b75d68d0d8a4b00c9088fc7b
data/CHANGELOG.md ADDED
@@ -0,0 +1,14 @@
1
+ ## 2.0.0
2
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
3
+ instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
4
+ - Dependency on logstash-core update to 2.0
5
+
6
+ ## 1.0.3
7
+ - Fix typo in module name in test (Causing CI build failures)
8
+
9
+ ## 1.0.2
10
+ - Fix typo in module name (Causing the module to not be loaded)
11
+
12
+ ## 1.0.1
13
+ - Make teardown more reliable
14
+ - Re-organise code and tests
data/CONTRIBUTORS ADDED
@@ -0,0 +1,21 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Aaron Mildenstein (untergeek)
6
+ * Bob Corsaro (dokipen)
7
+ * Colin Surprenant (colinsurprenant)
8
+ * Graham Bleach (bleach)
9
+ * Jason Woods (driskell)
10
+ * John E. Vincent (lusis)
11
+ * Jordan Sissel (jordansissel)
12
+ * Kurt Hurtado (kurtado)
13
+ * Pete Fritchman (fetep)
14
+ * Pier-Hugues Pellerin (ph)
15
+ * Richard Pijnenburg (electrical)
16
+ * piavlo
17
+
18
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
19
+ Logstash, and you aren't on the list above and want to be, please let us know
20
+ and we'll make sure you're here. Contributions from folks like you are what make
21
+ open source awesome.
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source 'https://rubygems.org'
2
+ gemspec
3
+ gem "redis-rb-cluster", :git => "git://github.com/Tschef/redis-rb-cluster.git"
data/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
data/NOTICE.TXT ADDED
@@ -0,0 +1,5 @@
1
+ Elasticsearch
2
+ Copyright 2012-2015 Elasticsearch
3
+
4
+ This product includes software developed by The Apache Software
5
+ Foundation (http://www.apache.org/).
data/README.md ADDED
@@ -0,0 +1,86 @@
1
+ # Logstash Plugin
2
+
3
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
4
+
5
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
6
+
7
+ ## Documentation
8
+
9
+ Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
10
+
11
+ - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
12
+ - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
13
+
14
+ ## Need Help?
15
+
16
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
17
+
18
+ ## Developing
19
+
20
+ ### 1. Plugin Developement and Testing
21
+
22
+ #### Code
23
+ - To get started, you'll need JRuby with the Bundler gem installed.
24
+
25
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
26
+
27
+ - Install dependencies
28
+ ```sh
29
+ bundle install
30
+ ```
31
+
32
+ #### Test
33
+
34
+ - Update your dependencies
35
+
36
+ ```sh
37
+ bundle install
38
+ ```
39
+
40
+ - Run tests
41
+
42
+ ```sh
43
+ bundle exec rspec
44
+ ```
45
+
46
+ ### 2. Running your unpublished Plugin in Logstash
47
+
48
+ #### 2.1 Run in a local Logstash clone
49
+
50
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
51
+ ```ruby
52
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
53
+ ```
54
+ - Install plugin
55
+ ```sh
56
+ bin/plugin install --no-verify
57
+ ```
58
+ - Run Logstash with your plugin
59
+ ```sh
60
+ bin/logstash -e 'filter {awesome {}}'
61
+ ```
62
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
63
+
64
+ #### 2.2 Run in an installed Logstash
65
+
66
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
67
+
68
+ - Build your plugin gem
69
+ ```sh
70
+ gem build logstash-filter-awesome.gemspec
71
+ ```
72
+ - Install the plugin from the Logstash home
73
+ ```sh
74
+ bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
75
+ ```
76
+ - Start Logstash and proceed to test the plugin
77
+
78
+ ## Contributing
79
+
80
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
81
+
82
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
83
+
84
+ It is more important to the community that you are able to contribute.
85
+
86
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,330 @@
1
+ # encoding: utf-8
2
+ require "logstash/namespace"
3
+ require "logstash/inputs/base"
4
+ require "logstash/inputs/threadable"
5
+
6
+ # This input will read events from a Redis instance; it supports both Redis channels and lists.
7
+ # The list command (BLPOP) used by Logstash is supported in Redis v1.3.1+, and
8
+ # the channel commands used by Logstash are found in Redis v1.3.8+.
9
+ # While you may be able to make these Redis versions work, the best performance
10
+ # and stability will be found in more recent stable versions. Versions 2.6.0+
11
+ # are recommended.
12
+ #
13
+ # For more information about Redis, see <http://redis.io/>
14
+ #
15
+ # `batch_count` note: If you use the `batch_count` setting, you *must* use a Redis version 2.6.0 or
16
+ # newer. Anything older does not support the operations used by batching.
17
+ #
18
+ module LogStash module Inputs class RedisCluster < LogStash::Inputs::Threadable
19
+ # class LogStash::Inputs::Redis < LogStash::Inputs::Threadable
20
+
21
+ config_name "redis_cluster"
22
+
23
+ default :codec, "json"
24
+
25
+ # The `name` configuration is used for logging in case there are multiple instances.
26
+ # This feature has no real function and will be removed in future versions.
27
+ config :name, :validate => :string, :default => "default", :deprecated => true
28
+
29
+ # The hostname of your Redis server.
30
+ config :host, :validate => :string, :default => "127.0.0.1"
31
+
32
+ # The port to connect on.
33
+ config :port, :validate => :number, :default => 6379
34
+
35
+ # The Redis database number.
36
+ config :db, :validate => :number, :default => 0
37
+
38
+ # Initial connection timeout in seconds.
39
+ config :timeout, :validate => :number, :default => 5
40
+
41
+ # Password to authenticate with. There is no authentication by default.
42
+ config :password, :validate => :password
43
+
44
+ # The name of a Redis list or channel.
45
+ # TODO: change required to true
46
+ config :keys, :validate => :array
47
+
48
+ # Specify either list or channel. If `redis\_type` is `list`, then we will BLPOP the
49
+ # key. If `redis\_type` is `channel`, then we will SUBSCRIBE to the key.
50
+ # If `redis\_type` is `pattern_channel`, then we will PSUBSCRIBE to the key.
51
+ # TODO: change required to true
52
+ config :data_type, :validate => [ "list", "channel", "pattern_channel" ], :required => false
53
+
54
+ # The number of events to return from Redis using EVAL.
55
+ config :batch_count, :validate => :number, :default => 1
56
+
57
+ public
58
+ # public API
59
+ # use to store a proc that can provide a redis instance or mock
60
+ def add_external_redis_builder(builder) #callable
61
+ @redis_builder = builder
62
+ self
63
+ end
64
+
65
+ # use to apply an instance directly and bypass the builder
66
+ def use_redis(instance)
67
+ @redis = instance
68
+ self
69
+ end
70
+
71
+ def new_redis_instance
72
+ @redis_builder.call
73
+ end
74
+
75
+ def register
76
+ require 'redis-rb-cluster'
77
+ @redis_url = "redis://#{@password}@#{@host}:#{@port}/#{@db}"
78
+
79
+ # TODO remove after setting key and data_type to true
80
+
81
+ if !@keys || !@data_type
82
+ raise RuntimeError.new(
83
+ "Must define queue, or key and data_type parameters"
84
+ )
85
+ end
86
+ # end TODO
87
+
88
+ @redis_builder ||= method(:internal_redis_builder)
89
+
90
+ # just switch on data_type once
91
+ if @data_type == 'list' || @data_type == 'dummy'
92
+ @run_method = method(:list_runner)
93
+ @stop_method = method(:list_stop)
94
+ elsif @data_type == 'channel'
95
+ @run_method = method(:channel_runner)
96
+ @stop_method = method(:subscribe_stop)
97
+ elsif @data_type == 'pattern_channel'
98
+ @run_method = method(:pattern_channel_runner)
99
+ @stop_method = method(:subscribe_stop)
100
+ end
101
+
102
+ # TODO(sissel, boertje): set @identity directly when @name config option is removed.
103
+ @identity = @name != 'default' ? @name : "#{@redis_url} #{@data_type}"
104
+ @logger.info("Registering Redis", :identity => @identity)
105
+ end # def register
106
+
107
+ def run(output_queue)
108
+ @run_method.call(output_queue)
109
+ rescue LogStash::ShutdownSignal
110
+ # ignore and quit
111
+ end # def run
112
+
113
+ def stop
114
+ @stop_method.call
115
+ end
116
+
117
+ # private methods -----------------------------
118
+ private
119
+
120
+ def batched?
121
+ @batch_count > 1
122
+ end
123
+
124
+ # private
125
+ def is_list_type?
126
+ @data_type == 'list'
127
+ end
128
+
129
+ # private
130
+ def redis_params
131
+ {
132
+ :host => @host,
133
+ :port => @port,
134
+ :timeout => @timeout,
135
+ :db => @db,
136
+ :password => @password.nil? ? nil : @password.value
137
+ }
138
+ end
139
+
140
+ # private
141
+ def internal_redis_builder
142
+ ::RedisCluster.new([redis_params],4)
143
+ end
144
+
145
+ # private
146
+ def connect
147
+ redis = new_redis_instance
148
+ load_batch_script(redis) if batched? && is_list_type?
149
+ redis
150
+ end # def connect
151
+
152
+ # private
153
+ def load_batch_script(redis)
154
+ #A Redis Lua EVAL script to fetch a count of keys
155
+ #in case count is bigger than current items in queue whole queue will be returned without extra nil values
156
+ redis_script = <<EOF
157
+ local i = tonumber(ARGV[1])
158
+ local res = {}
159
+ local length = redis.call('llen',KEYS[1])
160
+ if length < i then i = length end
161
+ while (i > 0) do
162
+ local item = redis.call("lpop", KEYS[1])
163
+ if (not item) then
164
+ break
165
+ end
166
+ table.insert(res, item)
167
+ i = i-1
168
+ end
169
+ return res
170
+ EOF
171
+ @redis_script_sha = redis.script(:load, redis_script)
172
+ end
173
+
174
+ # private
175
+ def queue_event(msg, output_queue)
176
+ begin
177
+ @codec.decode(msg) do |event|
178
+ decorate(event)
179
+ output_queue << event
180
+ end
181
+ rescue => e # parse or event creation error
182
+ @logger.error("Failed to create event", :message => msg, :exception => e, :backtrace => e.backtrace);
183
+ end
184
+ end
185
+
186
+ # private
187
+ def list_stop
188
+ return if @redis.nil? || !@redis.connected?
189
+
190
+ @redis.quit rescue nil
191
+ @redis = nil
192
+ end
193
+
194
+ # private
195
+ def list_runner(output_queue)
196
+ while !stop?
197
+ begin
198
+ @redis ||= connect
199
+ list_listener(@redis, output_queue)
200
+ rescue ::Redis::BaseError => e
201
+ @logger.warn("Redis connection problem", :exception => e)
202
+ # Reset the redis variable to trigger reconnect
203
+ @redis = nil
204
+ # this sleep does not need to be stoppable as its
205
+ # in a while !stop? loop
206
+ sleep 1
207
+ end
208
+ end
209
+ end
210
+
211
+ # private
212
+ def list_listener(redis, output_queue)
213
+ sampled = @keys.sample
214
+ item = redis.blpop(sampled, :timeout => 1)
215
+ return unless item # from timeout or other conditions
216
+
217
+ # blpop returns the 'key' read from as well as the item result
218
+ # we only care about the result (2nd item in the list).
219
+ queue_event(item.last, output_queue)
220
+
221
+ # If @batch_count is 1, there's no need to continue.
222
+ return if !batched?
223
+
224
+ begin
225
+ redis.evalsha(@redis_script_sha, [sampled], [@batch_count-1]).each do |item|
226
+ queue_event(item, output_queue)
227
+ end
228
+
229
+ # Below is a commented-out implementation of 'batch fetch'
230
+ # using pipelined LPOP calls. This in practice has been observed to
231
+ # perform exactly the same in terms of event throughput as
232
+ # the evalsha method. Given that the EVALSHA implementation uses
233
+ # one call to Redis instead of N (where N == @batch_count) calls,
234
+ # I decided to go with the 'evalsha' method of fetching N items
235
+ # from Redis in bulk.
236
+ #redis.pipelined do
237
+ #error, item = redis.lpop(@key)
238
+ #(@batch_count-1).times { redis.lpop(@key) }
239
+ #end.each do |item|
240
+ #queue_event(item, output_queue) if item
241
+ #end
242
+ # --- End commented out implementation of 'batch fetch'
243
+ rescue ::Redis::CommandError => e
244
+ if e.to_s =~ /NOSCRIPT/ then
245
+ @logger.warn("Redis may have been restarted, reloading Redis batch EVAL script", :exception => e);
246
+ load_batch_script(redis)
247
+ retry
248
+ else
249
+ raise e
250
+ end
251
+ end
252
+ end
253
+
254
+ # private
255
+ def subscribe_stop
256
+ return if @redis.nil? || !@redis.connected?
257
+ # if its a SubscribedClient then:
258
+ # it does not have a disconnect method (yet)
259
+ if @redis.client.is_a?(::Redis::SubscribedClient)
260
+ @redis.client.unsubscribe
261
+ else
262
+ @redis.client.disconnect
263
+ end
264
+ @redis = nil
265
+ end
266
+
267
+ # private
268
+ def redis_runner
269
+ begin
270
+ @redis ||= connect
271
+ yield
272
+ rescue ::Redis::BaseError => e
273
+ @logger.warn("Redis connection problem", :exception => e)
274
+ # Reset the redis variable to trigger reconnect
275
+ @redis = nil
276
+ Stud.stoppable_sleep(1) { stop? }
277
+ retry if !stop?
278
+ end
279
+ end
280
+
281
+ # private
282
+ def channel_runner(output_queue)
283
+ redis_runner do
284
+ channel_listener(output_queue)
285
+ end
286
+ end
287
+
288
+ # private
289
+ def channel_listener(output_queue)
290
+ @redis.subscribe(@keys.sample) do |on|
291
+ on.subscribe do |channel, count|
292
+ @logger.info("Subscribed", :channel => channel, :count => count)
293
+ end
294
+
295
+ on.message do |channel, message|
296
+ queue_event(message, output_queue)
297
+ end
298
+
299
+ on.unsubscribe do |channel, count|
300
+ @logger.info("Unsubscribed", :channel => channel, :count => count)
301
+ end
302
+ end
303
+ end
304
+
305
+ def pattern_channel_runner(output_queue)
306
+ redis_runner do
307
+ pattern_channel_listener(output_queue)
308
+ end
309
+ end
310
+
311
+ # private
312
+ def pattern_channel_listener(output_queue)
313
+ @redis.psubscribe @keys.sample do |on|
314
+ on.psubscribe do |channel, count|
315
+ @logger.info("Subscribed", :channel => channel, :count => count)
316
+ end
317
+
318
+ on.pmessage do |pattern, channel, message|
319
+ queue_event(message, output_queue)
320
+ end
321
+
322
+ on.punsubscribe do |channel, count|
323
+ @logger.info("Unsubscribed", :channel => channel, :count => count)
324
+ end
325
+ end
326
+ end
327
+
328
+ # end
329
+
330
+ end end end # Redis Inputs LogStash
@@ -0,0 +1,30 @@
1
+ Gem::Specification.new do |s|
2
+
3
+ s.name = 'logstash-input-redis-cluster'
4
+ s.version = '2.0.2'
5
+ s.licenses = ['Apache License (2.0)']
6
+ s.summary = "This input will read events from a Redis instance"
7
+ s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
8
+ s.authors = ["Elastic"]
9
+ s.email = 'info@elastic.co'
10
+ s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
11
+ s.require_paths = ["lib"]
12
+
13
+ # Files
14
+ s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
15
+
16
+ # Tests
17
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
18
+
19
+ # Special flag to let us know this is actually a logstash plugin
20
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
21
+
22
+ # Gem dependencies
23
+ s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
24
+
25
+ s.add_runtime_dependency 'logstash-codec-json'
26
+ s.add_runtime_dependency 'redis-rb-cluster'
27
+
28
+ s.add_development_dependency 'logstash-devutils'
29
+ end
30
+
@@ -0,0 +1,247 @@
1
+ require "logstash/devutils/rspec/spec_helper"
2
+ require "redis"
3
+ require "stud/try"
4
+ require 'logstash/inputs/redis_cluster'
5
+
6
+ def populate(key, event_count)
7
+ require "logstash/event"
8
+ redis = Redis.new(:host => "localhost")
9
+ event_count.times do |value|
10
+ event = LogStash::Event.new("sequence" => value)
11
+ Stud.try(10.times) do
12
+ redis.rpush(key, event.to_json)
13
+ end
14
+ end
15
+ end
16
+
17
+ def process(conf, event_count)
18
+ events = input(conf) do |pipeline, queue|
19
+ event_count.times.map{queue.pop}
20
+ end
21
+
22
+ events.each_with_index do |event, i|
23
+ insist { event["sequence"] } == i
24
+ end
25
+ end # process
26
+
27
+ # integration tests ---------------------
28
+
29
+ describe "inputs/redis_cluster", :redis => true do
30
+
31
+ it "should read events from a list" do
32
+ key = 10.times.collect { rand(10).to_s }.join("")
33
+ event_count = 1000 + rand(50)
34
+ # event_count = 100
35
+ conf = <<-CONFIG
36
+ input {
37
+ redis {
38
+ type => "blah"
39
+ key => "#{key}"
40
+ data_type => "list"
41
+ }
42
+ }
43
+ CONFIG
44
+
45
+ populate(key, event_count)
46
+ process(conf, event_count)
47
+ end
48
+
49
+ it "should read events from a list using batch_count" do
50
+ key = 10.times.collect { rand(10).to_s }.join("")
51
+ event_count = 1000 + rand(50)
52
+ conf = <<-CONFIG
53
+ input {
54
+ redis {
55
+ type => "blah"
56
+ key => "#{key}"
57
+ data_type => "list"
58
+ batch_count => #{rand(20)+1}
59
+ }
60
+ }
61
+ CONFIG
62
+
63
+ populate(key, event_count)
64
+ process(conf, event_count)
65
+ end
66
+ end
67
+
68
+ # unit tests ---------------------
69
+
70
+ describe LogStash::Inputs::RedisCluster do
71
+ let(:redis) { double('redis') }
72
+ let(:builder) { ->{ redis } }
73
+ let(:connection) { double('redis_connection') }
74
+ let(:connected) { [true] }
75
+ let(:data_type) { 'list' }
76
+ let(:cfg) { {'key' => 'foo', 'data_type' => data_type} }
77
+ let(:quit_calls) { [:quit] }
78
+ let(:accumulator) { [] }
79
+
80
+ subject do
81
+ LogStash::Plugin.lookup("input", "redis_cluster")
82
+ .new(cfg).add_external_redis_builder(builder)
83
+ end
84
+
85
+ context 'construction' do
86
+ it 'registers the input' do
87
+ expect {subject.register}.not_to raise_error
88
+ end
89
+ end
90
+
91
+ context 'runtime for list data_type' do
92
+ before do
93
+ subject.register
94
+ end
95
+
96
+ context 'close when redis is unset' do
97
+ let(:quit_calls) { [:quit, :unsubscribe, :punsubscribe, :connection, :disconnect!] }
98
+
99
+ it 'does not attempt to quit' do
100
+ allow(redis).to receive(:nil?).and_return(true)
101
+ quit_calls.each do |call|
102
+ expect(redis).not_to receive(call)
103
+ end
104
+ expect {subject.do_stop}.not_to raise_error
105
+ end
106
+ end
107
+
108
+ it 'calling the run method, adds events to the queue' do
109
+ expect(redis).to receive(:blpop).at_least(:once).and_return(['foo', 'l1'])
110
+
111
+ allow(redis).to receive(:connected?).and_return(connected.last)
112
+ allow(redis).to receive(:quit)
113
+
114
+ tt = Thread.new do
115
+ sleep 0.01
116
+ subject.do_stop
117
+ end
118
+
119
+ subject.run(accumulator)
120
+
121
+ tt.join
122
+
123
+ expect(accumulator.size).to be > 0
124
+ end
125
+
126
+ it 'multiple close calls, calls to redis once' do
127
+ subject.use_redis(redis)
128
+ allow(redis).to receive(:blpop).and_return(['foo', 'l1'])
129
+ expect(redis).to receive(:connected?).and_return(connected.last)
130
+ quit_calls.each do |call|
131
+ expect(redis).to receive(call).at_most(:once)
132
+ end
133
+
134
+ subject.do_stop
135
+ connected.push(false) #can't use let block here so push to array
136
+ expect {subject.do_stop}.not_to raise_error
137
+ subject.do_stop
138
+ end
139
+ end
140
+
141
+ context 'for the subscribe data_types' do
142
+ def run_it_thread(inst)
143
+ Thread.new(inst) do |subj|
144
+ subj.run(accumulator)
145
+ end
146
+ end
147
+
148
+ def publish_thread(new_redis, prefix)
149
+ Thread.new(new_redis, prefix) do |r, p|
150
+ sleep 0.1
151
+ 2.times do |i|
152
+ r.publish('foo', "#{p}#{i.next}")
153
+ end
154
+ end
155
+ end
156
+
157
+ def close_thread(inst, rt)
158
+ Thread.new(inst, rt) do |subj, runner|
159
+ sleep 0.4 # allow the messages through
160
+ runner.raise(LogStash::ShutdownSignal)
161
+ subj.close
162
+ end
163
+ end
164
+
165
+ let(:instance) do
166
+ inst = described_class.new(cfg)
167
+ inst.register
168
+ inst
169
+ end
170
+
171
+ before do
172
+ subject.register
173
+ subject.use_redis(redis)
174
+ allow(connection).to receive(:is_a?).and_return(true)
175
+ allow(redis).to receive(:client).and_return(connection)
176
+ end
177
+
178
+ before(:example, type: :mocked) do
179
+ expect(redis).to receive(:connected?).and_return(connected.last)
180
+ expect(connection).to receive(:unsubscribe)
181
+
182
+ quit_calls.each do |call|
183
+ expect(redis).to receive(call).at_most(:once)
184
+ end
185
+ end
186
+
187
+ context 'runtime for channel data_type' do
188
+ let(:data_type) { 'channel' }
189
+ let(:quit_calls) { [:unsubscribe, :connection] }
190
+
191
+ context 'mocked redis' do
192
+ it 'multiple stop calls, calls to redis once', type: :mocked do
193
+ subject.do_stop
194
+ connected.push(false) #can't use let block here so push to array
195
+ expect {subject.do_stop}.not_to raise_error
196
+ subject.do_stop
197
+ end
198
+ end
199
+
200
+ context 'real redis', :redis => true do
201
+ it 'calling the run method, adds events to the queue' do
202
+ #simulate the input thread
203
+ rt = run_it_thread(instance)
204
+ #simulate the other system thread
205
+ publish_thread(instance.new_redis_instance, 'c').join
206
+ #simulate the pipeline thread
207
+ close_thread(instance, rt).join
208
+
209
+ expect(accumulator.size).to eq(2)
210
+ end
211
+ end
212
+ end
213
+
214
+ context 'runtime for pattern_channel data_type' do
215
+ let(:data_type) { 'pattern_channel' }
216
+ let(:quit_calls) { [:punsubscribe, :connection] }
217
+
218
+ context 'mocked redis' do
219
+ it 'multiple stop calls, calls to redis once', type: :mocked do
220
+ subject.do_stop
221
+ connected.push(false) #can't use let block here so push to array
222
+ expect {subject.do_stop}.not_to raise_error
223
+ subject.do_stop
224
+ end
225
+ end
226
+
227
+ context 'real redis', :redis => true do
228
+ it 'calling the run method, adds events to the queue' do
229
+ #simulate the input thread
230
+ rt = run_it_thread(instance)
231
+ #simulate the other system thread
232
+ publish_thread(instance.new_redis_instance, 'pc').join
233
+ #simulate the pipeline thread
234
+ close_thread(instance, rt).join
235
+
236
+ expect(accumulator.size).to eq(2)
237
+ end
238
+ end
239
+ end
240
+ end
241
+
242
+ describe LogStash::Inputs::Redis do
243
+ it_behaves_like "an interruptible input plugin" do
244
+ let(:config) { {'keys' => ['foo'], 'data_type' => 'list'} }
245
+ end
246
+ end
247
+ end
metadata ADDED
@@ -0,0 +1,117 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-input-redis-cluster
3
+ version: !ruby/object:Gem::Version
4
+ version: 2.0.2
5
+ platform: ruby
6
+ authors:
7
+ - Elastic
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-10-18 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - '>='
17
+ - !ruby/object:Gem::Version
18
+ version: 2.0.0.beta2
19
+ - - <
20
+ - !ruby/object:Gem::Version
21
+ version: 3.0.0
22
+ name: logstash-core
23
+ prerelease: false
24
+ type: :runtime
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - '>='
28
+ - !ruby/object:Gem::Version
29
+ version: 2.0.0.beta2
30
+ - - <
31
+ - !ruby/object:Gem::Version
32
+ version: 3.0.0
33
+ - !ruby/object:Gem::Dependency
34
+ requirement: !ruby/object:Gem::Requirement
35
+ requirements:
36
+ - - '>='
37
+ - !ruby/object:Gem::Version
38
+ version: '0'
39
+ name: logstash-codec-json
40
+ prerelease: false
41
+ type: :runtime
42
+ version_requirements: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - '>='
45
+ - !ruby/object:Gem::Version
46
+ version: '0'
47
+ - !ruby/object:Gem::Dependency
48
+ requirement: !ruby/object:Gem::Requirement
49
+ requirements:
50
+ - - '>='
51
+ - !ruby/object:Gem::Version
52
+ version: '0'
53
+ name: redis-rb-cluster
54
+ prerelease: false
55
+ type: :runtime
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - '>='
59
+ - !ruby/object:Gem::Version
60
+ version: '0'
61
+ - !ruby/object:Gem::Dependency
62
+ requirement: !ruby/object:Gem::Requirement
63
+ requirements:
64
+ - - '>='
65
+ - !ruby/object:Gem::Version
66
+ version: '0'
67
+ name: logstash-devutils
68
+ prerelease: false
69
+ type: :development
70
+ version_requirements: !ruby/object:Gem::Requirement
71
+ requirements:
72
+ - - '>='
73
+ - !ruby/object:Gem::Version
74
+ version: '0'
75
+ description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
76
+ email: info@elastic.co
77
+ executables: []
78
+ extensions: []
79
+ extra_rdoc_files: []
80
+ files:
81
+ - CHANGELOG.md
82
+ - CONTRIBUTORS
83
+ - Gemfile
84
+ - LICENSE
85
+ - NOTICE.TXT
86
+ - README.md
87
+ - lib/logstash/inputs/redis_cluster.rb
88
+ - logstash-input-redis-cluster.gemspec
89
+ - spec/inputs/redis_spec.rb
90
+ homepage: http://www.elastic.co/guide/en/logstash/current/index.html
91
+ licenses:
92
+ - Apache License (2.0)
93
+ metadata:
94
+ logstash_plugin: 'true'
95
+ logstash_group: input
96
+ post_install_message:
97
+ rdoc_options: []
98
+ require_paths:
99
+ - lib
100
+ required_ruby_version: !ruby/object:Gem::Requirement
101
+ requirements:
102
+ - - '>='
103
+ - !ruby/object:Gem::Version
104
+ version: '0'
105
+ required_rubygems_version: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - '>='
108
+ - !ruby/object:Gem::Version
109
+ version: '0'
110
+ requirements: []
111
+ rubyforge_project:
112
+ rubygems_version: 2.4.5
113
+ signing_key:
114
+ specification_version: 4
115
+ summary: This input will read events from a Redis instance
116
+ test_files:
117
+ - spec/inputs/redis_spec.rb