logstash-input-redis-cluster 2.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +14 -0
- data/CONTRIBUTORS +21 -0
- data/Gemfile +3 -0
- data/LICENSE +13 -0
- data/NOTICE.TXT +5 -0
- data/README.md +86 -0
- data/lib/logstash/inputs/redis_cluster.rb +330 -0
- data/logstash-input-redis-cluster.gemspec +30 -0
- data/spec/inputs/redis_spec.rb +247 -0
- metadata +117 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 1c753cc2def1e0233efa5a49a580159d4ba2aa9f
|
4
|
+
data.tar.gz: e47850e128dee3d7f6a379bd2aaf25cafdeed534
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: f9a25fffed1c8b9f601e967546e17f81cb1e36d00869aa3ccc3fc1d4cc618cc749abd1eb88f97c95f1941893032b9139af36ad33feb611e12566d3579dcfc580
|
7
|
+
data.tar.gz: b0e9d2d6c56d4b80625d922b87140fb195ae8d5f4a7c1f5ff0537e31e420487d65475637859c8015a6ec67b990647d4ecc6cb513b75d68d0d8a4b00c9088fc7b
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,14 @@
|
|
1
|
+
## 2.0.0
|
2
|
+
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
|
3
|
+
instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
|
4
|
+
- Dependency on logstash-core update to 2.0
|
5
|
+
|
6
|
+
## 1.0.3
|
7
|
+
- Fix typo in module name in test (Causing CI build failures)
|
8
|
+
|
9
|
+
## 1.0.2
|
10
|
+
- Fix typo in module name (Causing the module to not be loaded)
|
11
|
+
|
12
|
+
## 1.0.1
|
13
|
+
- Make teardown more reliable
|
14
|
+
- Re-organise code and tests
|
data/CONTRIBUTORS
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The following is a list of people who have contributed ideas, code, bug
|
2
|
+
reports, or in general have helped logstash along its way.
|
3
|
+
|
4
|
+
Contributors:
|
5
|
+
* Aaron Mildenstein (untergeek)
|
6
|
+
* Bob Corsaro (dokipen)
|
7
|
+
* Colin Surprenant (colinsurprenant)
|
8
|
+
* Graham Bleach (bleach)
|
9
|
+
* Jason Woods (driskell)
|
10
|
+
* John E. Vincent (lusis)
|
11
|
+
* Jordan Sissel (jordansissel)
|
12
|
+
* Kurt Hurtado (kurtado)
|
13
|
+
* Pete Fritchman (fetep)
|
14
|
+
* Pier-Hugues Pellerin (ph)
|
15
|
+
* Richard Pijnenburg (electrical)
|
16
|
+
* piavlo
|
17
|
+
|
18
|
+
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
19
|
+
Logstash, and you aren't on the list above and want to be, please let us know
|
20
|
+
and we'll make sure you're here. Contributions from folks like you are what make
|
21
|
+
open source awesome.
|
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
|
2
|
+
|
3
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
+
you may not use this file except in compliance with the License.
|
5
|
+
You may obtain a copy of the License at
|
6
|
+
|
7
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
+
|
9
|
+
Unless required by applicable law or agreed to in writing, software
|
10
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
+
See the License for the specific language governing permissions and
|
13
|
+
limitations under the License.
|
data/NOTICE.TXT
ADDED
data/README.md
ADDED
@@ -0,0 +1,86 @@
|
|
1
|
+
# Logstash Plugin
|
2
|
+
|
3
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
4
|
+
|
5
|
+
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
|
6
|
+
|
7
|
+
## Documentation
|
8
|
+
|
9
|
+
Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
|
10
|
+
|
11
|
+
- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
|
12
|
+
- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
|
13
|
+
|
14
|
+
## Need Help?
|
15
|
+
|
16
|
+
Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
|
17
|
+
|
18
|
+
## Developing
|
19
|
+
|
20
|
+
### 1. Plugin Developement and Testing
|
21
|
+
|
22
|
+
#### Code
|
23
|
+
- To get started, you'll need JRuby with the Bundler gem installed.
|
24
|
+
|
25
|
+
- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
|
26
|
+
|
27
|
+
- Install dependencies
|
28
|
+
```sh
|
29
|
+
bundle install
|
30
|
+
```
|
31
|
+
|
32
|
+
#### Test
|
33
|
+
|
34
|
+
- Update your dependencies
|
35
|
+
|
36
|
+
```sh
|
37
|
+
bundle install
|
38
|
+
```
|
39
|
+
|
40
|
+
- Run tests
|
41
|
+
|
42
|
+
```sh
|
43
|
+
bundle exec rspec
|
44
|
+
```
|
45
|
+
|
46
|
+
### 2. Running your unpublished Plugin in Logstash
|
47
|
+
|
48
|
+
#### 2.1 Run in a local Logstash clone
|
49
|
+
|
50
|
+
- Edit Logstash `Gemfile` and add the local plugin path, for example:
|
51
|
+
```ruby
|
52
|
+
gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
|
53
|
+
```
|
54
|
+
- Install plugin
|
55
|
+
```sh
|
56
|
+
bin/plugin install --no-verify
|
57
|
+
```
|
58
|
+
- Run Logstash with your plugin
|
59
|
+
```sh
|
60
|
+
bin/logstash -e 'filter {awesome {}}'
|
61
|
+
```
|
62
|
+
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
|
63
|
+
|
64
|
+
#### 2.2 Run in an installed Logstash
|
65
|
+
|
66
|
+
You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
|
67
|
+
|
68
|
+
- Build your plugin gem
|
69
|
+
```sh
|
70
|
+
gem build logstash-filter-awesome.gemspec
|
71
|
+
```
|
72
|
+
- Install the plugin from the Logstash home
|
73
|
+
```sh
|
74
|
+
bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
|
75
|
+
```
|
76
|
+
- Start Logstash and proceed to test the plugin
|
77
|
+
|
78
|
+
## Contributing
|
79
|
+
|
80
|
+
All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
|
81
|
+
|
82
|
+
Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
|
83
|
+
|
84
|
+
It is more important to the community that you are able to contribute.
|
85
|
+
|
86
|
+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
|
@@ -0,0 +1,330 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
require "logstash/namespace"
|
3
|
+
require "logstash/inputs/base"
|
4
|
+
require "logstash/inputs/threadable"
|
5
|
+
|
6
|
+
# This input will read events from a Redis instance; it supports both Redis channels and lists.
|
7
|
+
# The list command (BLPOP) used by Logstash is supported in Redis v1.3.1+, and
|
8
|
+
# the channel commands used by Logstash are found in Redis v1.3.8+.
|
9
|
+
# While you may be able to make these Redis versions work, the best performance
|
10
|
+
# and stability will be found in more recent stable versions. Versions 2.6.0+
|
11
|
+
# are recommended.
|
12
|
+
#
|
13
|
+
# For more information about Redis, see <http://redis.io/>
|
14
|
+
#
|
15
|
+
# `batch_count` note: If you use the `batch_count` setting, you *must* use a Redis version 2.6.0 or
|
16
|
+
# newer. Anything older does not support the operations used by batching.
|
17
|
+
#
|
18
|
+
module LogStash module Inputs class RedisCluster < LogStash::Inputs::Threadable
|
19
|
+
# class LogStash::Inputs::Redis < LogStash::Inputs::Threadable
|
20
|
+
|
21
|
+
config_name "redis_cluster"
|
22
|
+
|
23
|
+
default :codec, "json"
|
24
|
+
|
25
|
+
# The `name` configuration is used for logging in case there are multiple instances.
|
26
|
+
# This feature has no real function and will be removed in future versions.
|
27
|
+
config :name, :validate => :string, :default => "default", :deprecated => true
|
28
|
+
|
29
|
+
# The hostname of your Redis server.
|
30
|
+
config :host, :validate => :string, :default => "127.0.0.1"
|
31
|
+
|
32
|
+
# The port to connect on.
|
33
|
+
config :port, :validate => :number, :default => 6379
|
34
|
+
|
35
|
+
# The Redis database number.
|
36
|
+
config :db, :validate => :number, :default => 0
|
37
|
+
|
38
|
+
# Initial connection timeout in seconds.
|
39
|
+
config :timeout, :validate => :number, :default => 5
|
40
|
+
|
41
|
+
# Password to authenticate with. There is no authentication by default.
|
42
|
+
config :password, :validate => :password
|
43
|
+
|
44
|
+
# The name of a Redis list or channel.
|
45
|
+
# TODO: change required to true
|
46
|
+
config :keys, :validate => :array
|
47
|
+
|
48
|
+
# Specify either list or channel. If `redis\_type` is `list`, then we will BLPOP the
|
49
|
+
# key. If `redis\_type` is `channel`, then we will SUBSCRIBE to the key.
|
50
|
+
# If `redis\_type` is `pattern_channel`, then we will PSUBSCRIBE to the key.
|
51
|
+
# TODO: change required to true
|
52
|
+
config :data_type, :validate => [ "list", "channel", "pattern_channel" ], :required => false
|
53
|
+
|
54
|
+
# The number of events to return from Redis using EVAL.
|
55
|
+
config :batch_count, :validate => :number, :default => 1
|
56
|
+
|
57
|
+
public
|
58
|
+
# public API
|
59
|
+
# use to store a proc that can provide a redis instance or mock
|
60
|
+
def add_external_redis_builder(builder) #callable
|
61
|
+
@redis_builder = builder
|
62
|
+
self
|
63
|
+
end
|
64
|
+
|
65
|
+
# use to apply an instance directly and bypass the builder
|
66
|
+
def use_redis(instance)
|
67
|
+
@redis = instance
|
68
|
+
self
|
69
|
+
end
|
70
|
+
|
71
|
+
def new_redis_instance
|
72
|
+
@redis_builder.call
|
73
|
+
end
|
74
|
+
|
75
|
+
def register
|
76
|
+
require 'redis-rb-cluster'
|
77
|
+
@redis_url = "redis://#{@password}@#{@host}:#{@port}/#{@db}"
|
78
|
+
|
79
|
+
# TODO remove after setting key and data_type to true
|
80
|
+
|
81
|
+
if !@keys || !@data_type
|
82
|
+
raise RuntimeError.new(
|
83
|
+
"Must define queue, or key and data_type parameters"
|
84
|
+
)
|
85
|
+
end
|
86
|
+
# end TODO
|
87
|
+
|
88
|
+
@redis_builder ||= method(:internal_redis_builder)
|
89
|
+
|
90
|
+
# just switch on data_type once
|
91
|
+
if @data_type == 'list' || @data_type == 'dummy'
|
92
|
+
@run_method = method(:list_runner)
|
93
|
+
@stop_method = method(:list_stop)
|
94
|
+
elsif @data_type == 'channel'
|
95
|
+
@run_method = method(:channel_runner)
|
96
|
+
@stop_method = method(:subscribe_stop)
|
97
|
+
elsif @data_type == 'pattern_channel'
|
98
|
+
@run_method = method(:pattern_channel_runner)
|
99
|
+
@stop_method = method(:subscribe_stop)
|
100
|
+
end
|
101
|
+
|
102
|
+
# TODO(sissel, boertje): set @identity directly when @name config option is removed.
|
103
|
+
@identity = @name != 'default' ? @name : "#{@redis_url} #{@data_type}"
|
104
|
+
@logger.info("Registering Redis", :identity => @identity)
|
105
|
+
end # def register
|
106
|
+
|
107
|
+
def run(output_queue)
|
108
|
+
@run_method.call(output_queue)
|
109
|
+
rescue LogStash::ShutdownSignal
|
110
|
+
# ignore and quit
|
111
|
+
end # def run
|
112
|
+
|
113
|
+
def stop
|
114
|
+
@stop_method.call
|
115
|
+
end
|
116
|
+
|
117
|
+
# private methods -----------------------------
|
118
|
+
private
|
119
|
+
|
120
|
+
def batched?
|
121
|
+
@batch_count > 1
|
122
|
+
end
|
123
|
+
|
124
|
+
# private
|
125
|
+
def is_list_type?
|
126
|
+
@data_type == 'list'
|
127
|
+
end
|
128
|
+
|
129
|
+
# private
|
130
|
+
def redis_params
|
131
|
+
{
|
132
|
+
:host => @host,
|
133
|
+
:port => @port,
|
134
|
+
:timeout => @timeout,
|
135
|
+
:db => @db,
|
136
|
+
:password => @password.nil? ? nil : @password.value
|
137
|
+
}
|
138
|
+
end
|
139
|
+
|
140
|
+
# private
|
141
|
+
def internal_redis_builder
|
142
|
+
::RedisCluster.new([redis_params],4)
|
143
|
+
end
|
144
|
+
|
145
|
+
# private
|
146
|
+
def connect
|
147
|
+
redis = new_redis_instance
|
148
|
+
load_batch_script(redis) if batched? && is_list_type?
|
149
|
+
redis
|
150
|
+
end # def connect
|
151
|
+
|
152
|
+
# private
|
153
|
+
def load_batch_script(redis)
|
154
|
+
#A Redis Lua EVAL script to fetch a count of keys
|
155
|
+
#in case count is bigger than current items in queue whole queue will be returned without extra nil values
|
156
|
+
redis_script = <<EOF
|
157
|
+
local i = tonumber(ARGV[1])
|
158
|
+
local res = {}
|
159
|
+
local length = redis.call('llen',KEYS[1])
|
160
|
+
if length < i then i = length end
|
161
|
+
while (i > 0) do
|
162
|
+
local item = redis.call("lpop", KEYS[1])
|
163
|
+
if (not item) then
|
164
|
+
break
|
165
|
+
end
|
166
|
+
table.insert(res, item)
|
167
|
+
i = i-1
|
168
|
+
end
|
169
|
+
return res
|
170
|
+
EOF
|
171
|
+
@redis_script_sha = redis.script(:load, redis_script)
|
172
|
+
end
|
173
|
+
|
174
|
+
# private
|
175
|
+
def queue_event(msg, output_queue)
|
176
|
+
begin
|
177
|
+
@codec.decode(msg) do |event|
|
178
|
+
decorate(event)
|
179
|
+
output_queue << event
|
180
|
+
end
|
181
|
+
rescue => e # parse or event creation error
|
182
|
+
@logger.error("Failed to create event", :message => msg, :exception => e, :backtrace => e.backtrace);
|
183
|
+
end
|
184
|
+
end
|
185
|
+
|
186
|
+
# private
|
187
|
+
def list_stop
|
188
|
+
return if @redis.nil? || !@redis.connected?
|
189
|
+
|
190
|
+
@redis.quit rescue nil
|
191
|
+
@redis = nil
|
192
|
+
end
|
193
|
+
|
194
|
+
# private
|
195
|
+
def list_runner(output_queue)
|
196
|
+
while !stop?
|
197
|
+
begin
|
198
|
+
@redis ||= connect
|
199
|
+
list_listener(@redis, output_queue)
|
200
|
+
rescue ::Redis::BaseError => e
|
201
|
+
@logger.warn("Redis connection problem", :exception => e)
|
202
|
+
# Reset the redis variable to trigger reconnect
|
203
|
+
@redis = nil
|
204
|
+
# this sleep does not need to be stoppable as its
|
205
|
+
# in a while !stop? loop
|
206
|
+
sleep 1
|
207
|
+
end
|
208
|
+
end
|
209
|
+
end
|
210
|
+
|
211
|
+
# private
|
212
|
+
def list_listener(redis, output_queue)
|
213
|
+
sampled = @keys.sample
|
214
|
+
item = redis.blpop(sampled, :timeout => 1)
|
215
|
+
return unless item # from timeout or other conditions
|
216
|
+
|
217
|
+
# blpop returns the 'key' read from as well as the item result
|
218
|
+
# we only care about the result (2nd item in the list).
|
219
|
+
queue_event(item.last, output_queue)
|
220
|
+
|
221
|
+
# If @batch_count is 1, there's no need to continue.
|
222
|
+
return if !batched?
|
223
|
+
|
224
|
+
begin
|
225
|
+
redis.evalsha(@redis_script_sha, [sampled], [@batch_count-1]).each do |item|
|
226
|
+
queue_event(item, output_queue)
|
227
|
+
end
|
228
|
+
|
229
|
+
# Below is a commented-out implementation of 'batch fetch'
|
230
|
+
# using pipelined LPOP calls. This in practice has been observed to
|
231
|
+
# perform exactly the same in terms of event throughput as
|
232
|
+
# the evalsha method. Given that the EVALSHA implementation uses
|
233
|
+
# one call to Redis instead of N (where N == @batch_count) calls,
|
234
|
+
# I decided to go with the 'evalsha' method of fetching N items
|
235
|
+
# from Redis in bulk.
|
236
|
+
#redis.pipelined do
|
237
|
+
#error, item = redis.lpop(@key)
|
238
|
+
#(@batch_count-1).times { redis.lpop(@key) }
|
239
|
+
#end.each do |item|
|
240
|
+
#queue_event(item, output_queue) if item
|
241
|
+
#end
|
242
|
+
# --- End commented out implementation of 'batch fetch'
|
243
|
+
rescue ::Redis::CommandError => e
|
244
|
+
if e.to_s =~ /NOSCRIPT/ then
|
245
|
+
@logger.warn("Redis may have been restarted, reloading Redis batch EVAL script", :exception => e);
|
246
|
+
load_batch_script(redis)
|
247
|
+
retry
|
248
|
+
else
|
249
|
+
raise e
|
250
|
+
end
|
251
|
+
end
|
252
|
+
end
|
253
|
+
|
254
|
+
# private
|
255
|
+
def subscribe_stop
|
256
|
+
return if @redis.nil? || !@redis.connected?
|
257
|
+
# if its a SubscribedClient then:
|
258
|
+
# it does not have a disconnect method (yet)
|
259
|
+
if @redis.client.is_a?(::Redis::SubscribedClient)
|
260
|
+
@redis.client.unsubscribe
|
261
|
+
else
|
262
|
+
@redis.client.disconnect
|
263
|
+
end
|
264
|
+
@redis = nil
|
265
|
+
end
|
266
|
+
|
267
|
+
# private
|
268
|
+
def redis_runner
|
269
|
+
begin
|
270
|
+
@redis ||= connect
|
271
|
+
yield
|
272
|
+
rescue ::Redis::BaseError => e
|
273
|
+
@logger.warn("Redis connection problem", :exception => e)
|
274
|
+
# Reset the redis variable to trigger reconnect
|
275
|
+
@redis = nil
|
276
|
+
Stud.stoppable_sleep(1) { stop? }
|
277
|
+
retry if !stop?
|
278
|
+
end
|
279
|
+
end
|
280
|
+
|
281
|
+
# private
|
282
|
+
def channel_runner(output_queue)
|
283
|
+
redis_runner do
|
284
|
+
channel_listener(output_queue)
|
285
|
+
end
|
286
|
+
end
|
287
|
+
|
288
|
+
# private
|
289
|
+
def channel_listener(output_queue)
|
290
|
+
@redis.subscribe(@keys.sample) do |on|
|
291
|
+
on.subscribe do |channel, count|
|
292
|
+
@logger.info("Subscribed", :channel => channel, :count => count)
|
293
|
+
end
|
294
|
+
|
295
|
+
on.message do |channel, message|
|
296
|
+
queue_event(message, output_queue)
|
297
|
+
end
|
298
|
+
|
299
|
+
on.unsubscribe do |channel, count|
|
300
|
+
@logger.info("Unsubscribed", :channel => channel, :count => count)
|
301
|
+
end
|
302
|
+
end
|
303
|
+
end
|
304
|
+
|
305
|
+
def pattern_channel_runner(output_queue)
|
306
|
+
redis_runner do
|
307
|
+
pattern_channel_listener(output_queue)
|
308
|
+
end
|
309
|
+
end
|
310
|
+
|
311
|
+
# private
|
312
|
+
def pattern_channel_listener(output_queue)
|
313
|
+
@redis.psubscribe @keys.sample do |on|
|
314
|
+
on.psubscribe do |channel, count|
|
315
|
+
@logger.info("Subscribed", :channel => channel, :count => count)
|
316
|
+
end
|
317
|
+
|
318
|
+
on.pmessage do |pattern, channel, message|
|
319
|
+
queue_event(message, output_queue)
|
320
|
+
end
|
321
|
+
|
322
|
+
on.punsubscribe do |channel, count|
|
323
|
+
@logger.info("Unsubscribed", :channel => channel, :count => count)
|
324
|
+
end
|
325
|
+
end
|
326
|
+
end
|
327
|
+
|
328
|
+
# end
|
329
|
+
|
330
|
+
end end end # Redis Inputs LogStash
|
@@ -0,0 +1,30 @@
|
|
1
|
+
Gem::Specification.new do |s|
|
2
|
+
|
3
|
+
s.name = 'logstash-input-redis-cluster'
|
4
|
+
s.version = '2.0.2'
|
5
|
+
s.licenses = ['Apache License (2.0)']
|
6
|
+
s.summary = "This input will read events from a Redis instance"
|
7
|
+
s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
|
8
|
+
s.authors = ["Elastic"]
|
9
|
+
s.email = 'info@elastic.co'
|
10
|
+
s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
|
11
|
+
s.require_paths = ["lib"]
|
12
|
+
|
13
|
+
# Files
|
14
|
+
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
|
15
|
+
|
16
|
+
# Tests
|
17
|
+
s.test_files = s.files.grep(%r{^(test|spec|features)/})
|
18
|
+
|
19
|
+
# Special flag to let us know this is actually a logstash plugin
|
20
|
+
s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
|
21
|
+
|
22
|
+
# Gem dependencies
|
23
|
+
s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
|
24
|
+
|
25
|
+
s.add_runtime_dependency 'logstash-codec-json'
|
26
|
+
s.add_runtime_dependency 'redis-rb-cluster'
|
27
|
+
|
28
|
+
s.add_development_dependency 'logstash-devutils'
|
29
|
+
end
|
30
|
+
|
@@ -0,0 +1,247 @@
|
|
1
|
+
require "logstash/devutils/rspec/spec_helper"
|
2
|
+
require "redis"
|
3
|
+
require "stud/try"
|
4
|
+
require 'logstash/inputs/redis_cluster'
|
5
|
+
|
6
|
+
def populate(key, event_count)
|
7
|
+
require "logstash/event"
|
8
|
+
redis = Redis.new(:host => "localhost")
|
9
|
+
event_count.times do |value|
|
10
|
+
event = LogStash::Event.new("sequence" => value)
|
11
|
+
Stud.try(10.times) do
|
12
|
+
redis.rpush(key, event.to_json)
|
13
|
+
end
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
def process(conf, event_count)
|
18
|
+
events = input(conf) do |pipeline, queue|
|
19
|
+
event_count.times.map{queue.pop}
|
20
|
+
end
|
21
|
+
|
22
|
+
events.each_with_index do |event, i|
|
23
|
+
insist { event["sequence"] } == i
|
24
|
+
end
|
25
|
+
end # process
|
26
|
+
|
27
|
+
# integration tests ---------------------
|
28
|
+
|
29
|
+
describe "inputs/redis_cluster", :redis => true do
|
30
|
+
|
31
|
+
it "should read events from a list" do
|
32
|
+
key = 10.times.collect { rand(10).to_s }.join("")
|
33
|
+
event_count = 1000 + rand(50)
|
34
|
+
# event_count = 100
|
35
|
+
conf = <<-CONFIG
|
36
|
+
input {
|
37
|
+
redis {
|
38
|
+
type => "blah"
|
39
|
+
key => "#{key}"
|
40
|
+
data_type => "list"
|
41
|
+
}
|
42
|
+
}
|
43
|
+
CONFIG
|
44
|
+
|
45
|
+
populate(key, event_count)
|
46
|
+
process(conf, event_count)
|
47
|
+
end
|
48
|
+
|
49
|
+
it "should read events from a list using batch_count" do
|
50
|
+
key = 10.times.collect { rand(10).to_s }.join("")
|
51
|
+
event_count = 1000 + rand(50)
|
52
|
+
conf = <<-CONFIG
|
53
|
+
input {
|
54
|
+
redis {
|
55
|
+
type => "blah"
|
56
|
+
key => "#{key}"
|
57
|
+
data_type => "list"
|
58
|
+
batch_count => #{rand(20)+1}
|
59
|
+
}
|
60
|
+
}
|
61
|
+
CONFIG
|
62
|
+
|
63
|
+
populate(key, event_count)
|
64
|
+
process(conf, event_count)
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
68
|
+
# unit tests ---------------------
|
69
|
+
|
70
|
+
describe LogStash::Inputs::RedisCluster do
|
71
|
+
let(:redis) { double('redis') }
|
72
|
+
let(:builder) { ->{ redis } }
|
73
|
+
let(:connection) { double('redis_connection') }
|
74
|
+
let(:connected) { [true] }
|
75
|
+
let(:data_type) { 'list' }
|
76
|
+
let(:cfg) { {'key' => 'foo', 'data_type' => data_type} }
|
77
|
+
let(:quit_calls) { [:quit] }
|
78
|
+
let(:accumulator) { [] }
|
79
|
+
|
80
|
+
subject do
|
81
|
+
LogStash::Plugin.lookup("input", "redis_cluster")
|
82
|
+
.new(cfg).add_external_redis_builder(builder)
|
83
|
+
end
|
84
|
+
|
85
|
+
context 'construction' do
|
86
|
+
it 'registers the input' do
|
87
|
+
expect {subject.register}.not_to raise_error
|
88
|
+
end
|
89
|
+
end
|
90
|
+
|
91
|
+
context 'runtime for list data_type' do
|
92
|
+
before do
|
93
|
+
subject.register
|
94
|
+
end
|
95
|
+
|
96
|
+
context 'close when redis is unset' do
|
97
|
+
let(:quit_calls) { [:quit, :unsubscribe, :punsubscribe, :connection, :disconnect!] }
|
98
|
+
|
99
|
+
it 'does not attempt to quit' do
|
100
|
+
allow(redis).to receive(:nil?).and_return(true)
|
101
|
+
quit_calls.each do |call|
|
102
|
+
expect(redis).not_to receive(call)
|
103
|
+
end
|
104
|
+
expect {subject.do_stop}.not_to raise_error
|
105
|
+
end
|
106
|
+
end
|
107
|
+
|
108
|
+
it 'calling the run method, adds events to the queue' do
|
109
|
+
expect(redis).to receive(:blpop).at_least(:once).and_return(['foo', 'l1'])
|
110
|
+
|
111
|
+
allow(redis).to receive(:connected?).and_return(connected.last)
|
112
|
+
allow(redis).to receive(:quit)
|
113
|
+
|
114
|
+
tt = Thread.new do
|
115
|
+
sleep 0.01
|
116
|
+
subject.do_stop
|
117
|
+
end
|
118
|
+
|
119
|
+
subject.run(accumulator)
|
120
|
+
|
121
|
+
tt.join
|
122
|
+
|
123
|
+
expect(accumulator.size).to be > 0
|
124
|
+
end
|
125
|
+
|
126
|
+
it 'multiple close calls, calls to redis once' do
|
127
|
+
subject.use_redis(redis)
|
128
|
+
allow(redis).to receive(:blpop).and_return(['foo', 'l1'])
|
129
|
+
expect(redis).to receive(:connected?).and_return(connected.last)
|
130
|
+
quit_calls.each do |call|
|
131
|
+
expect(redis).to receive(call).at_most(:once)
|
132
|
+
end
|
133
|
+
|
134
|
+
subject.do_stop
|
135
|
+
connected.push(false) #can't use let block here so push to array
|
136
|
+
expect {subject.do_stop}.not_to raise_error
|
137
|
+
subject.do_stop
|
138
|
+
end
|
139
|
+
end
|
140
|
+
|
141
|
+
context 'for the subscribe data_types' do
|
142
|
+
def run_it_thread(inst)
|
143
|
+
Thread.new(inst) do |subj|
|
144
|
+
subj.run(accumulator)
|
145
|
+
end
|
146
|
+
end
|
147
|
+
|
148
|
+
def publish_thread(new_redis, prefix)
|
149
|
+
Thread.new(new_redis, prefix) do |r, p|
|
150
|
+
sleep 0.1
|
151
|
+
2.times do |i|
|
152
|
+
r.publish('foo', "#{p}#{i.next}")
|
153
|
+
end
|
154
|
+
end
|
155
|
+
end
|
156
|
+
|
157
|
+
def close_thread(inst, rt)
|
158
|
+
Thread.new(inst, rt) do |subj, runner|
|
159
|
+
sleep 0.4 # allow the messages through
|
160
|
+
runner.raise(LogStash::ShutdownSignal)
|
161
|
+
subj.close
|
162
|
+
end
|
163
|
+
end
|
164
|
+
|
165
|
+
let(:instance) do
|
166
|
+
inst = described_class.new(cfg)
|
167
|
+
inst.register
|
168
|
+
inst
|
169
|
+
end
|
170
|
+
|
171
|
+
before do
|
172
|
+
subject.register
|
173
|
+
subject.use_redis(redis)
|
174
|
+
allow(connection).to receive(:is_a?).and_return(true)
|
175
|
+
allow(redis).to receive(:client).and_return(connection)
|
176
|
+
end
|
177
|
+
|
178
|
+
before(:example, type: :mocked) do
|
179
|
+
expect(redis).to receive(:connected?).and_return(connected.last)
|
180
|
+
expect(connection).to receive(:unsubscribe)
|
181
|
+
|
182
|
+
quit_calls.each do |call|
|
183
|
+
expect(redis).to receive(call).at_most(:once)
|
184
|
+
end
|
185
|
+
end
|
186
|
+
|
187
|
+
context 'runtime for channel data_type' do
|
188
|
+
let(:data_type) { 'channel' }
|
189
|
+
let(:quit_calls) { [:unsubscribe, :connection] }
|
190
|
+
|
191
|
+
context 'mocked redis' do
|
192
|
+
it 'multiple stop calls, calls to redis once', type: :mocked do
|
193
|
+
subject.do_stop
|
194
|
+
connected.push(false) #can't use let block here so push to array
|
195
|
+
expect {subject.do_stop}.not_to raise_error
|
196
|
+
subject.do_stop
|
197
|
+
end
|
198
|
+
end
|
199
|
+
|
200
|
+
context 'real redis', :redis => true do
|
201
|
+
it 'calling the run method, adds events to the queue' do
|
202
|
+
#simulate the input thread
|
203
|
+
rt = run_it_thread(instance)
|
204
|
+
#simulate the other system thread
|
205
|
+
publish_thread(instance.new_redis_instance, 'c').join
|
206
|
+
#simulate the pipeline thread
|
207
|
+
close_thread(instance, rt).join
|
208
|
+
|
209
|
+
expect(accumulator.size).to eq(2)
|
210
|
+
end
|
211
|
+
end
|
212
|
+
end
|
213
|
+
|
214
|
+
context 'runtime for pattern_channel data_type' do
|
215
|
+
let(:data_type) { 'pattern_channel' }
|
216
|
+
let(:quit_calls) { [:punsubscribe, :connection] }
|
217
|
+
|
218
|
+
context 'mocked redis' do
|
219
|
+
it 'multiple stop calls, calls to redis once', type: :mocked do
|
220
|
+
subject.do_stop
|
221
|
+
connected.push(false) #can't use let block here so push to array
|
222
|
+
expect {subject.do_stop}.not_to raise_error
|
223
|
+
subject.do_stop
|
224
|
+
end
|
225
|
+
end
|
226
|
+
|
227
|
+
context 'real redis', :redis => true do
|
228
|
+
it 'calling the run method, adds events to the queue' do
|
229
|
+
#simulate the input thread
|
230
|
+
rt = run_it_thread(instance)
|
231
|
+
#simulate the other system thread
|
232
|
+
publish_thread(instance.new_redis_instance, 'pc').join
|
233
|
+
#simulate the pipeline thread
|
234
|
+
close_thread(instance, rt).join
|
235
|
+
|
236
|
+
expect(accumulator.size).to eq(2)
|
237
|
+
end
|
238
|
+
end
|
239
|
+
end
|
240
|
+
end
|
241
|
+
|
242
|
+
describe LogStash::Inputs::Redis do
|
243
|
+
it_behaves_like "an interruptible input plugin" do
|
244
|
+
let(:config) { {'keys' => ['foo'], 'data_type' => 'list'} }
|
245
|
+
end
|
246
|
+
end
|
247
|
+
end
|
metadata
ADDED
@@ -0,0 +1,117 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: logstash-input-redis-cluster
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 2.0.2
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Elastic
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2015-10-18 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
15
|
+
requirements:
|
16
|
+
- - '>='
|
17
|
+
- !ruby/object:Gem::Version
|
18
|
+
version: 2.0.0.beta2
|
19
|
+
- - <
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: 3.0.0
|
22
|
+
name: logstash-core
|
23
|
+
prerelease: false
|
24
|
+
type: :runtime
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - '>='
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 2.0.0.beta2
|
30
|
+
- - <
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: 3.0.0
|
33
|
+
- !ruby/object:Gem::Dependency
|
34
|
+
requirement: !ruby/object:Gem::Requirement
|
35
|
+
requirements:
|
36
|
+
- - '>='
|
37
|
+
- !ruby/object:Gem::Version
|
38
|
+
version: '0'
|
39
|
+
name: logstash-codec-json
|
40
|
+
prerelease: false
|
41
|
+
type: :runtime
|
42
|
+
version_requirements: !ruby/object:Gem::Requirement
|
43
|
+
requirements:
|
44
|
+
- - '>='
|
45
|
+
- !ruby/object:Gem::Version
|
46
|
+
version: '0'
|
47
|
+
- !ruby/object:Gem::Dependency
|
48
|
+
requirement: !ruby/object:Gem::Requirement
|
49
|
+
requirements:
|
50
|
+
- - '>='
|
51
|
+
- !ruby/object:Gem::Version
|
52
|
+
version: '0'
|
53
|
+
name: redis-rb-cluster
|
54
|
+
prerelease: false
|
55
|
+
type: :runtime
|
56
|
+
version_requirements: !ruby/object:Gem::Requirement
|
57
|
+
requirements:
|
58
|
+
- - '>='
|
59
|
+
- !ruby/object:Gem::Version
|
60
|
+
version: '0'
|
61
|
+
- !ruby/object:Gem::Dependency
|
62
|
+
requirement: !ruby/object:Gem::Requirement
|
63
|
+
requirements:
|
64
|
+
- - '>='
|
65
|
+
- !ruby/object:Gem::Version
|
66
|
+
version: '0'
|
67
|
+
name: logstash-devutils
|
68
|
+
prerelease: false
|
69
|
+
type: :development
|
70
|
+
version_requirements: !ruby/object:Gem::Requirement
|
71
|
+
requirements:
|
72
|
+
- - '>='
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: '0'
|
75
|
+
description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
|
76
|
+
email: info@elastic.co
|
77
|
+
executables: []
|
78
|
+
extensions: []
|
79
|
+
extra_rdoc_files: []
|
80
|
+
files:
|
81
|
+
- CHANGELOG.md
|
82
|
+
- CONTRIBUTORS
|
83
|
+
- Gemfile
|
84
|
+
- LICENSE
|
85
|
+
- NOTICE.TXT
|
86
|
+
- README.md
|
87
|
+
- lib/logstash/inputs/redis_cluster.rb
|
88
|
+
- logstash-input-redis-cluster.gemspec
|
89
|
+
- spec/inputs/redis_spec.rb
|
90
|
+
homepage: http://www.elastic.co/guide/en/logstash/current/index.html
|
91
|
+
licenses:
|
92
|
+
- Apache License (2.0)
|
93
|
+
metadata:
|
94
|
+
logstash_plugin: 'true'
|
95
|
+
logstash_group: input
|
96
|
+
post_install_message:
|
97
|
+
rdoc_options: []
|
98
|
+
require_paths:
|
99
|
+
- lib
|
100
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
101
|
+
requirements:
|
102
|
+
- - '>='
|
103
|
+
- !ruby/object:Gem::Version
|
104
|
+
version: '0'
|
105
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
106
|
+
requirements:
|
107
|
+
- - '>='
|
108
|
+
- !ruby/object:Gem::Version
|
109
|
+
version: '0'
|
110
|
+
requirements: []
|
111
|
+
rubyforge_project:
|
112
|
+
rubygems_version: 2.4.5
|
113
|
+
signing_key:
|
114
|
+
specification_version: 4
|
115
|
+
summary: This input will read events from a Redis instance
|
116
|
+
test_files:
|
117
|
+
- spec/inputs/redis_spec.rb
|