logstash-filter-rediss 0.3.5 → 1.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -17
- data/CONTRIBUTORS +2 -8
- data/LICENSE +1 -1
- data/README.md +48 -10
- data/lib/logstash/filters/rediss.rb +48 -30
- data/logstash-filter-rediss.gemspec +10 -9
- metadata +6 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 300e9f93677fcf02905c91573ab605abcee5760a600d30d6cb7dddf83ca655be
|
4
|
+
data.tar.gz: a1034196a4ddeb9b22c3d2ee2def4cb5145a7ef1864917964a74a8393098ffbe
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: '0313399da64d527725b273f983c369a438815044776a54617476ec678d7058b18ded2c459a9e0372cccb0ceac353a2eec906bab7c1406bd0a7a46ede443d36aa'
|
7
|
+
data.tar.gz: ea8144cdf41d8fcfd8da4f87d849043d007ac7d783e8a2dd5877846e34849f56126d8a98bb6f335a454932a9a1d06d3181eaba456e7f52576755a4e57385e20d
|
data/CHANGELOG.md
CHANGED
@@ -1,19 +1,9 @@
|
|
1
|
-
## 0.
|
2
|
-
-
|
3
|
-
|
4
|
-
|
5
|
-
-
|
1
|
+
## 1.0.2
|
2
|
+
- Support for GET, HGET, SET, HSET, SETEX, EXISTS, DEL, SADD, SMEMBERS, SISMEMBER and SCARD;
|
3
|
+
- Support SSL (redis or rediss protocol);
|
4
|
+
- Support TTL (when you uses SETEX);
|
5
|
+
- Introduce new control field `cmd_key_is_formatted` for declaring commands
|
6
|
+
to be resolved through - see %{foo} handling;
|
6
7
|
- Introduce optional configuration options for controlling the count and
|
7
8
|
the interval of retries - see :max_retries, :lock_retry_interval and
|
8
|
-
:max_lock_retries
|
9
|
-
|
10
|
-
## 0.3.0
|
11
|
-
- Support for GET, SET, EXISTS, DEL, SADD, SMEMBERS and SCARD
|
12
|
-
- Introduce new control field `cmd_key_is_formatted` for declaring commands
|
13
|
-
to be resolved through - see %{foo} handling
|
14
|
-
|
15
|
-
## 0.2.0
|
16
|
-
- Support for HSET and HGET
|
17
|
-
|
18
|
-
## 0.1.0
|
19
|
-
- Initial version; simple caching with redis
|
9
|
+
:max_lock_retries;
|
data/CONTRIBUTORS
CHANGED
@@ -1,10 +1,4 @@
|
|
1
|
-
The following is a list of people who have contributed ideas, code, bug
|
2
|
-
reports, or in general have helped logstash-filter-cache-redis along its way.
|
3
|
-
|
4
1
|
Contributors:
|
2
|
+
* Thiago Barradas (ThiagoBarradas)
|
3
|
+
* Bruno Sales (bruno-sales)
|
5
4
|
* David Robakowski (drobakowski)
|
6
|
-
|
7
|
-
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
8
|
-
logstash-filter-cache-redis, and you aren't on the list above and want to be,
|
9
|
-
please let us know and we'll make sure you're here. Contributions from folks
|
10
|
-
like you are what make open source awesome.
|
data/LICENSE
CHANGED
data/README.md
CHANGED
@@ -1,23 +1,61 @@
|
|
1
|
-
# Logstash Filter
|
1
|
+
# Logstash Filter Rediss Plugin
|
2
2
|
|
3
|
-
[](https://github.com/synlay/logstash-filter-cache-redis)
|
3
|
+
[](https://travis-ci.org/thiagobarradas/logstash-filter-rediss)
|
4
|
+
[](https://badge.fury.io/rb/logstash-filter-rediss)
|
5
|
+
[](https://github.com/thiagobarradas/logstash-filter-rediss)
|
7
6
|
|
8
7
|
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
9
8
|
|
10
|
-
It is fully free and fully open source. The license is MIT, see [LICENSE](http://github.com/
|
9
|
+
It is fully free and fully open source. The license is MIT, see [LICENSE](http://github.com/thiagobarradas/logstash-filter-rediss/LICENSE) for further infos.
|
11
10
|
|
12
11
|
## Documentation
|
13
12
|
|
14
|
-
|
13
|
+
Actions allowed:
|
14
|
+
- `get` - Get cache with key in :get and set value in :target
|
15
|
+
- `set` - Set cache with key in :set and value from :source
|
16
|
+
- `setex` - Set cache with key in :setex, considering TTL from :ttl (in seconds) and value from :source
|
17
|
+
- `exists` - Checks if key :exists exists and save result in :target
|
18
|
+
- `del` - Deletes cache with key :del
|
19
|
+
- `llen` - Get length of a list with key :llen and save result in :target
|
20
|
+
- `rpush` - Appends a value :source in a list with key :rpush
|
21
|
+
- `rpushnx` - Appends a value :source in a list with key :rpush only with key not exists. This operation uses red lock;
|
22
|
+
- `hset` - Set hash with key in :hset, field from : field and value from :source
|
23
|
+
- `hget` - Get a value into :target from a hash field with :hget key
|
24
|
+
- `sadd` - Adds one or more members from :sadd to a set :source
|
25
|
+
- `sismember` - Determine if a :source value is a member of a set :sismember and save in :target
|
26
|
+
- `smembers` - Get all members from :smembers key and put in :target
|
27
|
+
- `scard` - Get number of members of set :scard and put in :target
|
28
|
+
- `rpop` - Removes and get last element from list :rpop and save in :target
|
29
|
+
- `lpop` - Removes and get first element from list :lpop and save in :target
|
30
|
+
- `lget` Get all elements from a list :lget and save in :target
|
31
|
+
|
32
|
+
Get Sample:
|
33
|
+
|
34
|
+
```ruby
|
35
|
+
filter {
|
36
|
+
rediss {
|
37
|
+
host => "redis.company.com"
|
38
|
+
port => 6379
|
39
|
+
db => 0
|
40
|
+
password => "authtoken"
|
41
|
+
get => "[data][id]"
|
42
|
+
target => "[data][result]"
|
43
|
+
}
|
44
|
+
}
|
45
|
+
```
|
46
|
+
|
47
|
+
Set with TTL Sample:
|
15
48
|
|
16
49
|
```ruby
|
17
50
|
filter {
|
18
|
-
|
19
|
-
|
20
|
-
|
51
|
+
rediss {
|
52
|
+
host => "redis.company.com"
|
53
|
+
port => 6379
|
54
|
+
db => 0
|
55
|
+
password => "authtoken"
|
56
|
+
ttl => 300
|
57
|
+
setex => "[data][id]"
|
58
|
+
source => "[data][content]"
|
21
59
|
}
|
22
60
|
}
|
23
61
|
```
|
@@ -8,20 +8,19 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
8
8
|
config_name "rediss"
|
9
9
|
|
10
10
|
# The field to perform filter
|
11
|
-
#
|
12
11
|
config :source, :validate => :string, :default => "message"
|
13
12
|
|
14
13
|
# The name of the container to put the result
|
15
|
-
#
|
16
14
|
config :target, :validate => :string, :default => "message"
|
17
15
|
|
16
|
+
# The field to use in hset
|
18
17
|
config :field, :validate => :string
|
19
18
|
|
20
|
-
# Expire time in seconds
|
19
|
+
# Expire time in seconds for setex
|
21
20
|
config :ttl => :number
|
22
21
|
|
23
22
|
# Informs if the connection is to be made with SSL or not
|
24
|
-
config :ssl => :boolean
|
23
|
+
config :ssl, :validate => :boolean, :default => false
|
25
24
|
|
26
25
|
# For now only working for rpushnx and llen!
|
27
26
|
config :cmd_key_is_formatted, :validate => :boolean, :default => false
|
@@ -62,62 +61,83 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
62
61
|
# Interval for retrying to acquire a lock
|
63
62
|
config :lock_retry_interval, :validate => :number, :default => 1
|
64
63
|
|
64
|
+
# config :get, :validate => :boolean, :default => false
|
65
|
+
config :lock_timeout, :validate => :number, :default => 5000
|
66
|
+
|
65
67
|
# Maximal count of retries to acquire a lock
|
66
68
|
config :max_lock_retries, :validate => :number, :default => 3
|
67
69
|
|
70
|
+
################# ACTIONS
|
71
|
+
|
72
|
+
# Sets the action. If has value, this action will be executed
|
73
|
+
# GET: get cache with key in :get and set value in :target
|
68
74
|
config :get, :validate => :string
|
69
75
|
|
76
|
+
# Sets the action. If has value, this action will be executed
|
77
|
+
# SET: set cache with key in :set and value from :source
|
70
78
|
config :set, :validate => :string
|
71
79
|
|
80
|
+
# Sets the action. If has value, this action will be executed
|
81
|
+
# SETEX: set cache with key in :setex, considering TTL from :ttl (in seconds) and value from :source
|
72
82
|
config :setex, :validate => :string
|
73
83
|
|
84
|
+
# Sets the action. If has value, this action will be executed
|
85
|
+
# EXISTS: perform exists operation in :exists key and save result in :target
|
74
86
|
config :exists, :validate => :string
|
75
87
|
|
88
|
+
# Sets the action. If has value, this action will be executed
|
89
|
+
# DEL: deletes cache with key :del
|
76
90
|
config :del, :validate => :string
|
77
91
|
|
78
|
-
#
|
79
|
-
#
|
92
|
+
# Sets the action. If has value, this action will be executed
|
93
|
+
# LLEN: get length of a list with key :llen and save result in :target
|
80
94
|
config :llen, :validate => :string
|
81
95
|
|
82
|
-
#
|
83
|
-
#
|
96
|
+
# Sets the action. If has value, this action will be executed
|
97
|
+
# RPUSH: aapend value :source in a list with key :rpush
|
84
98
|
config :rpush, :validate => :string
|
85
99
|
|
86
|
-
#
|
87
|
-
#
|
100
|
+
# Sets the action. If has value, this action will be executed
|
101
|
+
# RPUSHNX: aapend value :source in a list with key :rpush only with key not exists
|
102
|
+
# This operation uses red lock;
|
88
103
|
config :rpushnx, :validate => :string
|
89
104
|
|
90
|
-
# Sets the action. If
|
105
|
+
# Sets the action. If has value, this action will be executed
|
106
|
+
# HSET: set hash with key in :hset, field from : field and value from :source
|
91
107
|
config :hset, :validate => :string
|
92
108
|
|
93
|
-
# Sets the action. If
|
109
|
+
# Sets the action. If has value, this action will be executed
|
110
|
+
# HGET: get a value into :target from a hash field with :hget key
|
94
111
|
config :hget, :validate => :string
|
95
112
|
|
113
|
+
# Sets the action. If has value, this action will be executed
|
114
|
+
# SADD: add one or more members from :sadd to a set :source
|
96
115
|
config :sadd, :validate => :string
|
97
116
|
|
117
|
+
# Sets the action. If has value, this action will be executed
|
118
|
+
# SISMEMBER: Determine if a :source value is a member of a set :sismember and save in :target
|
98
119
|
config :sismember, :validate => :string
|
99
120
|
|
121
|
+
# Sets the action. If has value, this action will be executed
|
122
|
+
# SMEMBERS: Get all members from :smembers key and put in :target
|
100
123
|
config :smembers, :validate => :string
|
101
124
|
|
125
|
+
# Sets the action. If has value, this action will be executed
|
126
|
+
# SCARD: Get number of members of set :scard and put in :target
|
102
127
|
config :scard, :validate => :string
|
103
128
|
|
104
|
-
#
|
105
|
-
|
106
|
-
|
107
|
-
# # Sets the action. If set to true, it will get the data from redis cache
|
108
|
-
# config :get, :validate => :boolean, :default => false
|
129
|
+
# Sets the action. If has value, this action will be executed
|
130
|
+
# RPOP: Remove and get last element from list :rpop and save in :target
|
109
131
|
config :rpop, :validate => :string
|
110
132
|
|
111
|
-
#
|
112
|
-
#
|
133
|
+
# Sets the action. If has value, this action will be executed
|
134
|
+
# LPOP: Remove and get first element from list :lpop and save in :target
|
113
135
|
config :lpop, :validate => :string
|
114
136
|
|
115
|
-
#
|
116
|
-
#
|
117
|
-
# O(N)
|
137
|
+
# Sets the action. If has value, this action will be executed
|
138
|
+
# LGET: Get all elements from a list :lget and save in :target
|
118
139
|
config :lget, :validate => :string
|
119
140
|
|
120
|
-
|
121
141
|
public
|
122
142
|
def register
|
123
143
|
@redis = nil
|
@@ -128,7 +148,6 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
128
148
|
@host_idx = 0
|
129
149
|
end # def register
|
130
150
|
|
131
|
-
|
132
151
|
def filter(event)
|
133
152
|
|
134
153
|
# TODO: Maybe refactor the interface into a more flexible one with two
|
@@ -153,7 +172,7 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
153
172
|
end
|
154
173
|
|
155
174
|
if @exists
|
156
|
-
@redis.exists(event.get(@exists))
|
175
|
+
event.set(@target, @redis.exists(event.get(@exists)))
|
157
176
|
end
|
158
177
|
|
159
178
|
if @del
|
@@ -177,7 +196,7 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
177
196
|
end
|
178
197
|
|
179
198
|
if @smembers
|
180
|
-
@redis.smembers(event.get(@smembers))
|
199
|
+
event.set(@target, @redis.smembers(event.get(@smembers)))
|
181
200
|
end
|
182
201
|
|
183
202
|
if @scard
|
@@ -242,7 +261,6 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
242
261
|
filter_matched(event)
|
243
262
|
end # def filter
|
244
263
|
|
245
|
-
|
246
264
|
private
|
247
265
|
def connect
|
248
266
|
@current_host, @current_port = @host[@host_idx].split(':')
|
@@ -260,7 +278,7 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
260
278
|
:ssl => @ssl
|
261
279
|
}
|
262
280
|
|
263
|
-
@logger.debug("
|
281
|
+
@logger.debug("Connection params", params)
|
264
282
|
|
265
283
|
if @password
|
266
284
|
params[:password] = @password.value
|
@@ -279,6 +297,6 @@ class LogStash::Filters::Rediss < LogStash::Filters::Base
|
|
279
297
|
@logger.debug("lock_manager hosts", hosts)
|
280
298
|
|
281
299
|
Redlock::Client.new(hosts)
|
282
|
-
end # def
|
300
|
+
end # def connect_lockmanager
|
283
301
|
|
284
|
-
end # class LogStash::Filters::
|
302
|
+
end # class LogStash::Filters::Rediss
|
@@ -1,18 +1,19 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
|
-
s.name
|
3
|
-
s.version = '0.
|
2
|
+
s.name = 'logstash-filter-rediss'
|
3
|
+
s.version = '1.0.2'
|
4
4
|
s.licenses = ['MIT']
|
5
|
-
s.summary = "Redis
|
5
|
+
s.summary = "Redis (with SSL) Filter for Logstash"
|
6
6
|
s.description = "A Logstash filter plugin for storing and retrieving data from redis cache. This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gem_name. This gem is not a stand-alone program."
|
7
|
-
s.authors = ["David Robakowski"]
|
8
|
-
s.email = '
|
9
|
-
s.homepage = "https://github.com/
|
10
|
-
s.require_paths
|
11
|
-
s.platform
|
7
|
+
s.authors = ["Thiago Barradas", "Bruno Sales", "David Robakowski"]
|
8
|
+
s.email = 'th.barradas@gmail.com'
|
9
|
+
s.homepage = "https://github.com/thiagobarradas/logstash-filter-rediss"
|
10
|
+
s.require_paths = ["lib"]
|
11
|
+
s.platform = Gem::Platform::JAVA if defined?(RUBY_ENGINE) && RUBY_ENGINE == 'jruby'
|
12
12
|
|
13
13
|
# Files
|
14
14
|
s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE']
|
15
|
-
|
15
|
+
|
16
|
+
# Tests
|
16
17
|
s.test_files = s.files.grep(%r{^(test|spec|features)/})
|
17
18
|
|
18
19
|
# Special flag to let us know this is actually a logstash plugin
|
metadata
CHANGED
@@ -1,9 +1,11 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-filter-rediss
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
|
+
- Thiago Barradas
|
8
|
+
- Bruno Sales
|
7
9
|
- David Robakowski
|
8
10
|
autorequire:
|
9
11
|
bindir: bin
|
@@ -82,7 +84,7 @@ description: A Logstash filter plugin for storing and retrieving data from redis
|
|
82
84
|
This gem is a Logstash plugin required to be installed on top of the Logstash core
|
83
85
|
pipeline using $LS_HOME/bin/logstash-plugin install gem_name. This gem is not a
|
84
86
|
stand-alone program.
|
85
|
-
email:
|
87
|
+
email: th.barradas@gmail.com
|
86
88
|
executables: []
|
87
89
|
extensions: []
|
88
90
|
extra_rdoc_files: []
|
@@ -94,7 +96,7 @@ files:
|
|
94
96
|
- README.md
|
95
97
|
- lib/logstash/filters/rediss.rb
|
96
98
|
- logstash-filter-rediss.gemspec
|
97
|
-
homepage: https://github.com/
|
99
|
+
homepage: https://github.com/thiagobarradas/logstash-filter-rediss
|
98
100
|
licenses:
|
99
101
|
- MIT
|
100
102
|
metadata:
|
@@ -119,5 +121,5 @@ rubyforge_project:
|
|
119
121
|
rubygems_version: 2.7.7
|
120
122
|
signing_key:
|
121
123
|
specification_version: 4
|
122
|
-
summary: Redis
|
124
|
+
summary: Redis (with SSL) Filter for Logstash
|
123
125
|
test_files: []
|