fluent-plugin-redis-store-wejick 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/AUTHORS +2 -0
- data/Gemfile +3 -0
- data/README.md +158 -0
- data/Rakefile +11 -0
- data/fluent-plugin-redis-store-wejick.gemspec +21 -0
- data/lib/fluent/plugin/out_redis_store_wejick.rb +252 -0
- data/test/helpers.rb +28 -0
- data/test/plugin/test_out_redis_publish.rb +367 -0
- metadata +110 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 527ccf44f03adce04f4d258ebfd141682f458214
|
4
|
+
data.tar.gz: 2dffb41df7987498176f9c1f1d96fdfa28482c37
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 2996eac1fe39a32fdc986cf4ff5caada8841a272cfaa3df2ae90a84116678d4fdef3a77ca9c1afce46cb00c23530287ceed85ac38f6675fa1ab2776a3dcfdfb6
|
7
|
+
data.tar.gz: 498b42ca521cecda815e0e60e5a71821a601205bc70c67ecf101182fcdeece3746e94c6d4721dc22ede23473ad5aec47f17d8e143962d49087d6c6730d91b68b
|
data/AUTHORS
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,158 @@
|
|
1
|
+
Redis Output Plugin For fluentd
|
2
|
+
===============================
|
3
|
+
[Fluentd][] output plugin to upload/publish event data to [Redis][] storage.
|
4
|
+
|
5
|
+
[Fluentd]: http://fluentd.org/
|
6
|
+
[Redis]: http://redis.io/
|
7
|
+
|
8
|
+
Background
|
9
|
+
----------
|
10
|
+
|
11
|
+
This is folked project from [fluent-plugin-redisstore][].
|
12
|
+
|
13
|
+
[fluent-plugin-redisstore]: https://github.com/moaikids/fluent-plugin-redisstore
|
14
|
+
|
15
|
+
Features
|
16
|
+
--------
|
17
|
+
|
18
|
+
#### Supported Redis commands
|
19
|
+
|
20
|
+
Currently the plugin supports following Redis commands:
|
21
|
+
|
22
|
+
- **set** by `string` type (of the plugin)
|
23
|
+
- **lpush**/**rpush** by `list` type
|
24
|
+
- **sadd** by `set` type
|
25
|
+
- **zadd** by `zset` type
|
26
|
+
- **publish** by `publish` type
|
27
|
+
|
28
|
+
#### Supported _value_ format
|
29
|
+
|
30
|
+
- plain(as is)
|
31
|
+
- JSON
|
32
|
+
- [MessagePack](http://msgpack.org/)
|
33
|
+
|
34
|
+
#### _key_ string for Redis storage
|
35
|
+
|
36
|
+
Redis commands require _key_ and _value_.
|
37
|
+
For _key_, the plugin supports either way;
|
38
|
+
|
39
|
+
1. Specify a fixed key.
|
40
|
+
You can do this simply using `key` option in td-agent configuration file.
|
41
|
+
|
42
|
+
```apache
|
43
|
+
type redis_store
|
44
|
+
key userdata
|
45
|
+
```
|
46
|
+
|
47
|
+
2. Lookup a key string in every event data by a lookup path.
|
48
|
+
If event data have structured data like
|
49
|
+
|
50
|
+
```javascript
|
51
|
+
{ "user": { "name": "Kei" } }
|
52
|
+
```
|
53
|
+
|
54
|
+
and you want to use each name of user, you can use `key_path` option.
|
55
|
+
|
56
|
+
```apache
|
57
|
+
type redis_store
|
58
|
+
key_path user.name
|
59
|
+
```
|
60
|
+
|
61
|
+
With the above data, `Kei` will be a _key_.
|
62
|
+
|
63
|
+
In addition, `key_prefix` and `key_suffix` are useful in some cases. Both are available either `key` and `key_path`
|
64
|
+
|
65
|
+
```apache
|
66
|
+
type redis_store
|
67
|
+
key_path user.name
|
68
|
+
key_prefix ouruser.
|
69
|
+
key_suffix .accesslog
|
70
|
+
```
|
71
|
+
|
72
|
+
With the previous data, _key_ will be `outuser.Kei.accesslog`.
|
73
|
+
|
74
|
+
#### _value_ data for Redis storage
|
75
|
+
|
76
|
+
To determine what _value_ in every event data to be srtored, you have two options;
|
77
|
+
|
78
|
+
1. Store extracted data in event data, by a lookup path with `value_path` option.
|
79
|
+
It works like `key_path`.
|
80
|
+
2. Store whole data.
|
81
|
+
This is default behavior. To do it, simply omit `value_path` option.
|
82
|
+
|
83
|
+
Installation
|
84
|
+
------------
|
85
|
+
|
86
|
+
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-redis-store
|
87
|
+
|
88
|
+
Configuration
|
89
|
+
-------------
|
90
|
+
|
91
|
+
### Redis connection
|
92
|
+
|
93
|
+
| Key | Type | Required? | Default | Description |
|
94
|
+
| :---- | :----- | :---------- | :----------------------- | :------------ |
|
95
|
+
| `host` | string | Optional | 127.0.0.1 | host name of Redis server |
|
96
|
+
| `port` | int | Optional | 6379 | port number of Redis server |
|
97
|
+
| `password` | string | Optional | | password for Redis connection |
|
98
|
+
| `path` | string | Optional | | To connect via Unix socket, try '/tmp/redis.sock' |
|
99
|
+
| `db` | int | Optional | 0 | DB number of Redis |
|
100
|
+
| `timeout` | float | Optional | 5.0 | connection timeout in seconds |
|
101
|
+
|
102
|
+
### common options for storages
|
103
|
+
|
104
|
+
| Key | Type | Default | Description |
|
105
|
+
| :---- | :----- | :----------------------- | :------------ |
|
106
|
+
| `key` | string | | Fixed _key_ used to store(publish) in Redis |
|
107
|
+
| `key_path` | string | | path to lookup for _key_ in the event data |
|
108
|
+
| `key_prefix` | string | | prefix of _key_ |
|
109
|
+
| `key_suffix` | string | | suffix of _key_ |
|
110
|
+
| `value_path` | string | (whole event data) | path to lookup for _value_ in the event data |
|
111
|
+
| `store_type` | string | zset | `string`/`list`/`set`/`zset`/`publish` |
|
112
|
+
| `format_type` | string | plain | format type for _value_ (`plain`/`json`/`msgpack`) |
|
113
|
+
| `key_expire` | int | -1 | If set, the key will be expired in specified seconds |
|
114
|
+
| `flush_interval` | time | 1 | Time interval which events will be flushed to Redis |
|
115
|
+
| `prevent_duplicate` | (0,1) | 0 | Prevent duplicated value in one key (list)
|
116
|
+
Note: either `key` or `key_path` is required.
|
117
|
+
|
118
|
+
### `string` storage specific options
|
119
|
+
|
120
|
+
| Key | Type | Default | Description |
|
121
|
+
| :---- | :----- | :----------------------- | :------------ |
|
122
|
+
| `type` | string | | Fixed _key_ used to store(publish) in Redis |
|
123
|
+
No more options than common options.
|
124
|
+
|
125
|
+
### `list` storage specific options
|
126
|
+
|
127
|
+
| Key | Type | Default | Description |
|
128
|
+
| :---- | :----- | :----------------------- | :------------ |
|
129
|
+
| `order` | string | asc | `asc`: **rpush**, `desc`: **lpush** |
|
130
|
+
|
131
|
+
### `set` storage specific options
|
132
|
+
|
133
|
+
No more options than common options.
|
134
|
+
|
135
|
+
### `zset` storage specific options
|
136
|
+
|
137
|
+
| Key | Type | Default | Description |
|
138
|
+
| :---- | :----- | :----------------------- | :------------ |
|
139
|
+
| `score_path` | string | (_time_ of log event) | path to lookup for _score_ in the event data |
|
140
|
+
| `value_expire` | int | | value expiration in seconds |
|
141
|
+
|
142
|
+
If `value_expire` is set, the plugin assumes that the _score_ in the **SortedSet** is
|
143
|
+
based on *timestamp* and it deletes expired _members_ every after new event data arrives.
|
144
|
+
|
145
|
+
### `publish` storage specific options
|
146
|
+
|
147
|
+
No more options than common options.
|
148
|
+
|
149
|
+
|
150
|
+
Copyright
|
151
|
+
---------
|
152
|
+
|
153
|
+
Copyright (c) 2013 moaikids
|
154
|
+
Copyright (c) 2014 HANAI Tohru
|
155
|
+
|
156
|
+
License
|
157
|
+
-------
|
158
|
+
Apache License, Version 2.0
|
data/Rakefile
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
Gem::Specification.new do |gem|
|
3
|
+
gem.name = "fluent-plugin-redis-store-wejick"
|
4
|
+
gem.email = "wejick@gmail.com"
|
5
|
+
gem.version = "0.0.1"
|
6
|
+
gem.authors = ["wejick", "Gian Giovani"]
|
7
|
+
gem.licenses = ["Apache License Version 2.0"]
|
8
|
+
gem.summary = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd}
|
9
|
+
gem.description = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd...}
|
10
|
+
gem.homepage = "https://github.com/wejick/fluent-plugin-redis-store"
|
11
|
+
|
12
|
+
gem.files = `git ls-files`.split($\)
|
13
|
+
gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
14
|
+
gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
|
15
|
+
gem.require_paths = ["lib"]
|
16
|
+
|
17
|
+
gem.add_development_dependency "rake"
|
18
|
+
gem.add_development_dependency "test-unit"
|
19
|
+
gem.add_runtime_dependency "fluentd"
|
20
|
+
gem.add_runtime_dependency "redis"
|
21
|
+
end
|
@@ -0,0 +1,252 @@
|
|
1
|
+
module Fluent
|
2
|
+
class RedisStoreOutput < BufferedOutput
|
3
|
+
Fluent::Plugin.register_output('redis_store_wejick', self)
|
4
|
+
|
5
|
+
# redis connection
|
6
|
+
config_param :host, :string, :default => '127.0.0.1'
|
7
|
+
config_param :port, :integer, :default => 6379
|
8
|
+
config_param :path, :string, :default => nil
|
9
|
+
config_param :password, :string, :default => nil
|
10
|
+
config_param :db, :integer, :default => 0
|
11
|
+
config_param :timeout, :float, :default => 5.0
|
12
|
+
|
13
|
+
# redis command and parameters
|
14
|
+
config_param :format_type, :string, :default => 'json'
|
15
|
+
config_param :store_type, :string, :default => 'zset'
|
16
|
+
config_param :key_prefix, :string, :default => ''
|
17
|
+
config_param :key_suffix, :string, :default => ''
|
18
|
+
config_param :key, :string, :default => nil
|
19
|
+
config_param :key_path, :string, :default => nil
|
20
|
+
config_param :score_path, :string, :default => nil
|
21
|
+
config_param :value_path, :string, :default => ''
|
22
|
+
config_param :key_expire, :integer, :default => -1
|
23
|
+
config_param :value_expire, :integer, :default => -1
|
24
|
+
config_param :value_length, :integer, :default => -1
|
25
|
+
config_param :order, :string, :default => 'asc'
|
26
|
+
config_param :prevent_duplicate, :integer, :default => 0
|
27
|
+
config_param :only_alphabet,:integer, :default => 0
|
28
|
+
config_param :tidy_string, :integer, :default => 0
|
29
|
+
|
30
|
+
def initialize
|
31
|
+
super
|
32
|
+
require 'redis'
|
33
|
+
require 'msgpack'
|
34
|
+
require 'cgi'
|
35
|
+
end
|
36
|
+
|
37
|
+
def configure(conf)
|
38
|
+
super
|
39
|
+
|
40
|
+
if @key_path == nil and @key == nil
|
41
|
+
raise Fluent::ConfigError, "either key_path or key is required"
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def start
|
46
|
+
super
|
47
|
+
if @path
|
48
|
+
@redis = Redis.new(:path => @path, :password => @password,
|
49
|
+
:timeout => @timeout, :thread_safe => true, :db => @db)
|
50
|
+
else
|
51
|
+
@redis = Redis.new(:host => @host, :port => @port, :password => @password,
|
52
|
+
:timeout => @timeout, :thread_safe => true, :db => @db)
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
def shutdown
|
57
|
+
@redis.quit
|
58
|
+
end
|
59
|
+
|
60
|
+
def format(tag, time, record)
|
61
|
+
[tag, time, record].to_msgpack
|
62
|
+
end
|
63
|
+
|
64
|
+
def write(chunk)
|
65
|
+
@redis.pipelined {
|
66
|
+
chunk.open { |io|
|
67
|
+
begin
|
68
|
+
MessagePack::Unpacker.new(io).each { |message|
|
69
|
+
begin
|
70
|
+
(tag, time, record) = message
|
71
|
+
case @store_type
|
72
|
+
when 'zset'
|
73
|
+
operation_for_zset(record, time)
|
74
|
+
when 'set'
|
75
|
+
operation_for_set(record)
|
76
|
+
when 'list'
|
77
|
+
operation_for_list(record)
|
78
|
+
when 'string'
|
79
|
+
operation_for_string(record)
|
80
|
+
when 'publish'
|
81
|
+
operation_for_publish(record)
|
82
|
+
end
|
83
|
+
rescue NoMethodError => e
|
84
|
+
puts e
|
85
|
+
end
|
86
|
+
}
|
87
|
+
rescue EOFError
|
88
|
+
# EOFError always occured when reached end of chunk.
|
89
|
+
end
|
90
|
+
}
|
91
|
+
}
|
92
|
+
end
|
93
|
+
|
94
|
+
def operation_for_zset(record, time)
|
95
|
+
key = get_key_from(record)
|
96
|
+
value = get_value_from(record)
|
97
|
+
score = get_score_from(record, time)
|
98
|
+
@redis.zadd key, score, value
|
99
|
+
|
100
|
+
set_key_expire key
|
101
|
+
if 0 < @value_expire
|
102
|
+
now = Time.now.to_i
|
103
|
+
@redis.zremrangebyscore key , '-inf' , (now - @value_expire)
|
104
|
+
end
|
105
|
+
if 0 < @value_length
|
106
|
+
script = generate_zremrangebyrank_script(key, @value_length, @order)
|
107
|
+
@redis.eval script
|
108
|
+
end
|
109
|
+
end
|
110
|
+
|
111
|
+
def operation_for_set(record)
|
112
|
+
key = get_key_from(record)
|
113
|
+
value = get_value_from(record)
|
114
|
+
@redis.sadd key, value
|
115
|
+
set_key_expire key
|
116
|
+
end
|
117
|
+
|
118
|
+
def operation_for_list(record)
|
119
|
+
key = get_key_from(record)
|
120
|
+
value = get_value_from(record)
|
121
|
+
|
122
|
+
if 0 < @tidy_string
|
123
|
+
value = tidy_up_string(value)
|
124
|
+
end
|
125
|
+
if 0 < @only_alphabet
|
126
|
+
if ( /^[a-zA-Z0-9 ]*$/.match(value) ) != nil
|
127
|
+
else
|
128
|
+
return
|
129
|
+
end
|
130
|
+
end
|
131
|
+
if 0 < @prevent_duplicate
|
132
|
+
@redis.lrem key.to_s, 1, value.to_s
|
133
|
+
end
|
134
|
+
if @order == 'asc'
|
135
|
+
@redis.rpush key, value
|
136
|
+
else
|
137
|
+
@redis.lpush key, value
|
138
|
+
end
|
139
|
+
set_key_expire key
|
140
|
+
if 0 < @value_length
|
141
|
+
script = generate_ltrim_script(key, @value_length, @order)
|
142
|
+
@redis.eval script
|
143
|
+
end
|
144
|
+
end
|
145
|
+
|
146
|
+
def operation_for_string(record)
|
147
|
+
key = get_key_from(record)
|
148
|
+
value = get_value_from(record)
|
149
|
+
@redis.set key, value
|
150
|
+
|
151
|
+
set_key_expire key
|
152
|
+
end
|
153
|
+
|
154
|
+
def operation_for_publish(record)
|
155
|
+
key = get_key_from(record)
|
156
|
+
value = get_value_from(record)
|
157
|
+
@redis.publish key, value
|
158
|
+
end
|
159
|
+
|
160
|
+
# unescape and make it to downcase
|
161
|
+
def tidy_up_string(string)
|
162
|
+
string.downcase
|
163
|
+
string = CGI.unescape(string)
|
164
|
+
return string
|
165
|
+
end
|
166
|
+
|
167
|
+
def generate_zremrangebyrank_script(key, maxlen, order)
|
168
|
+
script = "local key = '" + key.to_s + "'\n"
|
169
|
+
script += "local maxlen = " + maxlen.to_s + "\n"
|
170
|
+
script += "local order ='" + order.to_s + "'\n"
|
171
|
+
script += "local len = tonumber(redis.call('ZCOUNT', key, '-inf', '+inf'))\n"
|
172
|
+
script += "if len > maxlen then\n"
|
173
|
+
script += " if order == 'asc' then\n"
|
174
|
+
script += " local l = len - maxlen\n"
|
175
|
+
script += " if l >= 0 then\n"
|
176
|
+
script += " return redis.call('ZREMRANGEBYRANK', key, 0, l)\n"
|
177
|
+
script += " end\n"
|
178
|
+
script += " else\n"
|
179
|
+
script += " return redis.call('ZREMRANGEBYRANK', key, maxlen, -1)\n"
|
180
|
+
script += " end\n"
|
181
|
+
script += "end\n"
|
182
|
+
return script
|
183
|
+
end
|
184
|
+
|
185
|
+
def generate_ltrim_script(key, maxlen, order)
|
186
|
+
script = "local key = '" + key.to_s + "'\n"
|
187
|
+
script += "local maxlen = " + maxlen.to_s + "\n"
|
188
|
+
script += "local order ='" + order.to_s + "'\n"
|
189
|
+
script += "local len = tonumber(redis.call('LLEN', key))\n"
|
190
|
+
script += "if len > maxlen then\n"
|
191
|
+
script += " if order == 'asc' then\n"
|
192
|
+
script += " local l = len - maxlen\n"
|
193
|
+
script += " return redis.call('LTRIM', key, l, -1)\n"
|
194
|
+
script += " else\n"
|
195
|
+
script += " return redis.call('LTRIM', key, 0, maxlen - 1)\n"
|
196
|
+
script += " end\n"
|
197
|
+
script += "end\n"
|
198
|
+
return script
|
199
|
+
end
|
200
|
+
|
201
|
+
def traverse(data, key)
|
202
|
+
val = data
|
203
|
+
key.split('.').each{ |k|
|
204
|
+
if val.has_key?(k)
|
205
|
+
val = val[k]
|
206
|
+
else
|
207
|
+
return nil
|
208
|
+
end
|
209
|
+
}
|
210
|
+
return val
|
211
|
+
end
|
212
|
+
|
213
|
+
def get_key_from(record)
|
214
|
+
if @key
|
215
|
+
k = @key
|
216
|
+
else
|
217
|
+
k = traverse(record, @key_path).to_s
|
218
|
+
end
|
219
|
+
key = @key_prefix + k + @key_suffix
|
220
|
+
|
221
|
+
raise Fluent::ConfigError, "key is empty" if key == ''
|
222
|
+
key
|
223
|
+
end
|
224
|
+
|
225
|
+
def get_value_from(record)
|
226
|
+
value = traverse(record, @value_path)
|
227
|
+
case @format_type
|
228
|
+
when 'json'
|
229
|
+
value.to_json
|
230
|
+
when 'msgpack'
|
231
|
+
value.to_msgpack
|
232
|
+
else
|
233
|
+
value
|
234
|
+
end
|
235
|
+
end
|
236
|
+
|
237
|
+
def get_score_from(record, time)
|
238
|
+
if @score_path
|
239
|
+
traverse(record, @score_path)
|
240
|
+
else
|
241
|
+
time
|
242
|
+
end
|
243
|
+
end
|
244
|
+
|
245
|
+
def set_key_expire(key)
|
246
|
+
if 0 < @key_expire
|
247
|
+
@redis.expire key, @key_expire
|
248
|
+
end
|
249
|
+
end
|
250
|
+
|
251
|
+
end
|
252
|
+
end
|
data/test/helpers.rb
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'bundler'
|
3
|
+
begin
|
4
|
+
Bundler.setup(:default, :development)
|
5
|
+
rescue Bundler::BundlerError => e
|
6
|
+
$stderr.puts e.message
|
7
|
+
$stderr.puts "Run `bundle install` to install missing gems"
|
8
|
+
exit e.status_code
|
9
|
+
end
|
10
|
+
require 'test/unit'
|
11
|
+
|
12
|
+
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
|
13
|
+
$LOAD_PATH.unshift(File.dirname(__FILE__))
|
14
|
+
require 'fluent/test'
|
15
|
+
unless ENV.has_key?('VERBOSE')
|
16
|
+
nulllogger = Object.new
|
17
|
+
nulllogger.instance_eval {|obj|
|
18
|
+
def method_missing(method, *args)
|
19
|
+
# pass
|
20
|
+
end
|
21
|
+
}
|
22
|
+
$log = nulllogger
|
23
|
+
end
|
24
|
+
|
25
|
+
require 'fluent/plugin/out_redis_store'
|
26
|
+
|
27
|
+
class Test::Unit::TestCase
|
28
|
+
end
|
@@ -0,0 +1,367 @@
|
|
1
|
+
require 'helpers'
|
2
|
+
|
3
|
+
require 'redis'
|
4
|
+
|
5
|
+
$channel = nil
|
6
|
+
$message = nil
|
7
|
+
|
8
|
+
class Redis
|
9
|
+
def initialize(options = {})
|
10
|
+
end
|
11
|
+
|
12
|
+
def pipelined
|
13
|
+
yield self
|
14
|
+
end
|
15
|
+
|
16
|
+
def set(key, message)
|
17
|
+
$command = :set
|
18
|
+
$key = key
|
19
|
+
$message = message
|
20
|
+
end
|
21
|
+
|
22
|
+
def rpush(key, message)
|
23
|
+
$command = :rpush
|
24
|
+
$key = key
|
25
|
+
$message = message
|
26
|
+
end
|
27
|
+
|
28
|
+
def lpush(key, message)
|
29
|
+
$command = :lpush
|
30
|
+
$key = key
|
31
|
+
$message = message
|
32
|
+
end
|
33
|
+
|
34
|
+
def sadd(key, message)
|
35
|
+
$command = :sadd
|
36
|
+
$key = key
|
37
|
+
$message = message
|
38
|
+
end
|
39
|
+
|
40
|
+
def zadd(key, score, message)
|
41
|
+
$command = :zadd
|
42
|
+
$key = key
|
43
|
+
$score = score
|
44
|
+
$message = message
|
45
|
+
end
|
46
|
+
|
47
|
+
def expire(key, ttl)
|
48
|
+
$expire_key = key
|
49
|
+
$ttl = ttl
|
50
|
+
end
|
51
|
+
|
52
|
+
def publish(channel, message)
|
53
|
+
$command = :publish
|
54
|
+
$channel = channel
|
55
|
+
$message = message
|
56
|
+
end
|
57
|
+
|
58
|
+
def quit
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
class RedisStoreOutputTest < Test::Unit::TestCase
|
63
|
+
def setup
|
64
|
+
Fluent::Test.setup
|
65
|
+
end
|
66
|
+
|
67
|
+
def create_driver(conf)
|
68
|
+
Fluent::Test::BufferedOutputTestDriver.new(Fluent::RedisStoreOutput).configure(conf)
|
69
|
+
end
|
70
|
+
|
71
|
+
def test_configure_defaults
|
72
|
+
config = %[
|
73
|
+
key_path a
|
74
|
+
score_path b
|
75
|
+
]
|
76
|
+
d = create_driver(config)
|
77
|
+
assert_equal "127.0.0.1", d.instance.host
|
78
|
+
assert_equal 6379, d.instance.port
|
79
|
+
assert_equal nil, d.instance.path
|
80
|
+
assert_equal nil, d.instance.password
|
81
|
+
assert_equal 0, d.instance.db
|
82
|
+
assert_equal 5.0, d.instance.timeout
|
83
|
+
assert_equal 'json', d.instance.format_type
|
84
|
+
assert_equal '', d.instance.key_prefix
|
85
|
+
assert_equal '', d.instance.key_suffix
|
86
|
+
assert_equal 'zset', d.instance.store_type
|
87
|
+
assert_equal 'a', d.instance.key_path
|
88
|
+
assert_equal nil, d.instance.key
|
89
|
+
assert_equal 'b', d.instance.score_path
|
90
|
+
assert_equal '', d.instance.value_path
|
91
|
+
assert_equal -1, d.instance.key_expire
|
92
|
+
assert_equal -1, d.instance.value_expire
|
93
|
+
assert_equal -1, d.instance.value_length
|
94
|
+
assert_equal 'asc', d.instance.order
|
95
|
+
end
|
96
|
+
|
97
|
+
def test_configure_host_port_db
|
98
|
+
config = %[
|
99
|
+
host 192.168.2.3
|
100
|
+
port 9999
|
101
|
+
password abc
|
102
|
+
db 3
|
103
|
+
timeout 7
|
104
|
+
key a
|
105
|
+
score_path b
|
106
|
+
]
|
107
|
+
d = create_driver(config)
|
108
|
+
assert_equal "192.168.2.3", d.instance.host
|
109
|
+
assert_equal 9999, d.instance.port
|
110
|
+
assert_equal nil, d.instance.path
|
111
|
+
assert_equal 'abc', d.instance.password
|
112
|
+
assert_equal 3, d.instance.db
|
113
|
+
assert_equal 7.0, d.instance.timeout
|
114
|
+
assert_equal nil, d.instance.key_path
|
115
|
+
assert_equal 'a', d.instance.key
|
116
|
+
end
|
117
|
+
|
118
|
+
def test_configure_path
|
119
|
+
config = %[
|
120
|
+
path /tmp/foo.sock
|
121
|
+
key a
|
122
|
+
score_path b
|
123
|
+
]
|
124
|
+
d = create_driver(config)
|
125
|
+
assert_equal "/tmp/foo.sock", d.instance.path
|
126
|
+
end
|
127
|
+
|
128
|
+
def test_configure_exception
|
129
|
+
assert_raise(Fluent::ConfigError) do
|
130
|
+
create_driver(%[])
|
131
|
+
end
|
132
|
+
end
|
133
|
+
|
134
|
+
# def test_write
|
135
|
+
# d = create_driver(CONFIG1)
|
136
|
+
#
|
137
|
+
# time = Time.parse("2011-01-02 13:14:15 UTC").to_i
|
138
|
+
# d.emit({ "foo" => "bar" }, time)
|
139
|
+
# d.run
|
140
|
+
#
|
141
|
+
# assert_equal "test", $channel
|
142
|
+
# assert_equal(%Q[{"foo":"bar","time":#{time}}], $message)
|
143
|
+
# end
|
144
|
+
|
145
|
+
def get_time
|
146
|
+
Time.parse("2011-01-02 13:14:15 UTC").to_i
|
147
|
+
end
|
148
|
+
|
149
|
+
# it should return whole message
|
150
|
+
def test_omit_value_path
|
151
|
+
config = %[
|
152
|
+
format_type plain
|
153
|
+
store_type string
|
154
|
+
key_path user
|
155
|
+
]
|
156
|
+
d = create_driver(config)
|
157
|
+
message = {
|
158
|
+
'user' => 'george',
|
159
|
+
'stat' => { 'attack' => 7 }
|
160
|
+
}
|
161
|
+
$ttl = nil
|
162
|
+
d.emit(message, get_time)
|
163
|
+
d.run
|
164
|
+
|
165
|
+
assert_equal "george", $key
|
166
|
+
assert_equal message, $message
|
167
|
+
assert_equal nil, $ttl
|
168
|
+
end
|
169
|
+
|
170
|
+
def test_key_value_paths
|
171
|
+
config = %[
|
172
|
+
format_type plain
|
173
|
+
store_type string
|
174
|
+
key_path user.name
|
175
|
+
value_path stat.attack
|
176
|
+
key_expire 3
|
177
|
+
]
|
178
|
+
d = create_driver(config)
|
179
|
+
message = {
|
180
|
+
'user' => { 'name' => 'george' },
|
181
|
+
'stat' => { 'attack' => 7 }
|
182
|
+
}
|
183
|
+
$ttl = nil
|
184
|
+
d.emit(message, get_time)
|
185
|
+
d.run
|
186
|
+
|
187
|
+
assert_equal "george", $key
|
188
|
+
assert_equal 7, $message
|
189
|
+
assert_equal 3, $ttl
|
190
|
+
end
|
191
|
+
|
192
|
+
def test_json
|
193
|
+
config = %[
|
194
|
+
format_type json
|
195
|
+
store_type string
|
196
|
+
key_path user
|
197
|
+
]
|
198
|
+
d = create_driver(config)
|
199
|
+
message = {
|
200
|
+
'user' => 'george',
|
201
|
+
'stat' => { 'attack' => 7 }
|
202
|
+
}
|
203
|
+
$ttl = nil
|
204
|
+
d.emit(message, get_time)
|
205
|
+
d.run
|
206
|
+
|
207
|
+
assert_equal "george", $key
|
208
|
+
assert_equal message.to_json, $message
|
209
|
+
end
|
210
|
+
|
211
|
+
def test_msgpack
|
212
|
+
config = %[
|
213
|
+
format_type msgpack
|
214
|
+
store_type string
|
215
|
+
key_path user
|
216
|
+
]
|
217
|
+
d = create_driver(config)
|
218
|
+
message = {
|
219
|
+
'user' => 'george',
|
220
|
+
'stat' => { 'attack' => 7 }
|
221
|
+
}
|
222
|
+
$ttl = nil
|
223
|
+
d.emit(message, get_time)
|
224
|
+
d.run
|
225
|
+
|
226
|
+
assert_equal "george", $key
|
227
|
+
assert_equal message.to_msgpack, $message
|
228
|
+
end
|
229
|
+
|
230
|
+
def test_list_asc
|
231
|
+
config = %[
|
232
|
+
format_type plain
|
233
|
+
store_type list
|
234
|
+
key_path user
|
235
|
+
]
|
236
|
+
d = create_driver(config)
|
237
|
+
message = {
|
238
|
+
'user' => 'george',
|
239
|
+
'stat' => { 'attack' => 7 }
|
240
|
+
}
|
241
|
+
d.emit(message, get_time)
|
242
|
+
d.run
|
243
|
+
|
244
|
+
assert_equal :rpush, $command
|
245
|
+
assert_equal "george", $key
|
246
|
+
assert_equal message, $message
|
247
|
+
end
|
248
|
+
|
249
|
+
def test_list_desc
|
250
|
+
config = %[
|
251
|
+
format_type plain
|
252
|
+
store_type list
|
253
|
+
key_path user
|
254
|
+
order desc
|
255
|
+
]
|
256
|
+
d = create_driver(config)
|
257
|
+
message = {
|
258
|
+
'user' => 'george',
|
259
|
+
'stat' => { 'attack' => 7 }
|
260
|
+
}
|
261
|
+
d.emit(message, get_time)
|
262
|
+
d.run
|
263
|
+
|
264
|
+
assert_equal :lpush, $command
|
265
|
+
assert_equal "george", $key
|
266
|
+
assert_equal message, $message
|
267
|
+
end
|
268
|
+
|
269
|
+
def test_set
|
270
|
+
config = %[
|
271
|
+
format_type plain
|
272
|
+
store_type set
|
273
|
+
key_path user
|
274
|
+
order desc
|
275
|
+
]
|
276
|
+
d = create_driver(config)
|
277
|
+
message = {
|
278
|
+
'user' => 'george',
|
279
|
+
'stat' => { 'attack' => 7 }
|
280
|
+
}
|
281
|
+
d.emit(message, get_time)
|
282
|
+
d.run
|
283
|
+
|
284
|
+
assert_equal :sadd, $command
|
285
|
+
assert_equal "george", $key
|
286
|
+
assert_equal message, $message
|
287
|
+
end
|
288
|
+
|
289
|
+
def test_zset
|
290
|
+
config = %[
|
291
|
+
format_type plain
|
292
|
+
store_type zset
|
293
|
+
key_path user
|
294
|
+
score_path result
|
295
|
+
]
|
296
|
+
|
297
|
+
d = create_driver(config)
|
298
|
+
message = {
|
299
|
+
'user' => 'george',
|
300
|
+
'stat' => { 'attack' => 7 },
|
301
|
+
'result' => 81
|
302
|
+
}
|
303
|
+
d.emit(message, get_time)
|
304
|
+
d.run
|
305
|
+
|
306
|
+
assert_equal :zadd, $command
|
307
|
+
assert_equal "george", $key
|
308
|
+
assert_equal 81, $score
|
309
|
+
assert_equal message, $message
|
310
|
+
end
|
311
|
+
|
312
|
+
def test_zset_with_no_score_path
|
313
|
+
config = %[
|
314
|
+
format_type plain
|
315
|
+
store_type zset
|
316
|
+
key_path user
|
317
|
+
]
|
318
|
+
|
319
|
+
d = create_driver(config)
|
320
|
+
message = {
|
321
|
+
'user' => 'george',
|
322
|
+
'stat' => { 'attack' => 7 },
|
323
|
+
'result' => 81
|
324
|
+
}
|
325
|
+
d.emit(message, get_time)
|
326
|
+
d.run
|
327
|
+
|
328
|
+
assert_equal :zadd, $command
|
329
|
+
assert_equal "george", $key
|
330
|
+
assert_equal get_time, $score
|
331
|
+
assert_equal message, $message
|
332
|
+
end
|
333
|
+
|
334
|
+
def test_publish
|
335
|
+
config = %[
|
336
|
+
format_type plain
|
337
|
+
store_type publish
|
338
|
+
key_path user
|
339
|
+
]
|
340
|
+
|
341
|
+
d = create_driver(config)
|
342
|
+
message = {
|
343
|
+
'user' => 'george'
|
344
|
+
}
|
345
|
+
d.emit(message, get_time)
|
346
|
+
d.run
|
347
|
+
|
348
|
+
assert_equal :publish, $command
|
349
|
+
assert_equal "george", $channel
|
350
|
+
assert_equal message, $message
|
351
|
+
end
|
352
|
+
|
353
|
+
def test_empty_key
|
354
|
+
config = %[
|
355
|
+
format_type plain
|
356
|
+
store_type string
|
357
|
+
key_path none
|
358
|
+
]
|
359
|
+
|
360
|
+
d = create_driver(config)
|
361
|
+
message = { 'user' => 'george' }
|
362
|
+
d.emit(message, get_time)
|
363
|
+
assert_raise(Fluent::ConfigError) do
|
364
|
+
d.run
|
365
|
+
end
|
366
|
+
end
|
367
|
+
end
|
metadata
ADDED
@@ -0,0 +1,110 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: fluent-plugin-redis-store-wejick
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- wejick
|
8
|
+
- Gian Giovani
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2016-08-02 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: rake
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
requirements:
|
18
|
+
- - ">="
|
19
|
+
- !ruby/object:Gem::Version
|
20
|
+
version: '0'
|
21
|
+
type: :development
|
22
|
+
prerelease: false
|
23
|
+
version_requirements: !ruby/object:Gem::Requirement
|
24
|
+
requirements:
|
25
|
+
- - ">="
|
26
|
+
- !ruby/object:Gem::Version
|
27
|
+
version: '0'
|
28
|
+
- !ruby/object:Gem::Dependency
|
29
|
+
name: test-unit
|
30
|
+
requirement: !ruby/object:Gem::Requirement
|
31
|
+
requirements:
|
32
|
+
- - ">="
|
33
|
+
- !ruby/object:Gem::Version
|
34
|
+
version: '0'
|
35
|
+
type: :development
|
36
|
+
prerelease: false
|
37
|
+
version_requirements: !ruby/object:Gem::Requirement
|
38
|
+
requirements:
|
39
|
+
- - ">="
|
40
|
+
- !ruby/object:Gem::Version
|
41
|
+
version: '0'
|
42
|
+
- !ruby/object:Gem::Dependency
|
43
|
+
name: fluentd
|
44
|
+
requirement: !ruby/object:Gem::Requirement
|
45
|
+
requirements:
|
46
|
+
- - ">="
|
47
|
+
- !ruby/object:Gem::Version
|
48
|
+
version: '0'
|
49
|
+
type: :runtime
|
50
|
+
prerelease: false
|
51
|
+
version_requirements: !ruby/object:Gem::Requirement
|
52
|
+
requirements:
|
53
|
+
- - ">="
|
54
|
+
- !ruby/object:Gem::Version
|
55
|
+
version: '0'
|
56
|
+
- !ruby/object:Gem::Dependency
|
57
|
+
name: redis
|
58
|
+
requirement: !ruby/object:Gem::Requirement
|
59
|
+
requirements:
|
60
|
+
- - ">="
|
61
|
+
- !ruby/object:Gem::Version
|
62
|
+
version: '0'
|
63
|
+
type: :runtime
|
64
|
+
prerelease: false
|
65
|
+
version_requirements: !ruby/object:Gem::Requirement
|
66
|
+
requirements:
|
67
|
+
- - ">="
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: '0'
|
70
|
+
description: Redis(zset/set/list/string/publish) output plugin for Fluentd...
|
71
|
+
email: wejick@gmail.com
|
72
|
+
executables: []
|
73
|
+
extensions: []
|
74
|
+
extra_rdoc_files: []
|
75
|
+
files:
|
76
|
+
- AUTHORS
|
77
|
+
- Gemfile
|
78
|
+
- README.md
|
79
|
+
- Rakefile
|
80
|
+
- fluent-plugin-redis-store-wejick.gemspec
|
81
|
+
- lib/fluent/plugin/out_redis_store_wejick.rb
|
82
|
+
- test/helpers.rb
|
83
|
+
- test/plugin/test_out_redis_publish.rb
|
84
|
+
homepage: https://github.com/wejick/fluent-plugin-redis-store
|
85
|
+
licenses:
|
86
|
+
- Apache License Version 2.0
|
87
|
+
metadata: {}
|
88
|
+
post_install_message:
|
89
|
+
rdoc_options: []
|
90
|
+
require_paths:
|
91
|
+
- lib
|
92
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
98
|
+
requirements:
|
99
|
+
- - ">="
|
100
|
+
- !ruby/object:Gem::Version
|
101
|
+
version: '0'
|
102
|
+
requirements: []
|
103
|
+
rubyforge_project:
|
104
|
+
rubygems_version: 2.5.1
|
105
|
+
signing_key:
|
106
|
+
specification_version: 4
|
107
|
+
summary: Redis(zset/set/list/string/publish) output plugin for Fluentd
|
108
|
+
test_files:
|
109
|
+
- test/helpers.rb
|
110
|
+
- test/plugin/test_out_redis_publish.rb
|