fluent-plugin-redis-store-wejick 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/AUTHORS +2 -0
- data/Gemfile +3 -0
- data/README.md +158 -0
- data/Rakefile +11 -0
- data/fluent-plugin-redis-store-wejick.gemspec +21 -0
- data/lib/fluent/plugin/out_redis_store_wejick.rb +252 -0
- data/test/helpers.rb +28 -0
- data/test/plugin/test_out_redis_publish.rb +367 -0
- metadata +110 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 527ccf44f03adce04f4d258ebfd141682f458214
|
4
|
+
data.tar.gz: 2dffb41df7987498176f9c1f1d96fdfa28482c37
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 2996eac1fe39a32fdc986cf4ff5caada8841a272cfaa3df2ae90a84116678d4fdef3a77ca9c1afce46cb00c23530287ceed85ac38f6675fa1ab2776a3dcfdfb6
|
7
|
+
data.tar.gz: 498b42ca521cecda815e0e60e5a71821a601205bc70c67ecf101182fcdeece3746e94c6d4721dc22ede23473ad5aec47f17d8e143962d49087d6c6730d91b68b
|
data/AUTHORS
ADDED
data/Gemfile
ADDED
data/README.md
ADDED
@@ -0,0 +1,158 @@
|
|
1
|
+
Redis Output Plugin For fluentd
|
2
|
+
===============================
|
3
|
+
[Fluentd][] output plugin to upload/publish event data to [Redis][] storage.
|
4
|
+
|
5
|
+
[Fluentd]: http://fluentd.org/
|
6
|
+
[Redis]: http://redis.io/
|
7
|
+
|
8
|
+
Background
|
9
|
+
----------
|
10
|
+
|
11
|
+
This is folked project from [fluent-plugin-redisstore][].
|
12
|
+
|
13
|
+
[fluent-plugin-redisstore]: https://github.com/moaikids/fluent-plugin-redisstore
|
14
|
+
|
15
|
+
Features
|
16
|
+
--------
|
17
|
+
|
18
|
+
#### Supported Redis commands
|
19
|
+
|
20
|
+
Currently the plugin supports following Redis commands:
|
21
|
+
|
22
|
+
- **set** by `string` type (of the plugin)
|
23
|
+
- **lpush**/**rpush** by `list` type
|
24
|
+
- **sadd** by `set` type
|
25
|
+
- **zadd** by `zset` type
|
26
|
+
- **publish** by `publish` type
|
27
|
+
|
28
|
+
#### Supported _value_ format
|
29
|
+
|
30
|
+
- plain(as is)
|
31
|
+
- JSON
|
32
|
+
- [MessagePack](http://msgpack.org/)
|
33
|
+
|
34
|
+
#### _key_ string for Redis storage
|
35
|
+
|
36
|
+
Redis commands require _key_ and _value_.
|
37
|
+
For _key_, the plugin supports either way;
|
38
|
+
|
39
|
+
1. Specify a fixed key.
|
40
|
+
You can do this simply using `key` option in td-agent configuration file.
|
41
|
+
|
42
|
+
```apache
|
43
|
+
type redis_store
|
44
|
+
key userdata
|
45
|
+
```
|
46
|
+
|
47
|
+
2. Lookup a key string in every event data by a lookup path.
|
48
|
+
If event data have structured data like
|
49
|
+
|
50
|
+
```javascript
|
51
|
+
{ "user": { "name": "Kei" } }
|
52
|
+
```
|
53
|
+
|
54
|
+
and you want to use each name of user, you can use `key_path` option.
|
55
|
+
|
56
|
+
```apache
|
57
|
+
type redis_store
|
58
|
+
key_path user.name
|
59
|
+
```
|
60
|
+
|
61
|
+
With the above data, `Kei` will be a _key_.
|
62
|
+
|
63
|
+
In addition, `key_prefix` and `key_suffix` are useful in some cases. Both are available either `key` and `key_path`
|
64
|
+
|
65
|
+
```apache
|
66
|
+
type redis_store
|
67
|
+
key_path user.name
|
68
|
+
key_prefix ouruser.
|
69
|
+
key_suffix .accesslog
|
70
|
+
```
|
71
|
+
|
72
|
+
With the previous data, _key_ will be `outuser.Kei.accesslog`.
|
73
|
+
|
74
|
+
#### _value_ data for Redis storage
|
75
|
+
|
76
|
+
To determine what _value_ in every event data to be srtored, you have two options;
|
77
|
+
|
78
|
+
1. Store extracted data in event data, by a lookup path with `value_path` option.
|
79
|
+
It works like `key_path`.
|
80
|
+
2. Store whole data.
|
81
|
+
This is default behavior. To do it, simply omit `value_path` option.
|
82
|
+
|
83
|
+
Installation
|
84
|
+
------------
|
85
|
+
|
86
|
+
/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-redis-store
|
87
|
+
|
88
|
+
Configuration
|
89
|
+
-------------
|
90
|
+
|
91
|
+
### Redis connection
|
92
|
+
|
93
|
+
| Key | Type | Required? | Default | Description |
|
94
|
+
| :---- | :----- | :---------- | :----------------------- | :------------ |
|
95
|
+
| `host` | string | Optional | 127.0.0.1 | host name of Redis server |
|
96
|
+
| `port` | int | Optional | 6379 | port number of Redis server |
|
97
|
+
| `password` | string | Optional | | password for Redis connection |
|
98
|
+
| `path` | string | Optional | | To connect via Unix socket, try '/tmp/redis.sock' |
|
99
|
+
| `db` | int | Optional | 0 | DB number of Redis |
|
100
|
+
| `timeout` | float | Optional | 5.0 | connection timeout in seconds |
|
101
|
+
|
102
|
+
### common options for storages
|
103
|
+
|
104
|
+
| Key | Type | Default | Description |
|
105
|
+
| :---- | :----- | :----------------------- | :------------ |
|
106
|
+
| `key` | string | | Fixed _key_ used to store(publish) in Redis |
|
107
|
+
| `key_path` | string | | path to lookup for _key_ in the event data |
|
108
|
+
| `key_prefix` | string | | prefix of _key_ |
|
109
|
+
| `key_suffix` | string | | suffix of _key_ |
|
110
|
+
| `value_path` | string | (whole event data) | path to lookup for _value_ in the event data |
|
111
|
+
| `store_type` | string | zset | `string`/`list`/`set`/`zset`/`publish` |
|
112
|
+
| `format_type` | string | plain | format type for _value_ (`plain`/`json`/`msgpack`) |
|
113
|
+
| `key_expire` | int | -1 | If set, the key will be expired in specified seconds |
|
114
|
+
| `flush_interval` | time | 1 | Time interval which events will be flushed to Redis |
|
115
|
+
| `prevent_duplicate` | (0,1) | 0 | Prevent duplicated value in one key (list)
|
116
|
+
Note: either `key` or `key_path` is required.
|
117
|
+
|
118
|
+
### `string` storage specific options
|
119
|
+
|
120
|
+
| Key | Type | Default | Description |
|
121
|
+
| :---- | :----- | :----------------------- | :------------ |
|
122
|
+
| `type` | string | | Fixed _key_ used to store(publish) in Redis |
|
123
|
+
No more options than common options.
|
124
|
+
|
125
|
+
### `list` storage specific options
|
126
|
+
|
127
|
+
| Key | Type | Default | Description |
|
128
|
+
| :---- | :----- | :----------------------- | :------------ |
|
129
|
+
| `order` | string | asc | `asc`: **rpush**, `desc`: **lpush** |
|
130
|
+
|
131
|
+
### `set` storage specific options
|
132
|
+
|
133
|
+
No more options than common options.
|
134
|
+
|
135
|
+
### `zset` storage specific options
|
136
|
+
|
137
|
+
| Key | Type | Default | Description |
|
138
|
+
| :---- | :----- | :----------------------- | :------------ |
|
139
|
+
| `score_path` | string | (_time_ of log event) | path to lookup for _score_ in the event data |
|
140
|
+
| `value_expire` | int | | value expiration in seconds |
|
141
|
+
|
142
|
+
If `value_expire` is set, the plugin assumes that the _score_ in the **SortedSet** is
|
143
|
+
based on *timestamp* and it deletes expired _members_ every after new event data arrives.
|
144
|
+
|
145
|
+
### `publish` storage specific options
|
146
|
+
|
147
|
+
No more options than common options.
|
148
|
+
|
149
|
+
|
150
|
+
Copyright
|
151
|
+
---------
|
152
|
+
|
153
|
+
Copyright (c) 2013 moaikids
|
154
|
+
Copyright (c) 2014 HANAI Tohru
|
155
|
+
|
156
|
+
License
|
157
|
+
-------
|
158
|
+
Apache License, Version 2.0
|
data/Rakefile
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
Gem::Specification.new do |gem|
|
3
|
+
gem.name = "fluent-plugin-redis-store-wejick"
|
4
|
+
gem.email = "wejick@gmail.com"
|
5
|
+
gem.version = "0.0.1"
|
6
|
+
gem.authors = ["wejick", "Gian Giovani"]
|
7
|
+
gem.licenses = ["Apache License Version 2.0"]
|
8
|
+
gem.summary = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd}
|
9
|
+
gem.description = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd...}
|
10
|
+
gem.homepage = "https://github.com/wejick/fluent-plugin-redis-store"
|
11
|
+
|
12
|
+
gem.files = `git ls-files`.split($\)
|
13
|
+
gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
|
14
|
+
gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
|
15
|
+
gem.require_paths = ["lib"]
|
16
|
+
|
17
|
+
gem.add_development_dependency "rake"
|
18
|
+
gem.add_development_dependency "test-unit"
|
19
|
+
gem.add_runtime_dependency "fluentd"
|
20
|
+
gem.add_runtime_dependency "redis"
|
21
|
+
end
|
@@ -0,0 +1,252 @@
|
|
1
|
+
module Fluent
|
2
|
+
class RedisStoreOutput < BufferedOutput
|
3
|
+
Fluent::Plugin.register_output('redis_store_wejick', self)
|
4
|
+
|
5
|
+
# redis connection
|
6
|
+
config_param :host, :string, :default => '127.0.0.1'
|
7
|
+
config_param :port, :integer, :default => 6379
|
8
|
+
config_param :path, :string, :default => nil
|
9
|
+
config_param :password, :string, :default => nil
|
10
|
+
config_param :db, :integer, :default => 0
|
11
|
+
config_param :timeout, :float, :default => 5.0
|
12
|
+
|
13
|
+
# redis command and parameters
|
14
|
+
config_param :format_type, :string, :default => 'json'
|
15
|
+
config_param :store_type, :string, :default => 'zset'
|
16
|
+
config_param :key_prefix, :string, :default => ''
|
17
|
+
config_param :key_suffix, :string, :default => ''
|
18
|
+
config_param :key, :string, :default => nil
|
19
|
+
config_param :key_path, :string, :default => nil
|
20
|
+
config_param :score_path, :string, :default => nil
|
21
|
+
config_param :value_path, :string, :default => ''
|
22
|
+
config_param :key_expire, :integer, :default => -1
|
23
|
+
config_param :value_expire, :integer, :default => -1
|
24
|
+
config_param :value_length, :integer, :default => -1
|
25
|
+
config_param :order, :string, :default => 'asc'
|
26
|
+
config_param :prevent_duplicate, :integer, :default => 0
|
27
|
+
config_param :only_alphabet,:integer, :default => 0
|
28
|
+
config_param :tidy_string, :integer, :default => 0
|
29
|
+
|
30
|
+
def initialize
|
31
|
+
super
|
32
|
+
require 'redis'
|
33
|
+
require 'msgpack'
|
34
|
+
require 'cgi'
|
35
|
+
end
|
36
|
+
|
37
|
+
def configure(conf)
|
38
|
+
super
|
39
|
+
|
40
|
+
if @key_path == nil and @key == nil
|
41
|
+
raise Fluent::ConfigError, "either key_path or key is required"
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def start
|
46
|
+
super
|
47
|
+
if @path
|
48
|
+
@redis = Redis.new(:path => @path, :password => @password,
|
49
|
+
:timeout => @timeout, :thread_safe => true, :db => @db)
|
50
|
+
else
|
51
|
+
@redis = Redis.new(:host => @host, :port => @port, :password => @password,
|
52
|
+
:timeout => @timeout, :thread_safe => true, :db => @db)
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
def shutdown
|
57
|
+
@redis.quit
|
58
|
+
end
|
59
|
+
|
60
|
+
def format(tag, time, record)
|
61
|
+
[tag, time, record].to_msgpack
|
62
|
+
end
|
63
|
+
|
64
|
+
def write(chunk)
|
65
|
+
@redis.pipelined {
|
66
|
+
chunk.open { |io|
|
67
|
+
begin
|
68
|
+
MessagePack::Unpacker.new(io).each { |message|
|
69
|
+
begin
|
70
|
+
(tag, time, record) = message
|
71
|
+
case @store_type
|
72
|
+
when 'zset'
|
73
|
+
operation_for_zset(record, time)
|
74
|
+
when 'set'
|
75
|
+
operation_for_set(record)
|
76
|
+
when 'list'
|
77
|
+
operation_for_list(record)
|
78
|
+
when 'string'
|
79
|
+
operation_for_string(record)
|
80
|
+
when 'publish'
|
81
|
+
operation_for_publish(record)
|
82
|
+
end
|
83
|
+
rescue NoMethodError => e
|
84
|
+
puts e
|
85
|
+
end
|
86
|
+
}
|
87
|
+
rescue EOFError
|
88
|
+
# EOFError always occured when reached end of chunk.
|
89
|
+
end
|
90
|
+
}
|
91
|
+
}
|
92
|
+
end
|
93
|
+
|
94
|
+
def operation_for_zset(record, time)
|
95
|
+
key = get_key_from(record)
|
96
|
+
value = get_value_from(record)
|
97
|
+
score = get_score_from(record, time)
|
98
|
+
@redis.zadd key, score, value
|
99
|
+
|
100
|
+
set_key_expire key
|
101
|
+
if 0 < @value_expire
|
102
|
+
now = Time.now.to_i
|
103
|
+
@redis.zremrangebyscore key , '-inf' , (now - @value_expire)
|
104
|
+
end
|
105
|
+
if 0 < @value_length
|
106
|
+
script = generate_zremrangebyrank_script(key, @value_length, @order)
|
107
|
+
@redis.eval script
|
108
|
+
end
|
109
|
+
end
|
110
|
+
|
111
|
+
def operation_for_set(record)
|
112
|
+
key = get_key_from(record)
|
113
|
+
value = get_value_from(record)
|
114
|
+
@redis.sadd key, value
|
115
|
+
set_key_expire key
|
116
|
+
end
|
117
|
+
|
118
|
+
def operation_for_list(record)
|
119
|
+
key = get_key_from(record)
|
120
|
+
value = get_value_from(record)
|
121
|
+
|
122
|
+
if 0 < @tidy_string
|
123
|
+
value = tidy_up_string(value)
|
124
|
+
end
|
125
|
+
if 0 < @only_alphabet
|
126
|
+
if ( /^[a-zA-Z0-9 ]*$/.match(value) ) != nil
|
127
|
+
else
|
128
|
+
return
|
129
|
+
end
|
130
|
+
end
|
131
|
+
if 0 < @prevent_duplicate
|
132
|
+
@redis.lrem key.to_s, 1, value.to_s
|
133
|
+
end
|
134
|
+
if @order == 'asc'
|
135
|
+
@redis.rpush key, value
|
136
|
+
else
|
137
|
+
@redis.lpush key, value
|
138
|
+
end
|
139
|
+
set_key_expire key
|
140
|
+
if 0 < @value_length
|
141
|
+
script = generate_ltrim_script(key, @value_length, @order)
|
142
|
+
@redis.eval script
|
143
|
+
end
|
144
|
+
end
|
145
|
+
|
146
|
+
def operation_for_string(record)
|
147
|
+
key = get_key_from(record)
|
148
|
+
value = get_value_from(record)
|
149
|
+
@redis.set key, value
|
150
|
+
|
151
|
+
set_key_expire key
|
152
|
+
end
|
153
|
+
|
154
|
+
def operation_for_publish(record)
|
155
|
+
key = get_key_from(record)
|
156
|
+
value = get_value_from(record)
|
157
|
+
@redis.publish key, value
|
158
|
+
end
|
159
|
+
|
160
|
+
# unescape and make it to downcase
|
161
|
+
def tidy_up_string(string)
|
162
|
+
string.downcase
|
163
|
+
string = CGI.unescape(string)
|
164
|
+
return string
|
165
|
+
end
|
166
|
+
|
167
|
+
def generate_zremrangebyrank_script(key, maxlen, order)
|
168
|
+
script = "local key = '" + key.to_s + "'\n"
|
169
|
+
script += "local maxlen = " + maxlen.to_s + "\n"
|
170
|
+
script += "local order ='" + order.to_s + "'\n"
|
171
|
+
script += "local len = tonumber(redis.call('ZCOUNT', key, '-inf', '+inf'))\n"
|
172
|
+
script += "if len > maxlen then\n"
|
173
|
+
script += " if order == 'asc' then\n"
|
174
|
+
script += " local l = len - maxlen\n"
|
175
|
+
script += " if l >= 0 then\n"
|
176
|
+
script += " return redis.call('ZREMRANGEBYRANK', key, 0, l)\n"
|
177
|
+
script += " end\n"
|
178
|
+
script += " else\n"
|
179
|
+
script += " return redis.call('ZREMRANGEBYRANK', key, maxlen, -1)\n"
|
180
|
+
script += " end\n"
|
181
|
+
script += "end\n"
|
182
|
+
return script
|
183
|
+
end
|
184
|
+
|
185
|
+
def generate_ltrim_script(key, maxlen, order)
|
186
|
+
script = "local key = '" + key.to_s + "'\n"
|
187
|
+
script += "local maxlen = " + maxlen.to_s + "\n"
|
188
|
+
script += "local order ='" + order.to_s + "'\n"
|
189
|
+
script += "local len = tonumber(redis.call('LLEN', key))\n"
|
190
|
+
script += "if len > maxlen then\n"
|
191
|
+
script += " if order == 'asc' then\n"
|
192
|
+
script += " local l = len - maxlen\n"
|
193
|
+
script += " return redis.call('LTRIM', key, l, -1)\n"
|
194
|
+
script += " else\n"
|
195
|
+
script += " return redis.call('LTRIM', key, 0, maxlen - 1)\n"
|
196
|
+
script += " end\n"
|
197
|
+
script += "end\n"
|
198
|
+
return script
|
199
|
+
end
|
200
|
+
|
201
|
+
def traverse(data, key)
|
202
|
+
val = data
|
203
|
+
key.split('.').each{ |k|
|
204
|
+
if val.has_key?(k)
|
205
|
+
val = val[k]
|
206
|
+
else
|
207
|
+
return nil
|
208
|
+
end
|
209
|
+
}
|
210
|
+
return val
|
211
|
+
end
|
212
|
+
|
213
|
+
def get_key_from(record)
|
214
|
+
if @key
|
215
|
+
k = @key
|
216
|
+
else
|
217
|
+
k = traverse(record, @key_path).to_s
|
218
|
+
end
|
219
|
+
key = @key_prefix + k + @key_suffix
|
220
|
+
|
221
|
+
raise Fluent::ConfigError, "key is empty" if key == ''
|
222
|
+
key
|
223
|
+
end
|
224
|
+
|
225
|
+
def get_value_from(record)
|
226
|
+
value = traverse(record, @value_path)
|
227
|
+
case @format_type
|
228
|
+
when 'json'
|
229
|
+
value.to_json
|
230
|
+
when 'msgpack'
|
231
|
+
value.to_msgpack
|
232
|
+
else
|
233
|
+
value
|
234
|
+
end
|
235
|
+
end
|
236
|
+
|
237
|
+
def get_score_from(record, time)
|
238
|
+
if @score_path
|
239
|
+
traverse(record, @score_path)
|
240
|
+
else
|
241
|
+
time
|
242
|
+
end
|
243
|
+
end
|
244
|
+
|
245
|
+
def set_key_expire(key)
|
246
|
+
if 0 < @key_expire
|
247
|
+
@redis.expire key, @key_expire
|
248
|
+
end
|
249
|
+
end
|
250
|
+
|
251
|
+
end
|
252
|
+
end
|
data/test/helpers.rb
ADDED
@@ -0,0 +1,28 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'bundler'
|
3
|
+
begin
|
4
|
+
Bundler.setup(:default, :development)
|
5
|
+
rescue Bundler::BundlerError => e
|
6
|
+
$stderr.puts e.message
|
7
|
+
$stderr.puts "Run `bundle install` to install missing gems"
|
8
|
+
exit e.status_code
|
9
|
+
end
|
10
|
+
require 'test/unit'
|
11
|
+
|
12
|
+
$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
|
13
|
+
$LOAD_PATH.unshift(File.dirname(__FILE__))
|
14
|
+
require 'fluent/test'
|
15
|
+
unless ENV.has_key?('VERBOSE')
|
16
|
+
nulllogger = Object.new
|
17
|
+
nulllogger.instance_eval {|obj|
|
18
|
+
def method_missing(method, *args)
|
19
|
+
# pass
|
20
|
+
end
|
21
|
+
}
|
22
|
+
$log = nulllogger
|
23
|
+
end
|
24
|
+
|
25
|
+
require 'fluent/plugin/out_redis_store'
|
26
|
+
|
27
|
+
class Test::Unit::TestCase
|
28
|
+
end
|
@@ -0,0 +1,367 @@
|
|
1
|
+
require 'helpers'
|
2
|
+
|
3
|
+
require 'redis'
|
4
|
+
|
5
|
+
$channel = nil
|
6
|
+
$message = nil
|
7
|
+
|
8
|
+
class Redis
|
9
|
+
def initialize(options = {})
|
10
|
+
end
|
11
|
+
|
12
|
+
def pipelined
|
13
|
+
yield self
|
14
|
+
end
|
15
|
+
|
16
|
+
def set(key, message)
|
17
|
+
$command = :set
|
18
|
+
$key = key
|
19
|
+
$message = message
|
20
|
+
end
|
21
|
+
|
22
|
+
def rpush(key, message)
|
23
|
+
$command = :rpush
|
24
|
+
$key = key
|
25
|
+
$message = message
|
26
|
+
end
|
27
|
+
|
28
|
+
def lpush(key, message)
|
29
|
+
$command = :lpush
|
30
|
+
$key = key
|
31
|
+
$message = message
|
32
|
+
end
|
33
|
+
|
34
|
+
def sadd(key, message)
|
35
|
+
$command = :sadd
|
36
|
+
$key = key
|
37
|
+
$message = message
|
38
|
+
end
|
39
|
+
|
40
|
+
def zadd(key, score, message)
|
41
|
+
$command = :zadd
|
42
|
+
$key = key
|
43
|
+
$score = score
|
44
|
+
$message = message
|
45
|
+
end
|
46
|
+
|
47
|
+
def expire(key, ttl)
|
48
|
+
$expire_key = key
|
49
|
+
$ttl = ttl
|
50
|
+
end
|
51
|
+
|
52
|
+
def publish(channel, message)
|
53
|
+
$command = :publish
|
54
|
+
$channel = channel
|
55
|
+
$message = message
|
56
|
+
end
|
57
|
+
|
58
|
+
def quit
|
59
|
+
end
|
60
|
+
end
|
61
|
+
|
62
|
+
class RedisStoreOutputTest < Test::Unit::TestCase
|
63
|
+
def setup
|
64
|
+
Fluent::Test.setup
|
65
|
+
end
|
66
|
+
|
67
|
+
def create_driver(conf)
|
68
|
+
Fluent::Test::BufferedOutputTestDriver.new(Fluent::RedisStoreOutput).configure(conf)
|
69
|
+
end
|
70
|
+
|
71
|
+
def test_configure_defaults
|
72
|
+
config = %[
|
73
|
+
key_path a
|
74
|
+
score_path b
|
75
|
+
]
|
76
|
+
d = create_driver(config)
|
77
|
+
assert_equal "127.0.0.1", d.instance.host
|
78
|
+
assert_equal 6379, d.instance.port
|
79
|
+
assert_equal nil, d.instance.path
|
80
|
+
assert_equal nil, d.instance.password
|
81
|
+
assert_equal 0, d.instance.db
|
82
|
+
assert_equal 5.0, d.instance.timeout
|
83
|
+
assert_equal 'json', d.instance.format_type
|
84
|
+
assert_equal '', d.instance.key_prefix
|
85
|
+
assert_equal '', d.instance.key_suffix
|
86
|
+
assert_equal 'zset', d.instance.store_type
|
87
|
+
assert_equal 'a', d.instance.key_path
|
88
|
+
assert_equal nil, d.instance.key
|
89
|
+
assert_equal 'b', d.instance.score_path
|
90
|
+
assert_equal '', d.instance.value_path
|
91
|
+
assert_equal -1, d.instance.key_expire
|
92
|
+
assert_equal -1, d.instance.value_expire
|
93
|
+
assert_equal -1, d.instance.value_length
|
94
|
+
assert_equal 'asc', d.instance.order
|
95
|
+
end
|
96
|
+
|
97
|
+
def test_configure_host_port_db
|
98
|
+
config = %[
|
99
|
+
host 192.168.2.3
|
100
|
+
port 9999
|
101
|
+
password abc
|
102
|
+
db 3
|
103
|
+
timeout 7
|
104
|
+
key a
|
105
|
+
score_path b
|
106
|
+
]
|
107
|
+
d = create_driver(config)
|
108
|
+
assert_equal "192.168.2.3", d.instance.host
|
109
|
+
assert_equal 9999, d.instance.port
|
110
|
+
assert_equal nil, d.instance.path
|
111
|
+
assert_equal 'abc', d.instance.password
|
112
|
+
assert_equal 3, d.instance.db
|
113
|
+
assert_equal 7.0, d.instance.timeout
|
114
|
+
assert_equal nil, d.instance.key_path
|
115
|
+
assert_equal 'a', d.instance.key
|
116
|
+
end
|
117
|
+
|
118
|
+
def test_configure_path
|
119
|
+
config = %[
|
120
|
+
path /tmp/foo.sock
|
121
|
+
key a
|
122
|
+
score_path b
|
123
|
+
]
|
124
|
+
d = create_driver(config)
|
125
|
+
assert_equal "/tmp/foo.sock", d.instance.path
|
126
|
+
end
|
127
|
+
|
128
|
+
def test_configure_exception
|
129
|
+
assert_raise(Fluent::ConfigError) do
|
130
|
+
create_driver(%[])
|
131
|
+
end
|
132
|
+
end
|
133
|
+
|
134
|
+
# def test_write
|
135
|
+
# d = create_driver(CONFIG1)
|
136
|
+
#
|
137
|
+
# time = Time.parse("2011-01-02 13:14:15 UTC").to_i
|
138
|
+
# d.emit({ "foo" => "bar" }, time)
|
139
|
+
# d.run
|
140
|
+
#
|
141
|
+
# assert_equal "test", $channel
|
142
|
+
# assert_equal(%Q[{"foo":"bar","time":#{time}}], $message)
|
143
|
+
# end
|
144
|
+
|
145
|
+
def get_time
|
146
|
+
Time.parse("2011-01-02 13:14:15 UTC").to_i
|
147
|
+
end
|
148
|
+
|
149
|
+
# it should return whole message
|
150
|
+
def test_omit_value_path
|
151
|
+
config = %[
|
152
|
+
format_type plain
|
153
|
+
store_type string
|
154
|
+
key_path user
|
155
|
+
]
|
156
|
+
d = create_driver(config)
|
157
|
+
message = {
|
158
|
+
'user' => 'george',
|
159
|
+
'stat' => { 'attack' => 7 }
|
160
|
+
}
|
161
|
+
$ttl = nil
|
162
|
+
d.emit(message, get_time)
|
163
|
+
d.run
|
164
|
+
|
165
|
+
assert_equal "george", $key
|
166
|
+
assert_equal message, $message
|
167
|
+
assert_equal nil, $ttl
|
168
|
+
end
|
169
|
+
|
170
|
+
def test_key_value_paths
|
171
|
+
config = %[
|
172
|
+
format_type plain
|
173
|
+
store_type string
|
174
|
+
key_path user.name
|
175
|
+
value_path stat.attack
|
176
|
+
key_expire 3
|
177
|
+
]
|
178
|
+
d = create_driver(config)
|
179
|
+
message = {
|
180
|
+
'user' => { 'name' => 'george' },
|
181
|
+
'stat' => { 'attack' => 7 }
|
182
|
+
}
|
183
|
+
$ttl = nil
|
184
|
+
d.emit(message, get_time)
|
185
|
+
d.run
|
186
|
+
|
187
|
+
assert_equal "george", $key
|
188
|
+
assert_equal 7, $message
|
189
|
+
assert_equal 3, $ttl
|
190
|
+
end
|
191
|
+
|
192
|
+
def test_json
|
193
|
+
config = %[
|
194
|
+
format_type json
|
195
|
+
store_type string
|
196
|
+
key_path user
|
197
|
+
]
|
198
|
+
d = create_driver(config)
|
199
|
+
message = {
|
200
|
+
'user' => 'george',
|
201
|
+
'stat' => { 'attack' => 7 }
|
202
|
+
}
|
203
|
+
$ttl = nil
|
204
|
+
d.emit(message, get_time)
|
205
|
+
d.run
|
206
|
+
|
207
|
+
assert_equal "george", $key
|
208
|
+
assert_equal message.to_json, $message
|
209
|
+
end
|
210
|
+
|
211
|
+
def test_msgpack
|
212
|
+
config = %[
|
213
|
+
format_type msgpack
|
214
|
+
store_type string
|
215
|
+
key_path user
|
216
|
+
]
|
217
|
+
d = create_driver(config)
|
218
|
+
message = {
|
219
|
+
'user' => 'george',
|
220
|
+
'stat' => { 'attack' => 7 }
|
221
|
+
}
|
222
|
+
$ttl = nil
|
223
|
+
d.emit(message, get_time)
|
224
|
+
d.run
|
225
|
+
|
226
|
+
assert_equal "george", $key
|
227
|
+
assert_equal message.to_msgpack, $message
|
228
|
+
end
|
229
|
+
|
230
|
+
def test_list_asc
|
231
|
+
config = %[
|
232
|
+
format_type plain
|
233
|
+
store_type list
|
234
|
+
key_path user
|
235
|
+
]
|
236
|
+
d = create_driver(config)
|
237
|
+
message = {
|
238
|
+
'user' => 'george',
|
239
|
+
'stat' => { 'attack' => 7 }
|
240
|
+
}
|
241
|
+
d.emit(message, get_time)
|
242
|
+
d.run
|
243
|
+
|
244
|
+
assert_equal :rpush, $command
|
245
|
+
assert_equal "george", $key
|
246
|
+
assert_equal message, $message
|
247
|
+
end
|
248
|
+
|
249
|
+
def test_list_desc
|
250
|
+
config = %[
|
251
|
+
format_type plain
|
252
|
+
store_type list
|
253
|
+
key_path user
|
254
|
+
order desc
|
255
|
+
]
|
256
|
+
d = create_driver(config)
|
257
|
+
message = {
|
258
|
+
'user' => 'george',
|
259
|
+
'stat' => { 'attack' => 7 }
|
260
|
+
}
|
261
|
+
d.emit(message, get_time)
|
262
|
+
d.run
|
263
|
+
|
264
|
+
assert_equal :lpush, $command
|
265
|
+
assert_equal "george", $key
|
266
|
+
assert_equal message, $message
|
267
|
+
end
|
268
|
+
|
269
|
+
def test_set
|
270
|
+
config = %[
|
271
|
+
format_type plain
|
272
|
+
store_type set
|
273
|
+
key_path user
|
274
|
+
order desc
|
275
|
+
]
|
276
|
+
d = create_driver(config)
|
277
|
+
message = {
|
278
|
+
'user' => 'george',
|
279
|
+
'stat' => { 'attack' => 7 }
|
280
|
+
}
|
281
|
+
d.emit(message, get_time)
|
282
|
+
d.run
|
283
|
+
|
284
|
+
assert_equal :sadd, $command
|
285
|
+
assert_equal "george", $key
|
286
|
+
assert_equal message, $message
|
287
|
+
end
|
288
|
+
|
289
|
+
def test_zset
|
290
|
+
config = %[
|
291
|
+
format_type plain
|
292
|
+
store_type zset
|
293
|
+
key_path user
|
294
|
+
score_path result
|
295
|
+
]
|
296
|
+
|
297
|
+
d = create_driver(config)
|
298
|
+
message = {
|
299
|
+
'user' => 'george',
|
300
|
+
'stat' => { 'attack' => 7 },
|
301
|
+
'result' => 81
|
302
|
+
}
|
303
|
+
d.emit(message, get_time)
|
304
|
+
d.run
|
305
|
+
|
306
|
+
assert_equal :zadd, $command
|
307
|
+
assert_equal "george", $key
|
308
|
+
assert_equal 81, $score
|
309
|
+
assert_equal message, $message
|
310
|
+
end
|
311
|
+
|
312
|
+
def test_zset_with_no_score_path
|
313
|
+
config = %[
|
314
|
+
format_type plain
|
315
|
+
store_type zset
|
316
|
+
key_path user
|
317
|
+
]
|
318
|
+
|
319
|
+
d = create_driver(config)
|
320
|
+
message = {
|
321
|
+
'user' => 'george',
|
322
|
+
'stat' => { 'attack' => 7 },
|
323
|
+
'result' => 81
|
324
|
+
}
|
325
|
+
d.emit(message, get_time)
|
326
|
+
d.run
|
327
|
+
|
328
|
+
assert_equal :zadd, $command
|
329
|
+
assert_equal "george", $key
|
330
|
+
assert_equal get_time, $score
|
331
|
+
assert_equal message, $message
|
332
|
+
end
|
333
|
+
|
334
|
+
def test_publish
|
335
|
+
config = %[
|
336
|
+
format_type plain
|
337
|
+
store_type publish
|
338
|
+
key_path user
|
339
|
+
]
|
340
|
+
|
341
|
+
d = create_driver(config)
|
342
|
+
message = {
|
343
|
+
'user' => 'george'
|
344
|
+
}
|
345
|
+
d.emit(message, get_time)
|
346
|
+
d.run
|
347
|
+
|
348
|
+
assert_equal :publish, $command
|
349
|
+
assert_equal "george", $channel
|
350
|
+
assert_equal message, $message
|
351
|
+
end
|
352
|
+
|
353
|
+
def test_empty_key
|
354
|
+
config = %[
|
355
|
+
format_type plain
|
356
|
+
store_type string
|
357
|
+
key_path none
|
358
|
+
]
|
359
|
+
|
360
|
+
d = create_driver(config)
|
361
|
+
message = { 'user' => 'george' }
|
362
|
+
d.emit(message, get_time)
|
363
|
+
assert_raise(Fluent::ConfigError) do
|
364
|
+
d.run
|
365
|
+
end
|
366
|
+
end
|
367
|
+
end
|
metadata
ADDED
@@ -0,0 +1,110 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: fluent-plugin-redis-store-wejick
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.0.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- wejick
|
8
|
+
- Gian Giovani
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2016-08-02 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: rake
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
|
+
requirements:
|
18
|
+
- - ">="
|
19
|
+
- !ruby/object:Gem::Version
|
20
|
+
version: '0'
|
21
|
+
type: :development
|
22
|
+
prerelease: false
|
23
|
+
version_requirements: !ruby/object:Gem::Requirement
|
24
|
+
requirements:
|
25
|
+
- - ">="
|
26
|
+
- !ruby/object:Gem::Version
|
27
|
+
version: '0'
|
28
|
+
- !ruby/object:Gem::Dependency
|
29
|
+
name: test-unit
|
30
|
+
requirement: !ruby/object:Gem::Requirement
|
31
|
+
requirements:
|
32
|
+
- - ">="
|
33
|
+
- !ruby/object:Gem::Version
|
34
|
+
version: '0'
|
35
|
+
type: :development
|
36
|
+
prerelease: false
|
37
|
+
version_requirements: !ruby/object:Gem::Requirement
|
38
|
+
requirements:
|
39
|
+
- - ">="
|
40
|
+
- !ruby/object:Gem::Version
|
41
|
+
version: '0'
|
42
|
+
- !ruby/object:Gem::Dependency
|
43
|
+
name: fluentd
|
44
|
+
requirement: !ruby/object:Gem::Requirement
|
45
|
+
requirements:
|
46
|
+
- - ">="
|
47
|
+
- !ruby/object:Gem::Version
|
48
|
+
version: '0'
|
49
|
+
type: :runtime
|
50
|
+
prerelease: false
|
51
|
+
version_requirements: !ruby/object:Gem::Requirement
|
52
|
+
requirements:
|
53
|
+
- - ">="
|
54
|
+
- !ruby/object:Gem::Version
|
55
|
+
version: '0'
|
56
|
+
- !ruby/object:Gem::Dependency
|
57
|
+
name: redis
|
58
|
+
requirement: !ruby/object:Gem::Requirement
|
59
|
+
requirements:
|
60
|
+
- - ">="
|
61
|
+
- !ruby/object:Gem::Version
|
62
|
+
version: '0'
|
63
|
+
type: :runtime
|
64
|
+
prerelease: false
|
65
|
+
version_requirements: !ruby/object:Gem::Requirement
|
66
|
+
requirements:
|
67
|
+
- - ">="
|
68
|
+
- !ruby/object:Gem::Version
|
69
|
+
version: '0'
|
70
|
+
description: Redis(zset/set/list/string/publish) output plugin for Fluentd...
|
71
|
+
email: wejick@gmail.com
|
72
|
+
executables: []
|
73
|
+
extensions: []
|
74
|
+
extra_rdoc_files: []
|
75
|
+
files:
|
76
|
+
- AUTHORS
|
77
|
+
- Gemfile
|
78
|
+
- README.md
|
79
|
+
- Rakefile
|
80
|
+
- fluent-plugin-redis-store-wejick.gemspec
|
81
|
+
- lib/fluent/plugin/out_redis_store_wejick.rb
|
82
|
+
- test/helpers.rb
|
83
|
+
- test/plugin/test_out_redis_publish.rb
|
84
|
+
homepage: https://github.com/wejick/fluent-plugin-redis-store
|
85
|
+
licenses:
|
86
|
+
- Apache License Version 2.0
|
87
|
+
metadata: {}
|
88
|
+
post_install_message:
|
89
|
+
rdoc_options: []
|
90
|
+
require_paths:
|
91
|
+
- lib
|
92
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
98
|
+
requirements:
|
99
|
+
- - ">="
|
100
|
+
- !ruby/object:Gem::Version
|
101
|
+
version: '0'
|
102
|
+
requirements: []
|
103
|
+
rubyforge_project:
|
104
|
+
rubygems_version: 2.5.1
|
105
|
+
signing_key:
|
106
|
+
specification_version: 4
|
107
|
+
summary: Redis(zset/set/list/string/publish) output plugin for Fluentd
|
108
|
+
test_files:
|
109
|
+
- test/helpers.rb
|
110
|
+
- test/plugin/test_out_redis_publish.rb
|