fluent-plugin-redis-store-seldon 0.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: b56cd9a0f6e04be5cc93536aa4329bf34843e1bc
4
+ data.tar.gz: f7e20910059d09cfb33551c2a07de1fcf4c65500
5
+ SHA512:
6
+ metadata.gz: 7409a2407f6f350b518e3efa82be8fd1b04d501c9e13a6dbb79a43bc9fc725711cb1975b3ca7fd5246ac3ed933eefe701485bbafee1fd9708df5498ee11a6b3c
7
+ data.tar.gz: 259872bd9c4aea59ff6a75e7efbe1956be8a9ca043bb427861b5929bebdd6ffed8a6be7f468a639b5f2305253cae6bb1441d61647efdc9d2c6b0c66bb23ab836
data/AUTHORS ADDED
@@ -0,0 +1,2 @@
1
+ moaikids
2
+ HANAI Tohru aka pokehanai
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source "http://rubygems.org"
2
+
3
+ gemspec
data/README.md ADDED
@@ -0,0 +1,161 @@
1
+ Fork of https://github.com/pokehanai/fluent-plugin-redis-store
2
+
3
+ Redis Output Plugin For fluentd
4
+ ===============================
5
+ [Fluentd][] output plugin to upload/publish event data to [Redis][] storage.
6
+
7
+ [Fluentd]: http://fluentd.org/
8
+ [Redis]: http://redis.io/
9
+
10
+ Background
11
+ ----------
12
+
13
+ This is folked project from [fluent-plugin-redisstore][].
14
+
15
+ [fluent-plugin-redisstore]: https://github.com/moaikids/fluent-plugin-redisstore
16
+
17
+ Features
18
+ --------
19
+
20
+ #### Supported Redis commands
21
+
22
+ Currently the plugin supports following Redis commands:
23
+
24
+ - **set** by `string` type (of the plugin)
25
+ - **lpush**/**rpush** by `list` type
26
+ - **sadd** by `set` type
27
+ - **zadd** by `zset` type
28
+ - **publish** by `publish` type
29
+
30
+ #### Supported _value_ format
31
+
32
+ - plain(as is)
33
+ - JSON
34
+ - [MessagePack](http://msgpack.org/)
35
+
36
+ #### _key_ string for Redis storage
37
+
38
+ Redis commands require _key_ and _value_.
39
+ For _key_, the plugin supports either way;
40
+
41
+ 1. Specify a fixed key.
42
+ You can do this simply using `key` option in td-agent configuration file.
43
+
44
+ ```apache
45
+ type redis_store
46
+ key userdata
47
+ ```
48
+
49
+ 2. Lookup a key string in every event data by a lookup path.
50
+ If event data have structured data like
51
+
52
+ ```javascript
53
+ { "user": { "name": "Kei" } }
54
+ ```
55
+
56
+ and you want to use each name of user, you can use `key_path` option.
57
+
58
+ ```apache
59
+ type redis_store
60
+ key_path user.name
61
+ ```
62
+
63
+ With the above data, `Kei` will be a _key_.
64
+
65
+ In addition, `key_prefix` and `key_suffix` are useful in some cases. Both are available either `key` and `key_path`
66
+
67
+ ```apache
68
+ type redis_store
69
+ key_path user.name
70
+ key_prefix ouruser.
71
+ key_suffix .accesslog
72
+ ```
73
+
74
+ With the previous data, _key_ will be `outuser.Kei.accesslog`.
75
+
76
+ #### _value_ data for Redis storage
77
+
78
+ To determine what _value_ in every event data to be srtored, you have two options;
79
+
80
+ 1. Store extracted data in event data, by a lookup path with `value_path` option.
81
+ It works like `key_path`.
82
+ 2. Store whole data.
83
+ This is default behavior. To do it, simply omit `value_path` option.
84
+
85
+ Installation
86
+ ------------
87
+
88
+ /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-redis-store
89
+
90
+ Configuration
91
+ -------------
92
+
93
+ ### Redis connection
94
+
95
+ | Key | Type | Required? | Default | Description |
96
+ | :---- | :----- | :---------- | :----------------------- | :------------ |
97
+ | `host` | string | Optional | 127.0.0.1 | host name of Redis server |
98
+ | `port` | int | Optional | 6379 | port number of Redis server |
99
+ | `password` | string | Optional | | password for Redis connection |
100
+ | `path` | string | Optional | | To connect via Unix socket, try '/tmp/redis.sock' |
101
+ | `db` | int | Optional | 0 | DB number of Redis |
102
+ | `timeout` | float | Optional | 5.0 | connection timeout in seconds |
103
+
104
+ ### common options for storages
105
+
106
+ | Key | Type | Default | Description |
107
+ | :---- | :----- | :----------------------- | :------------ |
108
+ | `key` | string | | Fixed _key_ used to store(publish) in Redis |
109
+ | `key_path` | string | | path to lookup for _key_ in the event data |
110
+ | `key_prefix` | string | | prefix of _key_ |
111
+ | `key_prefix_sep` | string | | separator to put after prefix _key_ |
112
+ | `key_prefix_path` | string | | path to lookup prefix of _key_ |
113
+ | `key_suffix` | string | | suffix of _key_ |
114
+ | `value_path` | string | (whole event data) | path to lookup for _value_ in the event data |
115
+ | `store_type` | string | zset | `string`/`list`/`set`/`zset`/`publish` |
116
+ | `format_type` | string | plain | format type for _value_ (`plain`/`json`/`msgpack`) |
117
+ | `key_expire` | int | -1 | If set, the key will be expired in specified seconds |
118
+
119
+ Note: either `key` or `key_path` is required.
120
+
121
+ ### `string` storage specific options
122
+
123
+ | Key | Type | Default | Description |
124
+ | :---- | :----- | :----------------------- | :------------ |
125
+ | `type` | string | | Fixed _key_ used to store(publish) in Redis |
126
+ No more options than common options.
127
+
128
+ ### `list` storage specific options
129
+
130
+ | Key | Type | Default | Description |
131
+ | :---- | :----- | :----------------------- | :------------ |
132
+ | `order` | string | asc | `asc`: **rpush**, `desc`: **lpush** |
133
+
134
+ ### `set` storage specific options
135
+
136
+ No more options than common options.
137
+
138
+ ### `zset` storage specific options
139
+
140
+ | Key | Type | Default | Description |
141
+ | :---- | :----- | :----------------------- | :------------ |
142
+ | `score_path` | string | (_time_ of log event) | path to lookup for _score_ in the event data |
143
+ | `value_expire` | int | | value expiration in seconds |
144
+
145
+ If `value_expire` is set, the plugin assumes that the _score_ in the **SortedSet** is
146
+ based on *timestamp* and it deletes expired _members_ every after new event data arrives.
147
+
148
+ ### `publish` storage specific options
149
+
150
+ No more options than common options.
151
+
152
+
153
+ Copyright
154
+ ---------
155
+
156
+ Copyright (c) 2013 moaikids
157
+ Copyright (c) 2014 HANAI Tohru
158
+
159
+ License
160
+ -------
161
+ Apache License, Version 2.0
data/Rakefile ADDED
@@ -0,0 +1,11 @@
1
+ #!/usr/bin/env rake
2
+ require "bundler/gem_tasks"
3
+
4
+ require 'rake/testtask'
5
+ Rake::TestTask.new(:test) do |test|
6
+ test.libs << 'lib' << 'test'
7
+ test.pattern = 'test/**/test_*.rb'
8
+ test.verbose = true
9
+ end
10
+
11
+ task :default => :test
@@ -0,0 +1,21 @@
1
+ # -*- encoding: utf-8 -*-
2
+ Gem::Specification.new do |gem|
3
+ gem.name = "fluent-plugin-redis-store-seldon"
4
+ gem.email = "cc@seldon.io"
5
+ gem.version = "0.1"
6
+ gem.authors = ["Clive Cox"]
7
+ gem.licenses = ["Apache License Version 2.0"]
8
+ gem.summary = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd}
9
+ gem.description = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd...}
10
+ gem.homepage = "https://github.com/SeldonIO/fluent-plugin-redis-store"
11
+
12
+ gem.files = `git ls-files`.split($\)
13
+ gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
14
+ gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
15
+ gem.require_paths = ["lib"]
16
+
17
+ gem.add_development_dependency "rake"
18
+ gem.add_development_dependency "test-unit"
19
+ gem.add_runtime_dependency "fluentd"
20
+ gem.add_runtime_dependency "redis"
21
+ end
@@ -0,0 +1,238 @@
1
+ module Fluent
2
+ class RedisStoreOutput < BufferedOutput
3
+ Fluent::Plugin.register_output('redis_store_seldon', self)
4
+
5
+ # redis connection
6
+ config_param :host, :string, :default => '127.0.0.1'
7
+ config_param :port, :integer, :default => 6379
8
+ config_param :path, :string, :default => nil
9
+ config_param :password, :string, :default => nil
10
+ config_param :db, :integer, :default => 0
11
+ config_param :timeout, :float, :default => 5.0
12
+
13
+ # redis command and parameters
14
+ config_param :format_type, :string, :default => 'json'
15
+ config_param :store_type, :string, :default => 'zset'
16
+ config_param :key_prefix, :string, :default => ''
17
+ config_param :key_prefix_path, :string, :default => nil
18
+ config_param :key_prefix_sep, :string, :default => ''
19
+ config_param :key_suffix, :string, :default => ''
20
+ config_param :key, :string, :default => nil
21
+ config_param :key_path, :string, :default => nil
22
+ config_param :score_path, :string, :default => nil
23
+ config_param :value_path, :string, :default => ''
24
+ config_param :key_expire, :integer, :default => -1
25
+ config_param :value_expire, :integer, :default => -1
26
+ config_param :value_length, :integer, :default => -1
27
+ config_param :order, :string, :default => 'asc'
28
+
29
+ def initialize
30
+ super
31
+ require 'redis'
32
+ require 'msgpack'
33
+ end
34
+
35
+ def configure(conf)
36
+ super
37
+
38
+ if @key_path == nil and @key == nil
39
+ raise Fluent::ConfigError, "either key_path or key is required"
40
+ end
41
+ end
42
+
43
+ def start
44
+ super
45
+ if @path
46
+ @redis = Redis.new(:path => @path, :password => @password,
47
+ :timeout => @timeout, :thread_safe => true, :db => @db)
48
+ else
49
+ @redis = Redis.new(:host => @host, :port => @port, :password => @password,
50
+ :timeout => @timeout, :thread_safe => true, :db => @db)
51
+ end
52
+ end
53
+
54
+ def shutdown
55
+ @redis.quit
56
+ end
57
+
58
+ def format(tag, time, record)
59
+ [tag, time, record].to_msgpack
60
+ end
61
+
62
+ def write(chunk)
63
+ @redis.pipelined {
64
+ chunk.open { |io|
65
+ begin
66
+ MessagePack::Unpacker.new(io).each { |message|
67
+ begin
68
+ (tag, time, record) = message
69
+ case @store_type
70
+ when 'zset'
71
+ operation_for_zset(record, time)
72
+ when 'set'
73
+ operation_for_set(record)
74
+ when 'list'
75
+ operation_for_list(record)
76
+ when 'string'
77
+ operation_for_string(record)
78
+ when 'publish'
79
+ operation_for_publish(record)
80
+ end
81
+ rescue NoMethodError => e
82
+ puts e
83
+ end
84
+ }
85
+ rescue EOFError
86
+ # EOFError always occured when reached end of chunk.
87
+ end
88
+ }
89
+ }
90
+ end
91
+
92
+ def operation_for_zset(record, time)
93
+ key = get_key_from(record)
94
+ value = get_value_from(record)
95
+ score = get_score_from(record, time)
96
+ @redis.zadd key, score, value
97
+
98
+ set_key_expire key
99
+ if 0 < @value_expire
100
+ now = Time.now.to_i
101
+ @redis.zremrangebyscore key , '-inf' , (now - @value_expire)
102
+ end
103
+ if 0 < @value_length
104
+ script = generate_zremrangebyrank_script(key, @value_length, @order)
105
+ @redis.eval script
106
+ end
107
+ end
108
+
109
+ def operation_for_set(record)
110
+ key = get_key_from(record)
111
+ value = get_value_from(record)
112
+ @redis.sadd key, value
113
+ set_key_expire key
114
+ end
115
+
116
+ def operation_for_list(record)
117
+ key = get_key_from(record)
118
+ value = get_value_from(record)
119
+
120
+ if @order == 'asc'
121
+ @redis.rpush key, value
122
+ else
123
+ @redis.lpush key, value
124
+ end
125
+ set_key_expire key
126
+ if 0 < @value_length
127
+ script = generate_ltrim_script(key, @value_length, @order)
128
+ @redis.eval script
129
+ end
130
+ end
131
+
132
+ def operation_for_string(record)
133
+ key = get_key_from(record)
134
+ value = get_value_from(record)
135
+ @redis.set key, value
136
+
137
+ set_key_expire key
138
+ end
139
+
140
+ def operation_for_publish(record)
141
+ key = get_key_from(record)
142
+ value = get_value_from(record)
143
+ @redis.publish key, value
144
+ end
145
+
146
+ def generate_zremrangebyrank_script(key, maxlen, order)
147
+ script = "local key = '" + key.to_s + "'\n"
148
+ script += "local maxlen = " + maxlen.to_s + "\n"
149
+ script += "local order ='" + order.to_s + "'\n"
150
+ script += "local len = tonumber(redis.call('ZCOUNT', key, '-inf', '+inf'))\n"
151
+ script += "if len > maxlen then\n"
152
+ script += " if order == 'asc' then\n"
153
+ script += " local l = len - maxlen\n"
154
+ script += " if l >= 0 then\n"
155
+ script += " return redis.call('ZREMRANGEBYRANK', key, 0, l)\n"
156
+ script += " end\n"
157
+ script += " else\n"
158
+ script += " return redis.call('ZREMRANGEBYRANK', key, maxlen, -1)\n"
159
+ script += " end\n"
160
+ script += "end\n"
161
+ return script
162
+ end
163
+
164
+ def generate_ltrim_script(key, maxlen, order)
165
+ script = "local key = '" + key.to_s + "'\n"
166
+ script += "local maxlen = " + maxlen.to_s + "\n"
167
+ script += "local order ='" + order.to_s + "'\n"
168
+ script += "local len = tonumber(redis.call('LLEN', key))\n"
169
+ script += "if len > maxlen then\n"
170
+ script += " if order == 'asc' then\n"
171
+ script += " local l = len - maxlen\n"
172
+ script += " return redis.call('LTRIM', key, l, -1)\n"
173
+ script += " else\n"
174
+ script += " return redis.call('LTRIM', key, 0, maxlen - 1)\n"
175
+ script += " end\n"
176
+ script += "end\n"
177
+ return script
178
+ end
179
+
180
+ def traverse(data, key)
181
+ val = data
182
+ key.split('.').each{ |k|
183
+ if val.has_key?(k)
184
+ val = val[k]
185
+ else
186
+ return nil
187
+ end
188
+ }
189
+ return val
190
+ end
191
+
192
+ def get_key_from(record)
193
+ if @key
194
+ k = @key
195
+ else
196
+ k = traverse(record, @key_path).to_s
197
+ end
198
+
199
+ if @key_prefix_path
200
+ prefix = traverse(record, @key_prefix_path).to_s
201
+ else
202
+ prefix = ''
203
+ end
204
+
205
+ key = @key_prefix + prefix + @key_prefix_sep + k + @key_suffix
206
+
207
+ raise Fluent::ConfigError, "key is empty" if key == ''
208
+ key
209
+ end
210
+
211
+ def get_value_from(record)
212
+ value = traverse(record, @value_path)
213
+ case @format_type
214
+ when 'json'
215
+ value.to_json
216
+ when 'msgpack'
217
+ value.to_msgpack
218
+ else
219
+ value
220
+ end
221
+ end
222
+
223
+ def get_score_from(record, time)
224
+ if @score_path
225
+ traverse(record, @score_path)
226
+ else
227
+ time
228
+ end
229
+ end
230
+
231
+ def set_key_expire(key)
232
+ if 0 < @key_expire
233
+ @redis.expire key, @key_expire
234
+ end
235
+ end
236
+
237
+ end
238
+ end
data/test/helpers.rb ADDED
@@ -0,0 +1,28 @@
1
+ require 'rubygems'
2
+ require 'bundler'
3
+ begin
4
+ Bundler.setup(:default, :development)
5
+ rescue Bundler::BundlerError => e
6
+ $stderr.puts e.message
7
+ $stderr.puts "Run `bundle install` to install missing gems"
8
+ exit e.status_code
9
+ end
10
+ require 'test/unit'
11
+
12
+ $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
13
+ $LOAD_PATH.unshift(File.dirname(__FILE__))
14
+ require 'fluent/test'
15
+ unless ENV.has_key?('VERBOSE')
16
+ nulllogger = Object.new
17
+ nulllogger.instance_eval {|obj|
18
+ def method_missing(method, *args)
19
+ # pass
20
+ end
21
+ }
22
+ $log = nulllogger
23
+ end
24
+
25
+ require 'fluent/plugin/out_redis_store'
26
+
27
+ class Test::Unit::TestCase
28
+ end
@@ -0,0 +1,367 @@
1
+ require 'helpers'
2
+
3
+ require 'redis'
4
+
5
+ $channel = nil
6
+ $message = nil
7
+
8
+ class Redis
9
+ def initialize(options = {})
10
+ end
11
+
12
+ def pipelined
13
+ yield self
14
+ end
15
+
16
+ def set(key, message)
17
+ $command = :set
18
+ $key = key
19
+ $message = message
20
+ end
21
+
22
+ def rpush(key, message)
23
+ $command = :rpush
24
+ $key = key
25
+ $message = message
26
+ end
27
+
28
+ def lpush(key, message)
29
+ $command = :lpush
30
+ $key = key
31
+ $message = message
32
+ end
33
+
34
+ def sadd(key, message)
35
+ $command = :sadd
36
+ $key = key
37
+ $message = message
38
+ end
39
+
40
+ def zadd(key, score, message)
41
+ $command = :zadd
42
+ $key = key
43
+ $score = score
44
+ $message = message
45
+ end
46
+
47
+ def expire(key, ttl)
48
+ $expire_key = key
49
+ $ttl = ttl
50
+ end
51
+
52
+ def publish(channel, message)
53
+ $command = :publish
54
+ $channel = channel
55
+ $message = message
56
+ end
57
+
58
+ def quit
59
+ end
60
+ end
61
+
62
+ class RedisStoreOutputTest < Test::Unit::TestCase
63
+ def setup
64
+ Fluent::Test.setup
65
+ end
66
+
67
+ def create_driver(conf)
68
+ Fluent::Test::BufferedOutputTestDriver.new(Fluent::RedisStoreOutput).configure(conf)
69
+ end
70
+
71
+ def test_configure_defaults
72
+ config = %[
73
+ key_path a
74
+ score_path b
75
+ ]
76
+ d = create_driver(config)
77
+ assert_equal "127.0.0.1", d.instance.host
78
+ assert_equal 6379, d.instance.port
79
+ assert_equal nil, d.instance.path
80
+ assert_equal nil, d.instance.password
81
+ assert_equal 0, d.instance.db
82
+ assert_equal 5.0, d.instance.timeout
83
+ assert_equal 'json', d.instance.format_type
84
+ assert_equal '', d.instance.key_prefix
85
+ assert_equal '', d.instance.key_suffix
86
+ assert_equal 'zset', d.instance.store_type
87
+ assert_equal 'a', d.instance.key_path
88
+ assert_equal nil, d.instance.key
89
+ assert_equal 'b', d.instance.score_path
90
+ assert_equal '', d.instance.value_path
91
+ assert_equal -1, d.instance.key_expire
92
+ assert_equal -1, d.instance.value_expire
93
+ assert_equal -1, d.instance.value_length
94
+ assert_equal 'asc', d.instance.order
95
+ end
96
+
97
+ def test_configure_host_port_db
98
+ config = %[
99
+ host 192.168.2.3
100
+ port 9999
101
+ password abc
102
+ db 3
103
+ timeout 7
104
+ key a
105
+ score_path b
106
+ ]
107
+ d = create_driver(config)
108
+ assert_equal "192.168.2.3", d.instance.host
109
+ assert_equal 9999, d.instance.port
110
+ assert_equal nil, d.instance.path
111
+ assert_equal 'abc', d.instance.password
112
+ assert_equal 3, d.instance.db
113
+ assert_equal 7.0, d.instance.timeout
114
+ assert_equal nil, d.instance.key_path
115
+ assert_equal 'a', d.instance.key
116
+ end
117
+
118
+ def test_configure_path
119
+ config = %[
120
+ path /tmp/foo.sock
121
+ key a
122
+ score_path b
123
+ ]
124
+ d = create_driver(config)
125
+ assert_equal "/tmp/foo.sock", d.instance.path
126
+ end
127
+
128
+ def test_configure_exception
129
+ assert_raise(Fluent::ConfigError) do
130
+ create_driver(%[])
131
+ end
132
+ end
133
+
134
+ # def test_write
135
+ # d = create_driver(CONFIG1)
136
+ #
137
+ # time = Time.parse("2011-01-02 13:14:15 UTC").to_i
138
+ # d.emit({ "foo" => "bar" }, time)
139
+ # d.run
140
+ #
141
+ # assert_equal "test", $channel
142
+ # assert_equal(%Q[{"foo":"bar","time":#{time}}], $message)
143
+ # end
144
+
145
+ def get_time
146
+ Time.parse("2011-01-02 13:14:15 UTC").to_i
147
+ end
148
+
149
+ # it should return whole message
150
+ def test_omit_value_path
151
+ config = %[
152
+ format_type plain
153
+ store_type string
154
+ key_path user
155
+ ]
156
+ d = create_driver(config)
157
+ message = {
158
+ 'user' => 'george',
159
+ 'stat' => { 'attack' => 7 }
160
+ }
161
+ $ttl = nil
162
+ d.emit(message, get_time)
163
+ d.run
164
+
165
+ assert_equal "george", $key
166
+ assert_equal message, $message
167
+ assert_equal nil, $ttl
168
+ end
169
+
170
+ def test_key_value_paths
171
+ config = %[
172
+ format_type plain
173
+ store_type string
174
+ key_path user.name
175
+ value_path stat.attack
176
+ key_expire 3
177
+ ]
178
+ d = create_driver(config)
179
+ message = {
180
+ 'user' => { 'name' => 'george' },
181
+ 'stat' => { 'attack' => 7 }
182
+ }
183
+ $ttl = nil
184
+ d.emit(message, get_time)
185
+ d.run
186
+
187
+ assert_equal "george", $key
188
+ assert_equal 7, $message
189
+ assert_equal 3, $ttl
190
+ end
191
+
192
+ def test_json
193
+ config = %[
194
+ format_type json
195
+ store_type string
196
+ key_path user
197
+ ]
198
+ d = create_driver(config)
199
+ message = {
200
+ 'user' => 'george',
201
+ 'stat' => { 'attack' => 7 }
202
+ }
203
+ $ttl = nil
204
+ d.emit(message, get_time)
205
+ d.run
206
+
207
+ assert_equal "george", $key
208
+ assert_equal message.to_json, $message
209
+ end
210
+
211
+ def test_msgpack
212
+ config = %[
213
+ format_type msgpack
214
+ store_type string
215
+ key_path user
216
+ ]
217
+ d = create_driver(config)
218
+ message = {
219
+ 'user' => 'george',
220
+ 'stat' => { 'attack' => 7 }
221
+ }
222
+ $ttl = nil
223
+ d.emit(message, get_time)
224
+ d.run
225
+
226
+ assert_equal "george", $key
227
+ assert_equal message.to_msgpack, $message
228
+ end
229
+
230
+ def test_list_asc
231
+ config = %[
232
+ format_type plain
233
+ store_type list
234
+ key_path user
235
+ ]
236
+ d = create_driver(config)
237
+ message = {
238
+ 'user' => 'george',
239
+ 'stat' => { 'attack' => 7 }
240
+ }
241
+ d.emit(message, get_time)
242
+ d.run
243
+
244
+ assert_equal :rpush, $command
245
+ assert_equal "george", $key
246
+ assert_equal message, $message
247
+ end
248
+
249
+ def test_list_desc
250
+ config = %[
251
+ format_type plain
252
+ store_type list
253
+ key_path user
254
+ order desc
255
+ ]
256
+ d = create_driver(config)
257
+ message = {
258
+ 'user' => 'george',
259
+ 'stat' => { 'attack' => 7 }
260
+ }
261
+ d.emit(message, get_time)
262
+ d.run
263
+
264
+ assert_equal :lpush, $command
265
+ assert_equal "george", $key
266
+ assert_equal message, $message
267
+ end
268
+
269
+ def test_set
270
+ config = %[
271
+ format_type plain
272
+ store_type set
273
+ key_path user
274
+ order desc
275
+ ]
276
+ d = create_driver(config)
277
+ message = {
278
+ 'user' => 'george',
279
+ 'stat' => { 'attack' => 7 }
280
+ }
281
+ d.emit(message, get_time)
282
+ d.run
283
+
284
+ assert_equal :sadd, $command
285
+ assert_equal "george", $key
286
+ assert_equal message, $message
287
+ end
288
+
289
+ def test_zset
290
+ config = %[
291
+ format_type plain
292
+ store_type zset
293
+ key_path user
294
+ score_path result
295
+ ]
296
+
297
+ d = create_driver(config)
298
+ message = {
299
+ 'user' => 'george',
300
+ 'stat' => { 'attack' => 7 },
301
+ 'result' => 81
302
+ }
303
+ d.emit(message, get_time)
304
+ d.run
305
+
306
+ assert_equal :zadd, $command
307
+ assert_equal "george", $key
308
+ assert_equal 81, $score
309
+ assert_equal message, $message
310
+ end
311
+
312
+ def test_zset_with_no_score_path
313
+ config = %[
314
+ format_type plain
315
+ store_type zset
316
+ key_path user
317
+ ]
318
+
319
+ d = create_driver(config)
320
+ message = {
321
+ 'user' => 'george',
322
+ 'stat' => { 'attack' => 7 },
323
+ 'result' => 81
324
+ }
325
+ d.emit(message, get_time)
326
+ d.run
327
+
328
+ assert_equal :zadd, $command
329
+ assert_equal "george", $key
330
+ assert_equal get_time, $score
331
+ assert_equal message, $message
332
+ end
333
+
334
+ def test_publish
335
+ config = %[
336
+ format_type plain
337
+ store_type publish
338
+ key_path user
339
+ ]
340
+
341
+ d = create_driver(config)
342
+ message = {
343
+ 'user' => 'george'
344
+ }
345
+ d.emit(message, get_time)
346
+ d.run
347
+
348
+ assert_equal :publish, $command
349
+ assert_equal "george", $channel
350
+ assert_equal message, $message
351
+ end
352
+
353
+ def test_empty_key
354
+ config = %[
355
+ format_type plain
356
+ store_type string
357
+ key_path none
358
+ ]
359
+
360
+ d = create_driver(config)
361
+ message = { 'user' => 'george' }
362
+ d.emit(message, get_time)
363
+ assert_raise(Fluent::ConfigError) do
364
+ d.run
365
+ end
366
+ end
367
+ end
metadata ADDED
@@ -0,0 +1,109 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: fluent-plugin-redis-store-seldon
3
+ version: !ruby/object:Gem::Version
4
+ version: '0.1'
5
+ platform: ruby
6
+ authors:
7
+ - Clive Cox
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-12-26 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: rake
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: test-unit
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: fluentd
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: redis
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ description: Redis(zset/set/list/string/publish) output plugin for Fluentd...
70
+ email: cc@seldon.io
71
+ executables: []
72
+ extensions: []
73
+ extra_rdoc_files: []
74
+ files:
75
+ - AUTHORS
76
+ - Gemfile
77
+ - README.md
78
+ - Rakefile
79
+ - fluent-plugin-redis-store-seldon.gemspec
80
+ - lib/fluent/plugin/out_redis_store_sedon.rb
81
+ - test/helpers.rb
82
+ - test/plugin/test_out_redis_publish.rb
83
+ homepage: https://github.com/SeldonIO/fluent-plugin-redis-store
84
+ licenses:
85
+ - Apache License Version 2.0
86
+ metadata: {}
87
+ post_install_message:
88
+ rdoc_options: []
89
+ require_paths:
90
+ - lib
91
+ required_ruby_version: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - ">="
94
+ - !ruby/object:Gem::Version
95
+ version: '0'
96
+ required_rubygems_version: !ruby/object:Gem::Requirement
97
+ requirements:
98
+ - - ">="
99
+ - !ruby/object:Gem::Version
100
+ version: '0'
101
+ requirements: []
102
+ rubyforge_project:
103
+ rubygems_version: 2.4.5
104
+ signing_key:
105
+ specification_version: 4
106
+ summary: Redis(zset/set/list/string/publish) output plugin for Fluentd
107
+ test_files:
108
+ - test/helpers.rb
109
+ - test/plugin/test_out_redis_publish.rb