fluent-plugin-redis-store-seldon 0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: b56cd9a0f6e04be5cc93536aa4329bf34843e1bc
4
+ data.tar.gz: f7e20910059d09cfb33551c2a07de1fcf4c65500
5
+ SHA512:
6
+ metadata.gz: 7409a2407f6f350b518e3efa82be8fd1b04d501c9e13a6dbb79a43bc9fc725711cb1975b3ca7fd5246ac3ed933eefe701485bbafee1fd9708df5498ee11a6b3c
7
+ data.tar.gz: 259872bd9c4aea59ff6a75e7efbe1956be8a9ca043bb427861b5929bebdd6ffed8a6be7f468a639b5f2305253cae6bb1441d61647efdc9d2c6b0c66bb23ab836
data/AUTHORS ADDED
@@ -0,0 +1,2 @@
1
+ moaikids
2
+ HANAI Tohru aka pokehanai
data/Gemfile ADDED
@@ -0,0 +1,3 @@
1
+ source "http://rubygems.org"
2
+
3
+ gemspec
data/README.md ADDED
@@ -0,0 +1,161 @@
1
+ Fork of https://github.com/pokehanai/fluent-plugin-redis-store
2
+
3
+ Redis Output Plugin For fluentd
4
+ ===============================
5
+ [Fluentd][] output plugin to upload/publish event data to [Redis][] storage.
6
+
7
+ [Fluentd]: http://fluentd.org/
8
+ [Redis]: http://redis.io/
9
+
10
+ Background
11
+ ----------
12
+
13
+ This is folked project from [fluent-plugin-redisstore][].
14
+
15
+ [fluent-plugin-redisstore]: https://github.com/moaikids/fluent-plugin-redisstore
16
+
17
+ Features
18
+ --------
19
+
20
+ #### Supported Redis commands
21
+
22
+ Currently the plugin supports following Redis commands:
23
+
24
+ - **set** by `string` type (of the plugin)
25
+ - **lpush**/**rpush** by `list` type
26
+ - **sadd** by `set` type
27
+ - **zadd** by `zset` type
28
+ - **publish** by `publish` type
29
+
30
+ #### Supported _value_ format
31
+
32
+ - plain(as is)
33
+ - JSON
34
+ - [MessagePack](http://msgpack.org/)
35
+
36
+ #### _key_ string for Redis storage
37
+
38
+ Redis commands require _key_ and _value_.
39
+ For _key_, the plugin supports either way;
40
+
41
+ 1. Specify a fixed key.
42
+ You can do this simply using `key` option in td-agent configuration file.
43
+
44
+ ```apache
45
+ type redis_store
46
+ key userdata
47
+ ```
48
+
49
+ 2. Lookup a key string in every event data by a lookup path.
50
+ If event data have structured data like
51
+
52
+ ```javascript
53
+ { "user": { "name": "Kei" } }
54
+ ```
55
+
56
+ and you want to use each name of user, you can use `key_path` option.
57
+
58
+ ```apache
59
+ type redis_store
60
+ key_path user.name
61
+ ```
62
+
63
+ With the above data, `Kei` will be a _key_.
64
+
65
+ In addition, `key_prefix` and `key_suffix` are useful in some cases. Both are available either `key` and `key_path`
66
+
67
+ ```apache
68
+ type redis_store
69
+ key_path user.name
70
+ key_prefix ouruser.
71
+ key_suffix .accesslog
72
+ ```
73
+
74
+ With the previous data, _key_ will be `outuser.Kei.accesslog`.
75
+
76
+ #### _value_ data for Redis storage
77
+
78
+ To determine what _value_ in every event data to be srtored, you have two options;
79
+
80
+ 1. Store extracted data in event data, by a lookup path with `value_path` option.
81
+ It works like `key_path`.
82
+ 2. Store whole data.
83
+ This is default behavior. To do it, simply omit `value_path` option.
84
+
85
+ Installation
86
+ ------------
87
+
88
+ /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-redis-store
89
+
90
+ Configuration
91
+ -------------
92
+
93
+ ### Redis connection
94
+
95
+ | Key | Type | Required? | Default | Description |
96
+ | :---- | :----- | :---------- | :----------------------- | :------------ |
97
+ | `host` | string | Optional | 127.0.0.1 | host name of Redis server |
98
+ | `port` | int | Optional | 6379 | port number of Redis server |
99
+ | `password` | string | Optional | | password for Redis connection |
100
+ | `path` | string | Optional | | To connect via Unix socket, try '/tmp/redis.sock' |
101
+ | `db` | int | Optional | 0 | DB number of Redis |
102
+ | `timeout` | float | Optional | 5.0 | connection timeout in seconds |
103
+
104
+ ### common options for storages
105
+
106
+ | Key | Type | Default | Description |
107
+ | :---- | :----- | :----------------------- | :------------ |
108
+ | `key` | string | | Fixed _key_ used to store(publish) in Redis |
109
+ | `key_path` | string | | path to lookup for _key_ in the event data |
110
+ | `key_prefix` | string | | prefix of _key_ |
111
+ | `key_prefix_sep` | string | | separator to put after prefix _key_ |
112
+ | `key_prefix_path` | string | | path to lookup prefix of _key_ |
113
+ | `key_suffix` | string | | suffix of _key_ |
114
+ | `value_path` | string | (whole event data) | path to lookup for _value_ in the event data |
115
+ | `store_type` | string | zset | `string`/`list`/`set`/`zset`/`publish` |
116
+ | `format_type` | string | plain | format type for _value_ (`plain`/`json`/`msgpack`) |
117
+ | `key_expire` | int | -1 | If set, the key will be expired in specified seconds |
118
+
119
+ Note: either `key` or `key_path` is required.
120
+
121
+ ### `string` storage specific options
122
+
123
+ | Key | Type | Default | Description |
124
+ | :---- | :----- | :----------------------- | :------------ |
125
+ | `type` | string | | Fixed _key_ used to store(publish) in Redis |
126
+ No more options than common options.
127
+
128
+ ### `list` storage specific options
129
+
130
+ | Key | Type | Default | Description |
131
+ | :---- | :----- | :----------------------- | :------------ |
132
+ | `order` | string | asc | `asc`: **rpush**, `desc`: **lpush** |
133
+
134
+ ### `set` storage specific options
135
+
136
+ No more options than common options.
137
+
138
+ ### `zset` storage specific options
139
+
140
+ | Key | Type | Default | Description |
141
+ | :---- | :----- | :----------------------- | :------------ |
142
+ | `score_path` | string | (_time_ of log event) | path to lookup for _score_ in the event data |
143
+ | `value_expire` | int | | value expiration in seconds |
144
+
145
+ If `value_expire` is set, the plugin assumes that the _score_ in the **SortedSet** is
146
+ based on *timestamp* and it deletes expired _members_ every after new event data arrives.
147
+
148
+ ### `publish` storage specific options
149
+
150
+ No more options than common options.
151
+
152
+
153
+ Copyright
154
+ ---------
155
+
156
+ Copyright (c) 2013 moaikids
157
+ Copyright (c) 2014 HANAI Tohru
158
+
159
+ License
160
+ -------
161
+ Apache License, Version 2.0
data/Rakefile ADDED
@@ -0,0 +1,11 @@
1
+ #!/usr/bin/env rake
2
+ require "bundler/gem_tasks"
3
+
4
+ require 'rake/testtask'
5
+ Rake::TestTask.new(:test) do |test|
6
+ test.libs << 'lib' << 'test'
7
+ test.pattern = 'test/**/test_*.rb'
8
+ test.verbose = true
9
+ end
10
+
11
+ task :default => :test
@@ -0,0 +1,21 @@
1
+ # -*- encoding: utf-8 -*-
2
+ Gem::Specification.new do |gem|
3
+ gem.name = "fluent-plugin-redis-store-seldon"
4
+ gem.email = "cc@seldon.io"
5
+ gem.version = "0.1"
6
+ gem.authors = ["Clive Cox"]
7
+ gem.licenses = ["Apache License Version 2.0"]
8
+ gem.summary = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd}
9
+ gem.description = %q{Redis(zset/set/list/string/publish) output plugin for Fluentd...}
10
+ gem.homepage = "https://github.com/SeldonIO/fluent-plugin-redis-store"
11
+
12
+ gem.files = `git ls-files`.split($\)
13
+ gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
14
+ gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
15
+ gem.require_paths = ["lib"]
16
+
17
+ gem.add_development_dependency "rake"
18
+ gem.add_development_dependency "test-unit"
19
+ gem.add_runtime_dependency "fluentd"
20
+ gem.add_runtime_dependency "redis"
21
+ end
@@ -0,0 +1,238 @@
1
+ module Fluent
2
+ class RedisStoreOutput < BufferedOutput
3
+ Fluent::Plugin.register_output('redis_store_seldon', self)
4
+
5
+ # redis connection
6
+ config_param :host, :string, :default => '127.0.0.1'
7
+ config_param :port, :integer, :default => 6379
8
+ config_param :path, :string, :default => nil
9
+ config_param :password, :string, :default => nil
10
+ config_param :db, :integer, :default => 0
11
+ config_param :timeout, :float, :default => 5.0
12
+
13
+ # redis command and parameters
14
+ config_param :format_type, :string, :default => 'json'
15
+ config_param :store_type, :string, :default => 'zset'
16
+ config_param :key_prefix, :string, :default => ''
17
+ config_param :key_prefix_path, :string, :default => nil
18
+ config_param :key_prefix_sep, :string, :default => ''
19
+ config_param :key_suffix, :string, :default => ''
20
+ config_param :key, :string, :default => nil
21
+ config_param :key_path, :string, :default => nil
22
+ config_param :score_path, :string, :default => nil
23
+ config_param :value_path, :string, :default => ''
24
+ config_param :key_expire, :integer, :default => -1
25
+ config_param :value_expire, :integer, :default => -1
26
+ config_param :value_length, :integer, :default => -1
27
+ config_param :order, :string, :default => 'asc'
28
+
29
+ def initialize
30
+ super
31
+ require 'redis'
32
+ require 'msgpack'
33
+ end
34
+
35
+ def configure(conf)
36
+ super
37
+
38
+ if @key_path == nil and @key == nil
39
+ raise Fluent::ConfigError, "either key_path or key is required"
40
+ end
41
+ end
42
+
43
+ def start
44
+ super
45
+ if @path
46
+ @redis = Redis.new(:path => @path, :password => @password,
47
+ :timeout => @timeout, :thread_safe => true, :db => @db)
48
+ else
49
+ @redis = Redis.new(:host => @host, :port => @port, :password => @password,
50
+ :timeout => @timeout, :thread_safe => true, :db => @db)
51
+ end
52
+ end
53
+
54
+ def shutdown
55
+ @redis.quit
56
+ end
57
+
58
+ def format(tag, time, record)
59
+ [tag, time, record].to_msgpack
60
+ end
61
+
62
+ def write(chunk)
63
+ @redis.pipelined {
64
+ chunk.open { |io|
65
+ begin
66
+ MessagePack::Unpacker.new(io).each { |message|
67
+ begin
68
+ (tag, time, record) = message
69
+ case @store_type
70
+ when 'zset'
71
+ operation_for_zset(record, time)
72
+ when 'set'
73
+ operation_for_set(record)
74
+ when 'list'
75
+ operation_for_list(record)
76
+ when 'string'
77
+ operation_for_string(record)
78
+ when 'publish'
79
+ operation_for_publish(record)
80
+ end
81
+ rescue NoMethodError => e
82
+ puts e
83
+ end
84
+ }
85
+ rescue EOFError
86
+ # EOFError always occured when reached end of chunk.
87
+ end
88
+ }
89
+ }
90
+ end
91
+
92
+ def operation_for_zset(record, time)
93
+ key = get_key_from(record)
94
+ value = get_value_from(record)
95
+ score = get_score_from(record, time)
96
+ @redis.zadd key, score, value
97
+
98
+ set_key_expire key
99
+ if 0 < @value_expire
100
+ now = Time.now.to_i
101
+ @redis.zremrangebyscore key , '-inf' , (now - @value_expire)
102
+ end
103
+ if 0 < @value_length
104
+ script = generate_zremrangebyrank_script(key, @value_length, @order)
105
+ @redis.eval script
106
+ end
107
+ end
108
+
109
+ def operation_for_set(record)
110
+ key = get_key_from(record)
111
+ value = get_value_from(record)
112
+ @redis.sadd key, value
113
+ set_key_expire key
114
+ end
115
+
116
+ def operation_for_list(record)
117
+ key = get_key_from(record)
118
+ value = get_value_from(record)
119
+
120
+ if @order == 'asc'
121
+ @redis.rpush key, value
122
+ else
123
+ @redis.lpush key, value
124
+ end
125
+ set_key_expire key
126
+ if 0 < @value_length
127
+ script = generate_ltrim_script(key, @value_length, @order)
128
+ @redis.eval script
129
+ end
130
+ end
131
+
132
+ def operation_for_string(record)
133
+ key = get_key_from(record)
134
+ value = get_value_from(record)
135
+ @redis.set key, value
136
+
137
+ set_key_expire key
138
+ end
139
+
140
+ def operation_for_publish(record)
141
+ key = get_key_from(record)
142
+ value = get_value_from(record)
143
+ @redis.publish key, value
144
+ end
145
+
146
+ def generate_zremrangebyrank_script(key, maxlen, order)
147
+ script = "local key = '" + key.to_s + "'\n"
148
+ script += "local maxlen = " + maxlen.to_s + "\n"
149
+ script += "local order ='" + order.to_s + "'\n"
150
+ script += "local len = tonumber(redis.call('ZCOUNT', key, '-inf', '+inf'))\n"
151
+ script += "if len > maxlen then\n"
152
+ script += " if order == 'asc' then\n"
153
+ script += " local l = len - maxlen\n"
154
+ script += " if l >= 0 then\n"
155
+ script += " return redis.call('ZREMRANGEBYRANK', key, 0, l)\n"
156
+ script += " end\n"
157
+ script += " else\n"
158
+ script += " return redis.call('ZREMRANGEBYRANK', key, maxlen, -1)\n"
159
+ script += " end\n"
160
+ script += "end\n"
161
+ return script
162
+ end
163
+
164
+ def generate_ltrim_script(key, maxlen, order)
165
+ script = "local key = '" + key.to_s + "'\n"
166
+ script += "local maxlen = " + maxlen.to_s + "\n"
167
+ script += "local order ='" + order.to_s + "'\n"
168
+ script += "local len = tonumber(redis.call('LLEN', key))\n"
169
+ script += "if len > maxlen then\n"
170
+ script += " if order == 'asc' then\n"
171
+ script += " local l = len - maxlen\n"
172
+ script += " return redis.call('LTRIM', key, l, -1)\n"
173
+ script += " else\n"
174
+ script += " return redis.call('LTRIM', key, 0, maxlen - 1)\n"
175
+ script += " end\n"
176
+ script += "end\n"
177
+ return script
178
+ end
179
+
180
+ def traverse(data, key)
181
+ val = data
182
+ key.split('.').each{ |k|
183
+ if val.has_key?(k)
184
+ val = val[k]
185
+ else
186
+ return nil
187
+ end
188
+ }
189
+ return val
190
+ end
191
+
192
+ def get_key_from(record)
193
+ if @key
194
+ k = @key
195
+ else
196
+ k = traverse(record, @key_path).to_s
197
+ end
198
+
199
+ if @key_prefix_path
200
+ prefix = traverse(record, @key_prefix_path).to_s
201
+ else
202
+ prefix = ''
203
+ end
204
+
205
+ key = @key_prefix + prefix + @key_prefix_sep + k + @key_suffix
206
+
207
+ raise Fluent::ConfigError, "key is empty" if key == ''
208
+ key
209
+ end
210
+
211
+ def get_value_from(record)
212
+ value = traverse(record, @value_path)
213
+ case @format_type
214
+ when 'json'
215
+ value.to_json
216
+ when 'msgpack'
217
+ value.to_msgpack
218
+ else
219
+ value
220
+ end
221
+ end
222
+
223
+ def get_score_from(record, time)
224
+ if @score_path
225
+ traverse(record, @score_path)
226
+ else
227
+ time
228
+ end
229
+ end
230
+
231
+ def set_key_expire(key)
232
+ if 0 < @key_expire
233
+ @redis.expire key, @key_expire
234
+ end
235
+ end
236
+
237
+ end
238
+ end
data/test/helpers.rb ADDED
@@ -0,0 +1,28 @@
1
+ require 'rubygems'
2
+ require 'bundler'
3
+ begin
4
+ Bundler.setup(:default, :development)
5
+ rescue Bundler::BundlerError => e
6
+ $stderr.puts e.message
7
+ $stderr.puts "Run `bundle install` to install missing gems"
8
+ exit e.status_code
9
+ end
10
+ require 'test/unit'
11
+
12
+ $LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
13
+ $LOAD_PATH.unshift(File.dirname(__FILE__))
14
+ require 'fluent/test'
15
+ unless ENV.has_key?('VERBOSE')
16
+ nulllogger = Object.new
17
+ nulllogger.instance_eval {|obj|
18
+ def method_missing(method, *args)
19
+ # pass
20
+ end
21
+ }
22
+ $log = nulllogger
23
+ end
24
+
25
+ require 'fluent/plugin/out_redis_store'
26
+
27
+ class Test::Unit::TestCase
28
+ end
@@ -0,0 +1,367 @@
1
+ require 'helpers'
2
+
3
+ require 'redis'
4
+
5
+ $channel = nil
6
+ $message = nil
7
+
8
+ class Redis
9
+ def initialize(options = {})
10
+ end
11
+
12
+ def pipelined
13
+ yield self
14
+ end
15
+
16
+ def set(key, message)
17
+ $command = :set
18
+ $key = key
19
+ $message = message
20
+ end
21
+
22
+ def rpush(key, message)
23
+ $command = :rpush
24
+ $key = key
25
+ $message = message
26
+ end
27
+
28
+ def lpush(key, message)
29
+ $command = :lpush
30
+ $key = key
31
+ $message = message
32
+ end
33
+
34
+ def sadd(key, message)
35
+ $command = :sadd
36
+ $key = key
37
+ $message = message
38
+ end
39
+
40
+ def zadd(key, score, message)
41
+ $command = :zadd
42
+ $key = key
43
+ $score = score
44
+ $message = message
45
+ end
46
+
47
+ def expire(key, ttl)
48
+ $expire_key = key
49
+ $ttl = ttl
50
+ end
51
+
52
+ def publish(channel, message)
53
+ $command = :publish
54
+ $channel = channel
55
+ $message = message
56
+ end
57
+
58
+ def quit
59
+ end
60
+ end
61
+
62
+ class RedisStoreOutputTest < Test::Unit::TestCase
63
+ def setup
64
+ Fluent::Test.setup
65
+ end
66
+
67
+ def create_driver(conf)
68
+ Fluent::Test::BufferedOutputTestDriver.new(Fluent::RedisStoreOutput).configure(conf)
69
+ end
70
+
71
+ def test_configure_defaults
72
+ config = %[
73
+ key_path a
74
+ score_path b
75
+ ]
76
+ d = create_driver(config)
77
+ assert_equal "127.0.0.1", d.instance.host
78
+ assert_equal 6379, d.instance.port
79
+ assert_equal nil, d.instance.path
80
+ assert_equal nil, d.instance.password
81
+ assert_equal 0, d.instance.db
82
+ assert_equal 5.0, d.instance.timeout
83
+ assert_equal 'json', d.instance.format_type
84
+ assert_equal '', d.instance.key_prefix
85
+ assert_equal '', d.instance.key_suffix
86
+ assert_equal 'zset', d.instance.store_type
87
+ assert_equal 'a', d.instance.key_path
88
+ assert_equal nil, d.instance.key
89
+ assert_equal 'b', d.instance.score_path
90
+ assert_equal '', d.instance.value_path
91
+ assert_equal -1, d.instance.key_expire
92
+ assert_equal -1, d.instance.value_expire
93
+ assert_equal -1, d.instance.value_length
94
+ assert_equal 'asc', d.instance.order
95
+ end
96
+
97
+ def test_configure_host_port_db
98
+ config = %[
99
+ host 192.168.2.3
100
+ port 9999
101
+ password abc
102
+ db 3
103
+ timeout 7
104
+ key a
105
+ score_path b
106
+ ]
107
+ d = create_driver(config)
108
+ assert_equal "192.168.2.3", d.instance.host
109
+ assert_equal 9999, d.instance.port
110
+ assert_equal nil, d.instance.path
111
+ assert_equal 'abc', d.instance.password
112
+ assert_equal 3, d.instance.db
113
+ assert_equal 7.0, d.instance.timeout
114
+ assert_equal nil, d.instance.key_path
115
+ assert_equal 'a', d.instance.key
116
+ end
117
+
118
+ def test_configure_path
119
+ config = %[
120
+ path /tmp/foo.sock
121
+ key a
122
+ score_path b
123
+ ]
124
+ d = create_driver(config)
125
+ assert_equal "/tmp/foo.sock", d.instance.path
126
+ end
127
+
128
+ def test_configure_exception
129
+ assert_raise(Fluent::ConfigError) do
130
+ create_driver(%[])
131
+ end
132
+ end
133
+
134
+ # def test_write
135
+ # d = create_driver(CONFIG1)
136
+ #
137
+ # time = Time.parse("2011-01-02 13:14:15 UTC").to_i
138
+ # d.emit({ "foo" => "bar" }, time)
139
+ # d.run
140
+ #
141
+ # assert_equal "test", $channel
142
+ # assert_equal(%Q[{"foo":"bar","time":#{time}}], $message)
143
+ # end
144
+
145
+ def get_time
146
+ Time.parse("2011-01-02 13:14:15 UTC").to_i
147
+ end
148
+
149
+ # it should return whole message
150
+ def test_omit_value_path
151
+ config = %[
152
+ format_type plain
153
+ store_type string
154
+ key_path user
155
+ ]
156
+ d = create_driver(config)
157
+ message = {
158
+ 'user' => 'george',
159
+ 'stat' => { 'attack' => 7 }
160
+ }
161
+ $ttl = nil
162
+ d.emit(message, get_time)
163
+ d.run
164
+
165
+ assert_equal "george", $key
166
+ assert_equal message, $message
167
+ assert_equal nil, $ttl
168
+ end
169
+
170
+ def test_key_value_paths
171
+ config = %[
172
+ format_type plain
173
+ store_type string
174
+ key_path user.name
175
+ value_path stat.attack
176
+ key_expire 3
177
+ ]
178
+ d = create_driver(config)
179
+ message = {
180
+ 'user' => { 'name' => 'george' },
181
+ 'stat' => { 'attack' => 7 }
182
+ }
183
+ $ttl = nil
184
+ d.emit(message, get_time)
185
+ d.run
186
+
187
+ assert_equal "george", $key
188
+ assert_equal 7, $message
189
+ assert_equal 3, $ttl
190
+ end
191
+
192
+ def test_json
193
+ config = %[
194
+ format_type json
195
+ store_type string
196
+ key_path user
197
+ ]
198
+ d = create_driver(config)
199
+ message = {
200
+ 'user' => 'george',
201
+ 'stat' => { 'attack' => 7 }
202
+ }
203
+ $ttl = nil
204
+ d.emit(message, get_time)
205
+ d.run
206
+
207
+ assert_equal "george", $key
208
+ assert_equal message.to_json, $message
209
+ end
210
+
211
+ def test_msgpack
212
+ config = %[
213
+ format_type msgpack
214
+ store_type string
215
+ key_path user
216
+ ]
217
+ d = create_driver(config)
218
+ message = {
219
+ 'user' => 'george',
220
+ 'stat' => { 'attack' => 7 }
221
+ }
222
+ $ttl = nil
223
+ d.emit(message, get_time)
224
+ d.run
225
+
226
+ assert_equal "george", $key
227
+ assert_equal message.to_msgpack, $message
228
+ end
229
+
230
+ def test_list_asc
231
+ config = %[
232
+ format_type plain
233
+ store_type list
234
+ key_path user
235
+ ]
236
+ d = create_driver(config)
237
+ message = {
238
+ 'user' => 'george',
239
+ 'stat' => { 'attack' => 7 }
240
+ }
241
+ d.emit(message, get_time)
242
+ d.run
243
+
244
+ assert_equal :rpush, $command
245
+ assert_equal "george", $key
246
+ assert_equal message, $message
247
+ end
248
+
249
+ def test_list_desc
250
+ config = %[
251
+ format_type plain
252
+ store_type list
253
+ key_path user
254
+ order desc
255
+ ]
256
+ d = create_driver(config)
257
+ message = {
258
+ 'user' => 'george',
259
+ 'stat' => { 'attack' => 7 }
260
+ }
261
+ d.emit(message, get_time)
262
+ d.run
263
+
264
+ assert_equal :lpush, $command
265
+ assert_equal "george", $key
266
+ assert_equal message, $message
267
+ end
268
+
269
+ def test_set
270
+ config = %[
271
+ format_type plain
272
+ store_type set
273
+ key_path user
274
+ order desc
275
+ ]
276
+ d = create_driver(config)
277
+ message = {
278
+ 'user' => 'george',
279
+ 'stat' => { 'attack' => 7 }
280
+ }
281
+ d.emit(message, get_time)
282
+ d.run
283
+
284
+ assert_equal :sadd, $command
285
+ assert_equal "george", $key
286
+ assert_equal message, $message
287
+ end
288
+
289
+ def test_zset
290
+ config = %[
291
+ format_type plain
292
+ store_type zset
293
+ key_path user
294
+ score_path result
295
+ ]
296
+
297
+ d = create_driver(config)
298
+ message = {
299
+ 'user' => 'george',
300
+ 'stat' => { 'attack' => 7 },
301
+ 'result' => 81
302
+ }
303
+ d.emit(message, get_time)
304
+ d.run
305
+
306
+ assert_equal :zadd, $command
307
+ assert_equal "george", $key
308
+ assert_equal 81, $score
309
+ assert_equal message, $message
310
+ end
311
+
312
+ def test_zset_with_no_score_path
313
+ config = %[
314
+ format_type plain
315
+ store_type zset
316
+ key_path user
317
+ ]
318
+
319
+ d = create_driver(config)
320
+ message = {
321
+ 'user' => 'george',
322
+ 'stat' => { 'attack' => 7 },
323
+ 'result' => 81
324
+ }
325
+ d.emit(message, get_time)
326
+ d.run
327
+
328
+ assert_equal :zadd, $command
329
+ assert_equal "george", $key
330
+ assert_equal get_time, $score
331
+ assert_equal message, $message
332
+ end
333
+
334
+ def test_publish
335
+ config = %[
336
+ format_type plain
337
+ store_type publish
338
+ key_path user
339
+ ]
340
+
341
+ d = create_driver(config)
342
+ message = {
343
+ 'user' => 'george'
344
+ }
345
+ d.emit(message, get_time)
346
+ d.run
347
+
348
+ assert_equal :publish, $command
349
+ assert_equal "george", $channel
350
+ assert_equal message, $message
351
+ end
352
+
353
+ def test_empty_key
354
+ config = %[
355
+ format_type plain
356
+ store_type string
357
+ key_path none
358
+ ]
359
+
360
+ d = create_driver(config)
361
+ message = { 'user' => 'george' }
362
+ d.emit(message, get_time)
363
+ assert_raise(Fluent::ConfigError) do
364
+ d.run
365
+ end
366
+ end
367
+ end
metadata ADDED
@@ -0,0 +1,109 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: fluent-plugin-redis-store-seldon
3
+ version: !ruby/object:Gem::Version
4
+ version: '0.1'
5
+ platform: ruby
6
+ authors:
7
+ - Clive Cox
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-12-26 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: rake
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '0'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - ">="
25
+ - !ruby/object:Gem::Version
26
+ version: '0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: test-unit
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: '0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: '0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: fluentd
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - ">="
46
+ - !ruby/object:Gem::Version
47
+ version: '0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - ">="
53
+ - !ruby/object:Gem::Version
54
+ version: '0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: redis
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - ">="
60
+ - !ruby/object:Gem::Version
61
+ version: '0'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: '0'
69
+ description: Redis(zset/set/list/string/publish) output plugin for Fluentd...
70
+ email: cc@seldon.io
71
+ executables: []
72
+ extensions: []
73
+ extra_rdoc_files: []
74
+ files:
75
+ - AUTHORS
76
+ - Gemfile
77
+ - README.md
78
+ - Rakefile
79
+ - fluent-plugin-redis-store-seldon.gemspec
80
+ - lib/fluent/plugin/out_redis_store_sedon.rb
81
+ - test/helpers.rb
82
+ - test/plugin/test_out_redis_publish.rb
83
+ homepage: https://github.com/SeldonIO/fluent-plugin-redis-store
84
+ licenses:
85
+ - Apache License Version 2.0
86
+ metadata: {}
87
+ post_install_message:
88
+ rdoc_options: []
89
+ require_paths:
90
+ - lib
91
+ required_ruby_version: !ruby/object:Gem::Requirement
92
+ requirements:
93
+ - - ">="
94
+ - !ruby/object:Gem::Version
95
+ version: '0'
96
+ required_rubygems_version: !ruby/object:Gem::Requirement
97
+ requirements:
98
+ - - ">="
99
+ - !ruby/object:Gem::Version
100
+ version: '0'
101
+ requirements: []
102
+ rubyforge_project:
103
+ rubygems_version: 2.4.5
104
+ signing_key:
105
+ specification_version: 4
106
+ summary: Redis(zset/set/list/string/publish) output plugin for Fluentd
107
+ test_files:
108
+ - test/helpers.rb
109
+ - test/plugin/test_out_redis_publish.rb