fluent-plugin-querycombiner 0.0.1 → 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 65fedb30907434824303a1a024616f1bc3431d3e
4
- data.tar.gz: 12a5292be841fc2c060892f0fc588a0172ce01df
3
+ metadata.gz: 1124703306d418f033d3ff5127b918532e8b3930
4
+ data.tar.gz: df04cbd16f7465ec26fe3acadffa687163da88b2
5
5
  SHA512:
6
- metadata.gz: fee92e2a0ffbe5c606663d55c7de4773ebd5d44b976f3523cd467dde981347a87ed752f803c0e4206070ca025a6db478703a7e0b4662771289467ed831132c57
7
- data.tar.gz: 546fa4062b3ab8e6b2820c29062f3ad615b3ab43e1acc5209570e614c68ae4479241a73fe3b090064e1b6b424acd5734d17b52d3b72caf200193468bc315980f
6
+ metadata.gz: 82d7aef42eacf95e1f3282205e872523d9f00f742c606820e51358d3710966a76e5a640223a707facea22d18d24bb7e384c7e4c2b9dc9b701d2931b412ab9af2
7
+ data.tar.gz: 26a39c63fb7b00e9e143485ed820739417b27fa99b7f424e6f40f95251d710622228cf2c0afe68c58dd465c1affa3115b4d0835c27484ebe0746484738b7a4dc
data/README.md CHANGED
@@ -78,7 +78,7 @@ Combined results will be:
78
78
 
79
79
  ### Replace some field names
80
80
 
81
- If messages has the same fields, these are overwritten in the combination process. You can use `replace` sentence in `<catch>` and `<dump>` blocks to avoid overwrite such fields.
81
+ If messages has the same fields, these are overwritten in the combination process. You can use `replace` sentence in `<catch>` and `<dump>` blocks to avoid overwriting such fields.
82
82
 
83
83
  For example, you have some event messages like:
84
84
 
@@ -113,7 +113,6 @@ You can keep `time` fields which defined both `event-start` and `event-finish` b
113
113
  <catch>
114
114
  condition status == 'event-start'
115
115
  replace time => time_start
116
-
117
116
  </catch>
118
117
 
119
118
  <dump>
@@ -135,6 +134,15 @@ Combined results will be:
135
134
  }
136
135
  ```
137
136
 
137
+ You can also replace multiple fields joined by comma(`,`):
138
+
139
+ ```
140
+ <catch>
141
+ condition status == 'event-start'
142
+ replace time => time_start, condition => condition_start
143
+ </catch>
144
+ ```
145
+
138
146
  ### \<release\> block
139
147
 
140
148
  In previous examples, messages with `"status": "event-start"` will be watched by plugin immediately.
@@ -166,7 +174,7 @@ You cannot use `replace` sentence in the `<release>` block.
166
174
 
167
175
  ### \<prolong\> block
168
176
 
169
- Suppose your `query_ttl` is **600** (10 minutes) and almost events are finished within **10 minutes**. But occasionally very-long events occur which finish about **1 hours**. These very-long events send `status: 'event-continue'` messages every 5 minutes for keep-alive.
177
+ Suppose your `query_ttl` is **600** (10 minutes) and almost events are finished within **10 minutes**. But occasionally very-long events occur which finish about **1 hour**. These very-long events send `status: 'event-continue'` messages every 5 minutes for keep-alive.
170
178
 
171
179
  In this case you can use `<prolong>` block to reset expired time.
172
180
 
@@ -180,6 +188,34 @@ You cannot use `replace` sentence in the `<prolong>` block.
180
188
 
181
189
  Also you cannot combine messages which defined `<prolong>` blocks.
182
190
 
191
+ ### Record time of the event
192
+
193
+ If you combine events, time of the events will be lost except defined in `<dump>` block.
194
+
195
+ If you want record time of the event, you can define `time` sentence in `<catch>` and `<dump>` blocks.
196
+
197
+ For example, if you configure your fluentd configuration like below,
198
+
199
+ ```
200
+ <catch>
201
+ condition status == 'event-start'
202
+ replace time => time_start, condition => condition_start
203
+ time time-catch
204
+ </catch>
205
+ ```
206
+
207
+ you can record time in `time-catch` field in the result.
208
+
209
+ ```
210
+ {
211
+ "event_id": "01234567",
212
+ "status": "event-finish",
213
+ "time-catch": 1414715801.112015,
214
+ }
215
+ ```
216
+
217
+ You can set time formats by `time_format` configuration.
218
+
183
219
 
184
220
  ## Configuration
185
221
 
@@ -193,7 +229,7 @@ The basic information for connecting to Redis. By default it's **redis://127.0.0
193
229
  How many times should the plugin retry when performing a redis operation before raising a error.
194
230
  By default it's 3.
195
231
 
196
- ### querl_ttl
232
+ ### query_ttl
197
233
  The inactive expire time in seconds. By default it's **1800** (30 minutes).
198
234
 
199
235
  ### buffer_size
@@ -202,7 +238,6 @@ The max queries to store in redis. By default it's **1000**.
202
238
  ### remove_interval
203
239
  The interval time to delete expired or overflowed queries which configured by `query_ttl` and `buffer_size`. By default it's 10 [sec].
204
240
 
205
-
206
241
  ### redis_key_prefix
207
242
 
208
243
  The key prefix for data stored in Redis. By default it's `query_combiner:`.
@@ -212,11 +247,19 @@ The key prefix for data stored in Redis. By default it's `query_combiner:`.
212
247
  Indicates how to extract the query identity from event record.
213
248
  It can be set as a single field name or multiple field names join by comma (`,`).
214
249
 
250
+ ### time_format
251
+
252
+ The time format for recording time of the events. Default is `$time` which holds event time. You can also use [Ruby's Time module](http://www.ruby-doc.org/core/Time.html).
253
+ If you want write ISO8601 format (e.g. `2014-10-31T09:32:57+09:00`), you can configure like below.
254
+
255
+ ```
256
+ time_format Time.at($time).iso8601
257
+ ```
258
+
215
259
 
216
260
  ## TODO
217
261
 
218
262
  - Multi-query combination
219
- - Support hyphen `-` and dollar `$` contained field names
220
263
 
221
264
 
222
265
  ## Copyright
@@ -3,7 +3,7 @@ $:.push File.expand_path("../lib", __FILE__)
3
3
 
4
4
  Gem::Specification.new do |spec|
5
5
  spec.name = "fluent-plugin-querycombiner"
6
- spec.version = "0.0.1"
6
+ spec.version = "0.0.2"
7
7
  spec.authors = ["Takahiro Kamatani"]
8
8
  spec.email = ["buhii314@gmail.com"]
9
9
  spec.description = %q{Fluent plugin to combine multiple queries.}
@@ -14,6 +14,8 @@ module Fluent
14
14
  config_param :query_ttl, :integer, :default => 1800
15
15
  config_param :buffer_size, :integer, :default => 1000
16
16
 
17
+ config_param :time_format, :string, :default => '$time'
18
+
17
19
  config_param :flush_interval, :integer, :default => 60
18
20
  config_param :remove_interval, :integer, :default => 10
19
21
  config_param :tag, :string, :default => "query_combiner"
@@ -24,6 +26,7 @@ module Fluent
24
26
  require 'msgpack'
25
27
  require 'json'
26
28
  require 'rubygems'
29
+ require 'time'
27
30
  end
28
31
 
29
32
  def configure(conf)
@@ -34,6 +37,19 @@ module Fluent
34
37
 
35
38
  @query_identify = @query_identify.split(',').map { |qid| qid.strip }
36
39
 
40
+ # functions for time format
41
+ def create_time_formatter(expr)
42
+ begin
43
+ f = eval('lambda {|__arg_time__| ' + expr.gsub("$time", "__arg_time__") + '}')
44
+ return f
45
+ rescue SyntaxError
46
+ raise Fluent::ConfigError, "SyntaxError at time_format `#{expr}`"
47
+ end
48
+ end
49
+ @_time_formatter = create_time_formatter(@time_format)
50
+
51
+ @_time_keys = {}
52
+
37
53
  # Create functions for each conditions
38
54
  @_cond_funcs = {}
39
55
  @_replace_keys = {
@@ -88,6 +104,16 @@ module Fluent
88
104
  else
89
105
  raise Fluent::ConfigError, "`replace` configuration in #{element.name}: only allowed in `catch` and `dump`"
90
106
  end
107
+
108
+ elsif var == 'time'
109
+ if %w{catch dump}.include? element.name
110
+ @_time_keys[element.name] = expr
111
+ else
112
+ raise Fluent::ConfigError, "`time` configuration in #{element.name}: only allowed in `catch` and `dump`"
113
+ end
114
+
115
+ else
116
+ raise Fluent::ConfigError, "Unknown configuration `#{var}` in #{element.name}"
91
117
  end
92
118
  }
93
119
  }
@@ -164,6 +190,11 @@ module Fluent
164
190
  record[after] = record[before]
165
191
  record.delete(before)
166
192
  }
193
+ # add time key if configured
194
+ if @_time_keys.has_key? 'catch'
195
+ record[@_time_keys['catch']] = @_time_formatter.call(time)
196
+ end
197
+
167
198
  # save record
168
199
  tryOnRedis 'set', @redis_key_prefix + qid, JSON.dump(record)
169
200
  # update qid's timestamp
@@ -179,7 +210,7 @@ module Fluent
179
210
  end
180
211
  end
181
212
 
182
- def do_dump(qid, record)
213
+ def do_dump(qid, record, time)
183
214
  if (tryOnRedis 'exists', @redis_key_prefix + qid)
184
215
  # replace record keys
185
216
  @_replace_keys['dump'].each_pair { |before, after|
@@ -187,6 +218,11 @@ module Fluent
187
218
  record.delete(before)
188
219
  }
189
220
 
221
+ # add time key if configured
222
+ if @_time_keys.has_key? 'dump'
223
+ record[@_time_keys['dump']] = @_time_formatter.call(time)
224
+ end
225
+
190
226
  # emit
191
227
  catched_record = JSON.load(tryOnRedis('get', @redis_key_prefix + qid))
192
228
  combined_record = catched_record.merge(record)
@@ -229,7 +265,7 @@ module Fluent
229
265
  when "prolong"
230
266
  do_prolong(qid, time)
231
267
  when "dump"
232
- do_dump(qid, record)
268
+ do_dump(qid, record, time)
233
269
  when "release"
234
270
  do_release(qid)
235
271
  end
@@ -81,6 +81,27 @@ class QueryCombinerOutputTest < Test::Unit::TestCase
81
81
  ]
82
82
  }
83
83
 
84
+ # `time` configuration only allowed in <catch> and <dump>
85
+ assert_raise(Fluent::ConfigError) {
86
+ d = create_driver %[
87
+ query_identify event_id
88
+ <catch>
89
+ condition status == 'start'
90
+ replace hoge => hoge_start
91
+ time time_catch
92
+ </catch>
93
+ <release>
94
+ condition status == 'error'
95
+ time time_release
96
+ </release>
97
+ <dump>
98
+ condition status == 'finish'
99
+ replace hoge => hoge_finish
100
+ time time_dump
101
+ </dump>
102
+ ]
103
+ }
104
+
84
105
  end
85
106
 
86
107
  def test_readme_sample_basic_example
@@ -137,6 +158,36 @@ class QueryCombinerOutputTest < Test::Unit::TestCase
137
158
  "time_finish"=>"2001-02-03T04:15:11Z"}
138
159
  end
139
160
 
161
+ def test_readme_sample_replace_multiple_fields
162
+ d = create_driver %[
163
+ query_identify event_id
164
+ query_ttl 3600 # time to live[sec]
165
+ buffer_size 1000 # queries
166
+
167
+ <catch>
168
+ condition status == 'event-start'
169
+ replace time => time_start, condition => condition_start
170
+ </catch>
171
+
172
+ <dump>
173
+ condition status == 'event-finish'
174
+ replace time => time_finish, condition => condition_end
175
+ </dump>
176
+ ]
177
+ time = Time.now.to_i
178
+ d.emit({"event_id"=>"01234567", "status"=>"event-start", "time"=>"2001-02-03T04:05:06Z", "condition"=>"bad"}, time)
179
+ d.emit({"event_id"=>"01234567", "status"=>"event-finish", "time"=>"2001-02-03T04:15:11Z", "condition"=>"excellent"}, time)
180
+ d.run
181
+ assert_equal d.emits.length, 1
182
+ assert_equal d.emits[0][2], {
183
+ "event_id"=>"01234567",
184
+ "status"=>"event-finish",
185
+ "time_start"=>"2001-02-03T04:05:06Z",
186
+ "condition_start"=>"bad",
187
+ "time_finish"=>"2001-02-03T04:15:11Z",
188
+ "condition_end"=>"excellent"}
189
+ end
190
+
140
191
  def test_readme_sample_release
141
192
  d = create_driver %[
142
193
  query_identify event_id
@@ -317,6 +368,48 @@ class QueryCombinerOutputTest < Test::Unit::TestCase
317
368
  assert_equal d.emits.size, finish_list.size
318
369
  end
319
370
 
371
+ def test_time_format_and_configuration
372
+ d = create_driver %[
373
+ query_identify event_id
374
+ query_ttl 3600 # time to live[sec]
375
+ buffer_size 1000 # queries
376
+ time_format Time.at($time).iso8601(3)
377
+
378
+ <catch>
379
+ condition status == 'event-start'
380
+ time time-catch
381
+ </catch>
382
+
383
+ <dump>
384
+ condition status == 'event-finish'
385
+ time time-dump
386
+ </dump>
387
+
388
+ <prolong>
389
+ condition status == 'event-continue'
390
+ </prolong>
391
+
392
+ <release>
393
+ condition status == 'event-error'
394
+ </release>
395
+ ]
396
+ def emit(d, status)
397
+ d.emit({"event_id"=>"01234567", "status"=>status}, Time.now.to_f)
398
+ end
399
+
400
+ emit(d, "event-start")
401
+ emit(d, "event-continue")
402
+ emit(d, "event-continue")
403
+ emit(d, "event-continue")
404
+ emit(d, "event-finish")
405
+ d.run
406
+
407
+ assert_equal d.emits.length, 1
408
+ assert_not_nil d.emits[0][2]['time-catch']
409
+ assert_not_nil d.emits[0][2]['time-dump']
410
+
411
+ end
412
+
320
413
  def test_buffer_size
321
414
 
322
415
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-querycombiner
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Takahiro Kamatani
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2014-07-15 00:00:00.000000000 Z
11
+ date: 2014-10-31 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: rake