fluent-plugin-groonga-query-log 1.0.0 → 1.0.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: acf3ada166906b4c1583d559239d1af2e4917fb4
4
- data.tar.gz: bb12b02153446679022f26e28ad299ca50c6b4fc
2
+ SHA256:
3
+ metadata.gz: 74fcfe61f76412fb02a18b924ecf9538f6792d920caa88fda7b22f7bd1d7216c
4
+ data.tar.gz: b81d0f452cb132cd280b79ec67b9bbea86a5531a5b47cb584dda49423ca3cdc7
5
5
  SHA512:
6
- metadata.gz: d7292a34808e12ba99aa0f381fe42d77e570819f016452c738611d9f7b203d29ed951181c6a423ed561aefb956b4e3852d0a2e92b36fcd0222d0dbc87c41d5ab
7
- data.tar.gz: 69d4bb292bf90fc2c8ea641aa02915bdc03b47dacbc34ba6738b03c53b11565e9380d868f86442c1e8255c5d2202429fd128be84e55fa79effce7b67568f9f90
6
+ metadata.gz: 5ecc7257af50cd3868ab7f3327bf9e9ad838357aa6bef2c1d3d13c7fbbe3cededa54dde3d55fdb181a5a11473b3b0adde010ac643ae73700913fc26a53c5abf1
7
+ data.tar.gz: d5054e6b5967f2f113f2bdb582c3759cfc5a099d9ff00adef9ce9bd8b3174ed6bc1c91262a409c7b079e5a8832441389cb1703acfb7068595caa7d98d35ba8b2
data/README.md CHANGED
@@ -6,20 +6,311 @@ fluent-plugin-groonga-query-log
6
6
 
7
7
  ## Description
8
8
 
9
- Fluent-plugin-groonga-query-log is a Fluentd plugin to parse
9
+ Fluent-plugin-groonga-query-log is a Fluentd filter plugin to parse
10
10
  [Groonga](http://groonga.org/)'s
11
11
  [query log](http://groonga.org/docs/reference/log.html#query-log) with
12
12
  Fluentd.
13
13
 
14
14
  You can detect slow query in real time by using this plugin.
15
15
 
16
+ Here is a sample configuration that stores slow queries to Groonga:
17
+
18
+ <source>
19
+ @type tail
20
+ path /var/log/groonga/query.log
21
+ pos_file /var/log/fluentd/groonga-query-log.pos
22
+ tag groonga.query
23
+ format none
24
+ </source>
25
+
26
+ <filter groonga.query>
27
+ @type groonga_query_log
28
+ </filter>
29
+
30
+ <filter groonga.query>
31
+ @type grep
32
+ regexp1 slow \Atrue\z
33
+ </filter>
34
+
35
+ <filter groonga.query>
36
+ @type record_transformer
37
+ enable_ruby true
38
+ renew_record true
39
+ keep_keys elapsed
40
+ <record>
41
+ log ${JSON.generate(to_h)}
42
+ </record>
43
+ </filter>
44
+
45
+ <match groonga.query>
46
+ @type groonga
47
+ store_table SlowQueries
48
+
49
+ protocol http
50
+ host 127.0.0.1
51
+
52
+ buffer_type file
53
+ buffer_path /var/lib/fluentd/groonga.buffer
54
+ flush_interval 1
55
+ </match>
56
+
57
+ You need to prepare your environment to use the configuration.
58
+
59
+ Create the following directories:
60
+
61
+ % sudo mkdir -p /var/log/fluentd
62
+ % sudo mkdir -p /var/lib/fluentd
63
+
64
+ User who runs Fluentd must have write permission of the
65
+ directories. Set suitable permission to the directories:
66
+
67
+ % sudo chown -R fluentd-user:fluentd-user /var/log/fluentd
68
+ % sudo chown -R fluentd-user:fluentd-user /var/lib/fluentd
69
+
70
+ Run Groonga that stores slow queries on `127.0.0.1`:
71
+
72
+ % groonga --protocol http -s DB_PATH
73
+
74
+ Run `fluentd` with the configuration:
75
+
76
+ % fluentd --config groonga-slow-queries.conf
77
+
78
+ Now, slow queries are stored `SlowQueries` table in Groonga:
79
+
80
+ % curl 'localhost:10041/d/select?table=SlowQueries&output_pretty=yes'
81
+ [
82
+ [
83
+ 0,
84
+ 1453454123.58033,
85
+ 8.70227813720703e-05
86
+ ],
87
+ [
88
+ [
89
+ [
90
+ 66
91
+ ],
92
+ [
93
+ [
94
+ "_id",
95
+ "UInt32"
96
+ ],
97
+ [
98
+ "elapsed",
99
+ "Float"
100
+ ],
101
+ [
102
+ "log",
103
+ "Text"
104
+ ]
105
+ ],
106
+ [
107
+ 1,
108
+ 0.265,
109
+ "{\"start_time\":...}"
110
+ ],
111
+ [
112
+ 2,
113
+ 0.303,
114
+ "{\"start_time\":...}"
115
+ ],
116
+ ...
117
+ ]
118
+ ]
119
+ ]
120
+
121
+ Each query log is stored as one record. Record has the following two
122
+ columns:
123
+
124
+ * `elapsed`: The elapsed time to execute the query.
125
+
126
+ * `log`: The query details as JSON. It includes executed command,
127
+ elapsed time for each condition and so on.
128
+
16
129
  ## Install
17
130
 
18
131
  % gem install fluent-plugin-groonga-query-log
19
132
 
20
133
  ## Usage
21
134
 
22
- TODO
135
+ You can use `groonga-query-log` filter for parsing raw Groonga's query
136
+ log text.
137
+
138
+ Here is a sample raw Groonga's query log text:
139
+
140
+ 2015-08-12 15:50:40.130990|0x7fb07d113da0|>/d/select?table=Entries&match_columns=name&query=xml
141
+ 2015-08-12 15:50:40.296165|0x7fb07d113da0|:000000165177838 filter(10)
142
+ 2015-08-12 15:50:40.296172|0x7fb07d113da0|:000000165184723 select(10)
143
+ 2015-08-12 15:50:41.228129|0x7fb07d113da0|:000001097153433 output(10)
144
+ 2015-08-12 15:50:41.228317|0x7fb07d113da0|<000001097334986 rc=0
145
+
146
+ `groonga-query-log` filter emits the following record by parsing the
147
+ above raw Groonga's query log text:
148
+
149
+ {
150
+ "start_time": "2015-08-12T06:50:40.130990Z",
151
+ "last_time": "2015-08-12T06:50:41.228324Z",
152
+ "elapsed": 1.0973349860000001,
153
+ "return_code": 0,
154
+ "slow": true,
155
+ "command": {
156
+ "raw": "/d/select?table=Entries&match_columns=name&query=xml",
157
+ "name": "select",
158
+ "parameters": [
159
+ {
160
+ "key": "table",
161
+ "value": "Entries"
162
+ },
163
+ {
164
+ "key": "match_columns",
165
+ "value": "name"
166
+ },
167
+ {
168
+ "key": "query",
169
+ "value": "xml"
170
+ }
171
+ ]
172
+ },
173
+ "operations": [
174
+ {
175
+ "context": "query: xml",
176
+ "name": "filter",
177
+ "relative_elapsed": 0.165177838,
178
+ "slow": true
179
+ },
180
+ {
181
+ "context": null,
182
+ "name": "select",
183
+ "relative_elapsed": 6.884999999999999e-06,
184
+ "slow": false
185
+ },
186
+ {
187
+ "context": null,
188
+ "name": "output",
189
+ "relative_elapsed": 0.93196871,
190
+ "slow": true
191
+ }
192
+ ]
193
+ }
194
+
195
+ Here are parameters of this filter:
196
+
197
+ * `raw_data_column_name`: It specifies column name that stores raw
198
+ Groonga's query log text.
199
+ * Default: `message`
200
+
201
+ * `slow_operation_threshold`: It specifies threshold to treat an
202
+ operation is slow. If one or more operations in a query spend more
203
+ than the threshold, the query is slow query.
204
+ * Default: `0.1`
205
+
206
+ * `slow_response_threshold`: It specifies threshold to treat a
207
+ request is slow. If a request spends more than the threshold, the
208
+ query in the request is slow query.
209
+ * Default: `0.2`
210
+
211
+ * `flatten`: It specifies whether parsed query log is mapped to a
212
+ flat object or a nested object. A float object will be useful to
213
+ store the parsed log to non document oriented database such as
214
+ RDBMS.
215
+
216
+ Here is a sample record of parsed query log:
217
+
218
+ {
219
+ ...,
220
+ "command": {
221
+ "raw": "/d/select?table=Entries&match_columns=name&query=xml",
222
+ "name": "select",
223
+ "parameters": [
224
+ {
225
+ "key": "table",
226
+ "value": "Entries"
227
+ },
228
+ {
229
+ "key": "match_columns",
230
+ "value": "name"
231
+ },
232
+ {
233
+ "key": "query",
234
+ "value": "xml"
235
+ }
236
+ ]
237
+ },
238
+ ...
239
+ }
240
+
241
+ Here is the flatten record of the above record:
242
+
243
+ {
244
+ ...,
245
+ "command.raw": "/d/select?table=Entries&match_columns=name&query=xml",
246
+ "command.name": "select",
247
+ "command.parameters[0].key": "table",
248
+ "command.parameters[0].value": "Entries",
249
+ "command.parameters[1].key": "match_columns",
250
+ "command.parameters[1].value": "name",
251
+ "command.parameters[0].key": "query",
252
+ "command.parameters[0].value": "xml",
253
+ ...
254
+ }
255
+
256
+ * Default: `false` (nested object)
257
+
258
+ * `flatten_separator`: It specifies separator that is used when
259
+ `flatten` is `true`. If `flatten` is `true`, nested keys are
260
+ mapped to one flatten key. This separator is used to concatenate
261
+ nested keys.
262
+
263
+ `.` is used for nested object by default. For example,
264
+
265
+ {
266
+ "a": {
267
+ "b": 1
268
+ }
269
+ }
270
+
271
+ is flatten to the following:
272
+
273
+ {
274
+ "a.b": 1
275
+ }
276
+
277
+ `[...]` is used for element in an array by default. For example,
278
+
279
+ {
280
+ "a": [
281
+ 1,
282
+ 2
283
+ ]
284
+ }
285
+
286
+ is flatten to the following:
287
+
288
+ {
289
+ "a[0]": 1,
290
+ "a[1]": 2
291
+ }
292
+
293
+ If `"_"` is used as the separator,
294
+
295
+ {
296
+ "a": [
297
+ 1,
298
+ 2
299
+ ],
300
+ "b": {
301
+ "c": 3
302
+ }
303
+ }
304
+
305
+ is flatten to the following:
306
+
307
+ {
308
+ "a_0": 1,
309
+ "a_1": 2,
310
+ "b_c": 3
311
+ }
312
+
313
+ * Default: `.` for object and `[...]` for array
23
314
 
24
315
  ## Authors
25
316
 
@@ -27,7 +318,7 @@ TODO
27
318
 
28
319
  ## License
29
320
 
30
- LGPL 3. See doc/text/lgpl-3.txt for details.
321
+ LGPL 3 or later. See doc/text/lgpl-3.txt for details.
31
322
 
32
323
  (Kouhei Sutou has a right to change the license including
33
324
  contributed patches.)
data/doc/text/news.md CHANGED
@@ -1,5 +1,11 @@
1
1
  # News
2
2
 
3
+ ## 1.0.1: 2019-01-09
4
+
5
+ ### Improvements
6
+
7
+ * Stopped to use deprecated groonga-query-log API.
8
+
3
9
  ## 1.0.0: 2015-08-12
4
10
 
5
11
  The first release!!!
@@ -1,6 +1,6 @@
1
1
  # -*- mode: ruby -*-
2
2
  #
3
- # Copyright (C) 2015 Kouhei Sutou <kou@clear-code.com>
3
+ # Copyright (C) 2015-2018 Kouhei Sutou <kou@clear-code.com>
4
4
  #
5
5
  # This library is free software: you can redistribute it and/or modify
6
6
  # it under the terms of the GNU Lesser General Public License as published by
@@ -17,7 +17,7 @@
17
17
 
18
18
  Gem::Specification.new do |spec|
19
19
  spec.name = "fluent-plugin-groonga-query-log"
20
- spec.version = "1.0.0"
20
+ spec.version = "1.0.1"
21
21
  spec.authors = ["Kouhei Sutou"]
22
22
  spec.email = ["kou@clear-code.com"]
23
23
  spec.summary = "Fluentd plugin to parse Groonga's query log."
@@ -34,7 +34,7 @@ Gem::Specification.new do |spec|
34
34
  spec.require_paths = ["lib"]
35
35
 
36
36
  spec.add_runtime_dependency("fluentd")
37
- spec.add_runtime_dependency("groonga-query-log")
37
+ spec.add_runtime_dependency("groonga-query-log", ">= 1.2.9")
38
38
 
39
39
  spec.add_development_dependency("rake")
40
40
  spec.add_development_dependency("bundler")
@@ -1,4 +1,4 @@
1
- # Copyright (C) 2015 Kouhei Sutou <kou@clear-code.com>
1
+ # Copyright (C) 2015-2018 Kouhei Sutou <kou@clear-code.com>
2
2
  #
3
3
  # This library is free software: you can redistribute it and/or modify
4
4
  # it under the terms of the GNU Lesser General Public License as published by
@@ -15,7 +15,7 @@
15
15
 
16
16
  require "time"
17
17
 
18
- require "groonga/query-log"
18
+ require "groonga-query-log"
19
19
 
20
20
  module Fluent
21
21
  class GroongaQueryLogFilter < Filter
@@ -30,7 +30,11 @@ module Fluent
30
30
  def configure(conf)
31
31
  super
32
32
 
33
- @parser = Groonga::QueryLog::Parser.new
33
+ options = {
34
+ :slow_operation_threshold => @slow_operation_threshold,
35
+ :slow_response_threshold => @slow_response_threshold,
36
+ }
37
+ @parser = GroongaQueryLog::Parser.new(options)
34
38
  end
35
39
 
36
40
  def filter_stream(tag, event_stream)
data/sample/dump.conf CHANGED
@@ -1,5 +1,5 @@
1
1
  <source>
2
- type tail
2
+ @type tail
3
3
  path "#{ENV['GROONGA_QUERY_LOG_PATH'] || '/var/log/groonga/query.log'}"
4
4
  read_from_head true
5
5
  tag groonga.query
@@ -7,9 +7,9 @@
7
7
  </source>
8
8
 
9
9
  <filter groonga.query>
10
- type groonga_query_log
10
+ @type groonga_query_log
11
11
  </filter>
12
12
 
13
13
  <match **>
14
- type stdout
14
+ @type stdout
15
15
  </match>
@@ -0,0 +1,41 @@
1
+ <source>
2
+ @type tail
3
+ path "#{ENV['GROONGA_QUERY_LOG_PATH'] || '/var/log/groonga/query.log'}"
4
+ pos_file /var/log/fluentd/groonga-query-log.pos
5
+ read_from_head "#{ENV['GROONGA_QUERY_LOG_READ_FROM_HEAD'] || 'false'}"
6
+ tag groonga.query
7
+ format none
8
+ </source>
9
+
10
+ <filter groonga.query>
11
+ @type groonga_query_log
12
+ </filter>
13
+
14
+ <filter groonga.query>
15
+ @type grep
16
+ regexp1 slow \Atrue\z
17
+ </filter>
18
+
19
+ <filter groonga.query>
20
+ @type record_transformer
21
+ enable_ruby true
22
+ renew_record true
23
+ keep_keys elapsed
24
+ <record>
25
+ log ${JSON.generate(to_h)}
26
+ </record>
27
+ </filter>
28
+
29
+ <match groonga.query>
30
+ @type groonga
31
+ store_table SlowQueries
32
+
33
+ protocol http
34
+ host 127.0.0.1
35
+
36
+ <buffer>
37
+ @type file
38
+ path /var/lib/fluentd/groonga.buffer
39
+ flush_interval 1
40
+ </buffer>
41
+ </match>
data/sample/store.conf CHANGED
@@ -1,5 +1,5 @@
1
1
  <source>
2
- type tail
2
+ @type tail
3
3
  path "#{ENV['GROONGA_QUERY_LOG_PATH'] || '/var/log/groonga/query.log'}"
4
4
  pos_file /var/log/td-agent/groonga-query-log.pos
5
5
  read_from_head "#{ENV['GROONGA_QUERY_LOG_READ_FROM_HEAD'] || 'false'}"
@@ -8,19 +8,21 @@
8
8
  </source>
9
9
 
10
10
  <filter groonga.query>
11
- type groonga_query_log
11
+ @type groonga_query_log
12
12
  flatten true
13
13
  flatten_separator _
14
14
  </filter>
15
15
 
16
16
  <match groonga.query>
17
- type groonga
18
- store_table QueyLogs
17
+ @type groonga
18
+ store_table QueryLogs
19
19
 
20
20
  protocol http
21
21
  host 127.0.0.1
22
22
 
23
- buffer_type file
24
- buffer_path /tmp/buffer
25
- flush_interval 1
23
+ <buffer>
24
+ @type file
25
+ path /tmp/buffer
26
+ flush_interval 1
27
+ </buffer>
26
28
  </match>
@@ -231,5 +231,97 @@ class GroongaQueryLogFilterTest < Test::Unit::TestCase
231
231
  CONFIGURATION
232
232
  assert_equal([[@now, statistic]], event_stream.to_a)
233
233
  end
234
+
235
+ test "slow_operation_threshold" do
236
+ messages = [
237
+ "2017-07-12 15:00:00.000000|0x7fb07d113da0|>/d/select?table=Entries&match_columns=name&query=xml",
238
+ "2017-07-12 15:00:00.100000|0x7fb07d113da0|:000000010000000 filter(10)",
239
+ "2017-07-12 15:00:00.200000|0x7fb07d113da0|:000000020000000 select(10)",
240
+ "2017-07-12 15:00:00.300000|0x7fb07d113da0|:000000030000000 output(10)",
241
+ "2017-07-12 15:00:00.400000|0x7fb07d113da0|<000000040000000 rc=0",
242
+ ]
243
+ statistics = {
244
+ "start_time" => "2017-07-12T06:00:00.000000Z",
245
+ "last_time" => "2017-07-12T06:00:00.040000Z",
246
+ "elapsed" => 0.04,
247
+ "return_code" => 0,
248
+ "slow" => false,
249
+ "command" => {
250
+ "name"=>"select",
251
+ "parameters" => [
252
+ {"key" => :table, "value" => "Entries"},
253
+ {"key" => :match_columns, "value" => "name"},
254
+ {"key" => :query, "value" => "xml"}
255
+ ],
256
+ "raw" => "/d/select?table=Entries&match_columns=name&query=xml"},
257
+ "operations" => [
258
+ {"context" => "query: xml",
259
+ "name" => "filter",
260
+ "relative_elapsed" => 0.01,
261
+ "slow" => true
262
+ },
263
+ {
264
+ "context" => nil,
265
+ "name" => "select",
266
+ "relative_elapsed" => 0.01,
267
+ "slow" => true
268
+ },
269
+ {
270
+ "context" => nil,
271
+ "name" => "output",
272
+ "relative_elapsed" => 0.01,
273
+ "slow" => true
274
+ }
275
+ ],
276
+ }
277
+ event_stream = emit("slow_operation_threshold 0.01", messages)
278
+ assert_equal([[@now, statistics]], event_stream.to_a)
279
+ end
280
+
281
+ test "slow_response_threshold" do
282
+ messages = [
283
+ "2017-07-12 15:00:00.000000|0x7fb07d113da0|>/d/select?table=Entries&match_columns=name&query=xml",
284
+ "2017-07-12 15:00:00.100000|0x7fb07d113da0|:000000010000000 filter(10)",
285
+ "2017-07-12 15:00:00.200000|0x7fb07d113da0|:000000020000000 select(10)",
286
+ "2017-07-12 15:00:00.300000|0x7fb07d113da0|:000000030000000 output(10)",
287
+ "2017-07-12 15:00:00.400000|0x7fb07d113da0|<000000040000000 rc=0",
288
+ ]
289
+ statistics = {
290
+ "start_time" => "2017-07-12T06:00:00.000000Z",
291
+ "last_time" => "2017-07-12T06:00:00.040000Z",
292
+ "elapsed" => 0.04,
293
+ "return_code" => 0,
294
+ "slow" => true,
295
+ "command" => {
296
+ "name"=>"select",
297
+ "parameters" => [
298
+ {"key" => :table, "value" => "Entries"},
299
+ {"key" => :match_columns, "value" => "name"},
300
+ {"key" => :query, "value" => "xml"}
301
+ ],
302
+ "raw" => "/d/select?table=Entries&match_columns=name&query=xml"},
303
+ "operations" => [
304
+ {"context" => "query: xml",
305
+ "name" => "filter",
306
+ "relative_elapsed" => 0.01,
307
+ "slow" => false
308
+ },
309
+ {
310
+ "context" => nil,
311
+ "name" => "select",
312
+ "relative_elapsed" => 0.01,
313
+ "slow" => false
314
+ },
315
+ {
316
+ "context" => nil,
317
+ "name" => "output",
318
+ "relative_elapsed" => 0.01,
319
+ "slow" => false
320
+ }
321
+ ],
322
+ }
323
+ event_stream = emit("slow_response_threshold 0.01", messages)
324
+ assert_equal([[@now, statistics]], event_stream.to_a)
325
+ end
234
326
  end
235
327
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-groonga-query-log
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.0.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Kouhei Sutou
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-08-12 00:00:00.000000000 Z
11
+ date: 2019-01-09 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -30,14 +30,14 @@ dependencies:
30
30
  requirements:
31
31
  - - ">="
32
32
  - !ruby/object:Gem::Version
33
- version: '0'
33
+ version: 1.2.9
34
34
  type: :runtime
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
38
  - - ">="
39
39
  - !ruby/object:Gem::Version
40
- version: '0'
40
+ version: 1.2.9
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: rake
43
43
  requirement: !ruby/object:Gem::Requirement
@@ -109,6 +109,7 @@ files:
109
109
  - fluent-plugin-groonga-query-log.gemspec
110
110
  - lib/fluent/plugin/filter_groonga_query_log.rb
111
111
  - sample/dump.conf
112
+ - sample/store-slow-query.conf
112
113
  - sample/store.conf
113
114
  - test/run-test.rb
114
115
  - test/test_filter_groonga_query_log.rb
@@ -131,12 +132,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
131
132
  - !ruby/object:Gem::Version
132
133
  version: '0'
133
134
  requirements: []
134
- rubyforge_project:
135
- rubygems_version: 2.2.2
135
+ rubygems_version: 3.0.2
136
136
  signing_key:
137
137
  specification_version: 4
138
138
  summary: Fluentd plugin to parse Groonga's query log.
139
139
  test_files:
140
140
  - test/run-test.rb
141
141
  - test/test_filter_groonga_query_log.rb
142
- has_rdoc: