fluent-plugin-groonga 1.0.3 → 1.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 34f2b071eacb59e8af9d91a384a8227e5ea64f58
4
- data.tar.gz: aacc29322894f6a03cd5681e3bd7870b64df6bb7
3
+ metadata.gz: 91510d22d9e94a522c9ee54b3a4d0caf4c6f4cc9
4
+ data.tar.gz: 0e77c6fc21121575977a288c131272c780ae0ecb
5
5
  SHA512:
6
- metadata.gz: 9e82b5797718f9fde41edf6c8f74cfb8ee484fe98f932aaacbfe58fd0c179b141dd4389a29d5f9a3fb69c4bb56cd4166d417580828e34d71dad2ef07ebb3aa51
7
- data.tar.gz: 6d4309efaaf8bbbfb46e5f46f99322b4ad8f810425a2ea80b2eaa8f1c5b393884f2293b4eefd4d3b40072c7f755c1b423bc7cf957a144c53772db6872b876600
6
+ metadata.gz: 6d00c8ff58928fc8c94b1a6ab98f803cac53517a022bb968ecf1b7e4862442e89f6583c89cedcd2b3b6022e3ff6acb82bb9ad270ec17a98b14b998c00727425a
7
+ data.tar.gz: 1fb6b8d3faeae3bf096b1df0845f590620ecf08cc41c2e4234ade39297e42efecd2a908c61f64ad316096ddba98c9833c2a8b6439e83ed178133374617b06fc6
data/README.md CHANGED
@@ -8,19 +8,59 @@ fluent-plugin-groonga
8
8
 
9
9
  ## Description
10
10
 
11
- Fluent-plugin-groonga is fluentd plugin collection for
12
- [groonga](http://groonga.org/) users. Groonga users can replicate
13
- their data by fluent-plugin-groonga.
11
+ Fluent-plugin-groonga is a Fluentd plugin collection to use
12
+ [Groonga](http://groonga.org/) with Fluentd. Fluent-plugin-groonga
13
+ supports the following two usages:
14
+
15
+ * Store logs collected by Fluentd to Groonga.
16
+ * Implement replication system for Groonga.
17
+
18
+ The first usage is normal usage. You can store logs to Groonga and
19
+ find logs by full-text search.
20
+
21
+ The second usage is for Groonga users. Groonga itself doesn't support
22
+ replication. But Groonga users can replicate their data by
23
+ fluent-plugin-groonga.
14
24
 
15
25
  Fluent-plugin-groonga includes an input plugin and an output
16
26
  plugin. Both of them are named `groonga`.
17
27
 
18
- The input plugin provides groonga compatible interface. It means that
19
- HTTP and GQTP interface. You can use the input plugin as groonga
20
- server. The input plugin receives groonga commands and sends them to
21
- the output plugin through zero or more fluentds.
28
+ If you want to use fluent-plugin-groonga to store logs to Groonga, you
29
+ need to use only `groonga` output plugin.
30
+
31
+ The following configuration stores all data in `/var/log/messages`
32
+ into Groonga:
33
+
34
+ <source>
35
+ type tail
36
+ format syslog
37
+ path /var/log/syslog.1
38
+ pos_file /tmp/messages.pos
39
+ tag log.messages
40
+ read_from_head true
41
+ </source>
42
+
43
+ <match log.**>
44
+ type groonga
45
+ table logs
46
+
47
+ protocol http
48
+ host 127.0.0.1
49
+
50
+ buffer_type file
51
+ buffer_path /tmp/buffer
52
+ flush_interval 1
53
+ </match>
22
54
 
23
- The output plugin sends received groonga commands to groonga. The
55
+ If you want to use fluent-plugin-groonga to implement Groonga
56
+ replication system, you need to use both plugins.
57
+
58
+ The input plugin provides Groonga compatible interface. It means that
59
+ HTTP and GQTP interface. You can use the input plugin as Groonga
60
+ server. The input plugin receives Groonga commands and sends them to
61
+ the output plugin through zero or more Fluentds.
62
+
63
+ The output plugin sends received Groonga commands to Groonga. The
24
64
  output plugin supports all interfaces, HTTP, GQTP and command
25
65
  interface.
26
66
 
@@ -32,6 +72,78 @@ You can replicate your data by using `copy` output plugin.
32
72
 
33
73
  ## Usage
34
74
 
75
+ There are two usages:
76
+
77
+ * Store logs collected by Fluentd to Groonga.
78
+ * Implement replication system for Groonga.
79
+
80
+ They are described in other sections.
81
+
82
+ ### Store logs into Groonga
83
+
84
+ You need to use `groonga` output plugin to store logs into Groonga.
85
+
86
+ The output plugin has auto schema define feature. So you don't need to
87
+ define schema in Groonga before running Fluentd. You just run Groonga.
88
+
89
+ There is one required parameter:
90
+
91
+ * `table`: It specifies table name for storing logs.
92
+
93
+ Here is a minimum configuration:
94
+
95
+ <match log.**>
96
+ type groonga
97
+ table logs
98
+ </match>
99
+
100
+ The configuration stores logs into `logs` table in Groonga that runs
101
+ on `localhost`.
102
+
103
+ There are optional parameters:
104
+
105
+ * `protocol`: It specifies protocol to communicate Groonga server.
106
+ * Available values: `http`, `gqtp`, `command`
107
+ * Default: `http`
108
+ * `host`: It specifies host name or IP address of Groonga server.
109
+ * Default: `127.0.0.1`
110
+ * `port`: It specifies port number of Groonga server.
111
+ * Default for `http` protocol: `10041`
112
+ * Default for `gqtp` protocol: `10043`
113
+
114
+ Here is a configuration that specifies optional parameters explicitly:
115
+
116
+ <match log.**>
117
+ type groonga
118
+ table logs
119
+
120
+ protocol http
121
+ host 127.0.0.1
122
+ port 10041
123
+ </match>
124
+
125
+ `groonga` output plugin supports buffer. So you can use buffer related
126
+ parameters. See
127
+ [Buffer Plugin Overview | Fluentd](http://docs.fluentd.org/articles/buffer-plugin-overview)
128
+ for details.
129
+
130
+ Note that there is special tag name. You can't use
131
+ `groonga.command.XXX` tag name for this usage. It means that you can't
132
+ use the following configuration:
133
+
134
+ <match groonga.command.*>
135
+ type groonga
136
+ # ...
137
+ </match>
138
+
139
+ `groonga.command.XXX` tag name is reserved for implementing
140
+ replication system for Groonga.
141
+
142
+ ### Implement replication system for Groonga
143
+
144
+ See the following documents how to implement replication system for
145
+ Groonga:
146
+
35
147
  * [Configuration](doc/text/configuration.md)
36
148
  ([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.configuration.html))
37
149
  * [Constitution](doc/text/constitution.md)
@@ -2,6 +2,21 @@
2
2
 
3
3
  # News
4
4
 
5
+ ## 1.0.4: 2014-10-20
6
+
7
+ ### Improvements
8
+
9
+ * Supported the latest http_parser gem.
10
+ * Removed no buffer mode. Use `flush_interval 0` for no buffer like
11
+ behavior.
12
+ * Changed the default port number to `10043` for `gqtp` protocol usage.
13
+ Because Groonga changed the default port number for `gqtp` protocol.
14
+ * Reduced the number of `load` calls. It improves `load` performance.
15
+ * Supported auto schema define. You don't need to define schema in Groonga
16
+ before running Fluentd.
17
+ * Added document to use fluent-plugin-groonga to store logs into Groonga.
18
+ It fits normal Fluentd usage.
19
+
5
20
  ## 1.0.3: 2013-09-29
6
21
 
7
22
  ### Improvements
@@ -1,6 +1,6 @@
1
1
  # -*- mode: ruby; coding: utf-8 -*-
2
2
  #
3
- # Copyright (C) 2012-2013 Kouhei Sutou <kou@clear-code.com>
3
+ # Copyright (C) 2012-2014 Kouhei Sutou <kou@clear-code.com>
4
4
  #
5
5
  # This library is free software; you can redistribute it and/or
6
6
  # modify it under the terms of the GNU Lesser General Public
@@ -17,12 +17,12 @@
17
17
 
18
18
  Gem::Specification.new do |spec|
19
19
  spec.name = "fluent-plugin-groonga"
20
- spec.version = "1.0.3"
20
+ spec.version = "1.0.4"
21
21
  spec.authors = ["Kouhei Sutou"]
22
22
  spec.email = ["kou@clear-code.com"]
23
- spec.summary = "Fluentd plugin collection for groonga users"
23
+ spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
24
24
  spec.description =
25
- "Groonga users can replicate their data by fluent-plugin-groonga"
25
+ "There are two usages. 1) Store data into Groonga. 2) Implement Groonga replication system. See documentation for details."
26
26
  spec.homepage = "https://github.com/groonga/fluent-plugin-groonga"
27
27
  spec.license = "LGPL-2.1"
28
28
 
@@ -35,7 +35,7 @@ Gem::Specification.new do |spec|
35
35
  spec.require_paths = ["lib"]
36
36
 
37
37
  spec.add_runtime_dependency("fluentd")
38
- spec.add_runtime_dependency("gqtp", ">= 1.0.3")
38
+ spec.add_runtime_dependency("groonga-client")
39
39
  spec.add_runtime_dependency("groonga-command-parser")
40
40
 
41
41
  spec.add_development_dependency("rake")
@@ -1,6 +1,6 @@
1
1
  # -*- coding: utf-8 -*-
2
2
  #
3
- # Copyright (C) 2012-2013 Kouhei Sutou <kou@clear-code.com>
3
+ # Copyright (C) 2012-2014 Kouhei Sutou <kou@clear-code.com>
4
4
  #
5
5
  # This library is free software; you can redistribute it and/or
6
6
  # modify it under the terms of the GNU Lesser General Public
@@ -16,6 +16,7 @@
16
16
  # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
17
17
 
18
18
  require "English"
19
+ require "uri"
19
20
  require "webrick/httputils"
20
21
 
21
22
  require "http_parser"
@@ -31,19 +32,22 @@ module Fluent
31
32
  super
32
33
  end
33
34
 
34
- config_param :protocol, :string, :default => "http"
35
+ config_param :protocol, :defalut => :http do |value|
36
+ case value
37
+ when "http", "gqtp"
38
+ value.to_sym
39
+ else
40
+ raise ConfigError, "must be http or gqtp: <#{value}>"
41
+ end
42
+ end
35
43
 
36
44
  def configure(conf)
37
45
  super
38
46
  case @protocol
39
- when "http"
47
+ when :http
40
48
  @input = HTTPInput.new
41
- when "gqtp"
49
+ when :gqtp
42
50
  @input = GQTPInput.new
43
- else
44
- message = "unknown protocol: <#{@protocol.inspect}>"
45
- $log.error message
46
- raise ConfigError, message
47
51
  end
48
52
  @input.configure(conf)
49
53
  end
@@ -76,9 +80,9 @@ module Fluent
76
80
  include DetachMultiProcessMixin
77
81
 
78
82
  config_param :bind, :string, :default => "0.0.0.0"
79
- config_param :port, :integer, :default => 10041
83
+ config_param :port, :integer, :default => nil
80
84
  config_param :real_host, :string
81
- config_param :real_port, :integer, :default => 10041
85
+ config_param :real_port, :integer, :default => nil
82
86
  DEFAULT_EMIT_COMMANDS = [
83
87
  /\Atable_/,
84
88
  /\Acolumn_/,
@@ -102,6 +106,13 @@ module Fluent
102
106
  end
103
107
  end
104
108
 
109
+ def configure(conf)
110
+ super
111
+
112
+ @port ||= default_port
113
+ @real_port ||= default_port
114
+ end
115
+
105
116
  def start
106
117
  listen_socket = TCPServer.new(@bind, @port)
107
118
  detach_multi_process do
@@ -156,6 +167,10 @@ module Fluent
156
167
 
157
168
  class HTTPInput < BaseInput
158
169
  private
170
+ def default_port
171
+ 10041
172
+ end
173
+
159
174
  def handler_class
160
175
  Handler
161
176
  end
@@ -188,8 +203,9 @@ module Fluent
188
203
  end
189
204
 
190
205
  def on_message_complete
191
- params = WEBrick::HTTPUtils.parse_query(@parser.query_string)
192
- path_info = @parser.request_path
206
+ uri = URI.parse(@parser.request_url)
207
+ params = WEBrick::HTTPUtils.parse_query(uri.query)
208
+ path_info = uri.path
193
209
  case path_info
194
210
  when /\A\/d\//
195
211
  command = $POSTMATCH
@@ -204,6 +220,10 @@ module Fluent
204
220
 
205
221
  class GQTPInput < BaseInput
206
222
  private
223
+ def default_port
224
+ 10043
225
+ end
226
+
207
227
  def handler_class
208
228
  Handler
209
229
  end
@@ -1,6 +1,6 @@
1
1
  # -*- coding: utf-8 -*-
2
2
  #
3
- # Copyright (C) 2012 Kouhei Sutou <kou@clear-code.com>
3
+ # Copyright (C) 2012-2014 Kouhei Sutou <kou@clear-code.com>
4
4
  #
5
5
  # This library is free software; you can redistribute it and/or
6
6
  # modify it under the terms of the GNU Lesser General Public
@@ -17,23 +17,26 @@
17
17
 
18
18
  require "fileutils"
19
19
 
20
+ require "yajl"
21
+
22
+ require "groonga/client"
23
+
20
24
  module Fluent
21
- class GroongaOutput < Output
25
+ class GroongaOutput < BufferedOutput
22
26
  Plugin.register_output("groonga", self)
23
27
 
24
28
  def initialize
25
29
  super
26
30
  end
27
31
 
28
- BufferedOutput.config_params.each do |name, (block, options)|
29
- if options[:type]
30
- config_param(name, options[:type], options)
32
+ config_param :protocol, :default => :http do |value|
33
+ case value
34
+ when "http", "gqtp", "command"
35
+ value.to_sym
31
36
  else
32
- config_param(name, options, &block)
37
+ raise ConfigError, "must be http, gqtp or command: <#{value}>"
33
38
  end
34
39
  end
35
-
36
- config_param :protocol, :string, :default => "http"
37
40
  config_param :table, :string, :default => nil
38
41
 
39
42
  def configure(conf)
@@ -42,168 +45,318 @@ module Fluent
42
45
  @client.configure(conf)
43
46
 
44
47
  @emitter = Emitter.new(@client, @table)
45
- @output = create_output(@buffer_type, @emitter)
46
- @output.configure(conf)
47
48
  end
48
49
 
49
50
  def start
50
51
  super
51
52
  @client.start
52
- @output.start
53
+ @emitter.start
53
54
  end
54
55
 
55
56
  def shutdown
56
57
  super
57
- @output.shutdown
58
+ @emitter.shutdown
58
59
  @client.shutdown
59
60
  end
60
61
 
61
- def emit(tag, event_stream, chain)
62
- @output.emit(tag, event_stream, chain)
62
+ def format(tag, time, record)
63
+ [tag, time, record].to_msgpack
63
64
  end
64
65
 
66
+ def write(chunk)
67
+ @emitter.emit(chunk)
68
+ end
69
+
70
+ private
65
71
  def create_client(protocol)
66
72
  case protocol
67
- when "http"
68
- HTTPClient.new
69
- when "gqtp"
70
- GQTPClient.new
71
- when "command"
73
+ when :http, :gqtp
74
+ NetworkClient.new(protocol)
75
+ when :command
72
76
  CommandClient.new
73
77
  end
74
78
  end
75
79
 
76
- def create_output(buffer_type, emitter)
77
- if buffer_type == "none"
78
- RawGroongaOutput.new(emitter)
79
- else
80
- BufferedGroongaOutput.new(emitter)
80
+ class Schema
81
+ def initialize(client, table_name)
82
+ @client = client
83
+ @table_name = table_name
84
+ @table = nil
85
+ @columns = nil
81
86
  end
82
- end
83
87
 
84
- class Emitter
85
- def initialize(client, table)
86
- @client = client
87
- @table = table
88
+ def populate
89
+ # TODO
88
90
  end
89
91
 
90
- def emit(tag, record)
91
- if /\Agroonga\.command\./ =~ tag
92
- name = $POSTMATCH
93
- send_command(name, record)
94
- else
95
- store_chunk(data)
92
+ def update(records)
93
+ ensure_table
94
+ ensure_columns
95
+
96
+ nonexistent_columns = {}
97
+ records.each do |record|
98
+ record.each do |key, value|
99
+ column = @columns[key]
100
+ if column.nil?
101
+ nonexistent_columns[key] ||= []
102
+ nonexistent_columns[key] << value
103
+ end
104
+ end
105
+ end
106
+
107
+ nonexistent_columns.each do |name, values|
108
+ @columns[name] = create_column(name, values)
96
109
  end
97
110
  end
98
111
 
99
112
  private
100
- def send_command(name, arguments)
101
- command_class = Groonga::Command.find(name)
102
- command = command_class.new(name, arguments)
103
- @client.send(command)
113
+ def ensure_table
114
+ return if @table
115
+
116
+ table_list = @client.execute("table_list")
117
+ target_table = table_list.find do |table|
118
+ table.name == @table_name
119
+ end
120
+ if target_table
121
+ @table = Table.new(@table_name, target_table.domain)
122
+ else
123
+ # TODO: Check response
124
+ @client.execute("table_create",
125
+ "name" => @table_name,
126
+ "flags" => "TABLE_NO_KEY")
127
+ @table = Table.new(@table_name, nil)
128
+ end
104
129
  end
105
130
 
106
- def store_chunk(value)
107
- return if @table.nil?
131
+ def ensure_columns
132
+ return if @columns
108
133
 
109
- values = [value]
110
- arguments = {
111
- "table" => @table,
112
- "values" => Yajl::Enocder.encode(values),
113
- }
114
- send_command("load", arguments)
134
+ column_list = @client.execute("column_list", "table" => @table_name)
135
+ @columns = {}
136
+ column_list.each do |column|
137
+ vector_p = column.flags.split("|").include?("COLUMN_VECTOR")
138
+ @columns[column.name] = Column.new(column.name,
139
+ column.range,
140
+ vector_p)
141
+ end
115
142
  end
116
- end
117
143
 
118
- class RawGroongaOutput < Output
119
- def initialize(emitter)
120
- @emitter = emitter
121
- super()
144
+ def create_column(name, sample_values)
145
+ guesser = TypeGuesser.new(sample_values)
146
+ value_type = guesser.guess
147
+ vector_p = guesser.vector?
148
+ if vector_p
149
+ flags = "COLUMN_VECTOR"
150
+ else
151
+ flags = "COLUMN_SCALAR"
152
+ end
153
+ # TODO: Check response
154
+ @client.execute("column_create",
155
+ "table" => @table_name,
156
+ "name" => name,
157
+ "flags" => flags,
158
+ "type" => value_type)
159
+ Column.new(name, value_type, vector_p)
122
160
  end
123
161
 
124
- def emit(tag, event_stream, chain)
125
- event_stream.each do |time, record|
126
- @emitter.emit(tag, record)
162
+ class TypeGuesser
163
+ def initialize(sample_values)
164
+ @sample_values = sample_values
127
165
  end
128
- chain.next
129
- end
130
- end
131
166
 
132
- class BufferedGroongaOutput < BufferedOutput
133
- def initialize(emitter)
134
- @emitter = emitter
135
- super()
167
+ def guess
168
+ return "Time" if time_values?
169
+ return "Int32" if int32_values?
170
+ return "Int64" if int64_values?
171
+ return "Float" if float_values?
172
+ return "WGS84GeoPoint" if geo_point_values?
173
+
174
+ "Text"
175
+ end
176
+
177
+ def vector?
178
+ @sample_values.any? do |sample_value|
179
+ sample_value.is_a?(Array)
180
+ end
181
+ end
182
+
183
+ private
184
+ def time_values?
185
+ now = Time.now.to_i
186
+ year_in_seconds = 365 * 24 * 60 * 60
187
+ window = 10 * year_in_seconds
188
+ new = now + window
189
+ old = now - window
190
+ recent_range = old..new
191
+ @sample_values.all? do |sample_value|
192
+ sample_value.is_a?(Integer) and
193
+ recent_range.cover?(sample_value)
194
+ end
195
+ end
196
+
197
+ def integer_value?(value)
198
+ case value
199
+ when String
200
+ begin
201
+ Integer(value)
202
+ true
203
+ rescue ArgumentError
204
+ false
205
+ end
206
+ when Integer
207
+ true
208
+ else
209
+ false
210
+ end
211
+ end
212
+
213
+ def int32_values?
214
+ int32_min = -(2 ** 31)
215
+ int32_max = 2 ** 31 - 1
216
+ range = int32_min..int32_max
217
+ @sample_values.all? do |sample_value|
218
+ integer_value?(sample_value) and
219
+ range.cover?(Integer(sample_value))
220
+ end
221
+ end
222
+
223
+ def int64_values?
224
+ @sample_values.all? do |sample_value|
225
+ integer_value?(sample_value)
226
+ end
227
+ end
228
+
229
+ def float_value?(value)
230
+ case value
231
+ when String
232
+ begin
233
+ Float(value)
234
+ true
235
+ rescue ArgumentError
236
+ false
237
+ end
238
+ when Float
239
+ true
240
+ else
241
+ false
242
+ end
243
+ end
244
+
245
+ def float_values?
246
+ @sample_values.all? do |sample_value|
247
+ float_value?(sample_value)
248
+ end
249
+ end
250
+
251
+ def geo_point_values?
252
+ @sample_values.all? do |sample_value|
253
+ sample_value.is_a?(String) and
254
+ /\A-?\d+(?:\.\d+)[,x]-?\d+(?:\.\d+)\z/ =~ sample_value
255
+ end
256
+ end
136
257
  end
137
258
 
138
- def format(tag, time, record)
139
- [tag, time, record].to_msgpack
259
+ class Table
260
+ def initialize(name, key_type)
261
+ @name = name
262
+ @key_type = key_type
263
+ end
140
264
  end
141
265
 
142
- def write(chunk)
143
- chunk.msgpack_each do |tag, time, record|
144
- @emitter.emit(tag, record)
266
+ class Column
267
+ def initialize(name, value_type, vector_p)
268
+ @name = name
269
+ @value_type = value_type
270
+ @vector_p = vector_p
145
271
  end
146
272
  end
147
273
  end
148
274
 
149
- class HTTPClient
150
- include Configurable
151
-
152
- config_param :host, :string, :default => "localhost"
153
- config_param :port, :integer, :default => 10041
275
+ class Emitter
276
+ def initialize(client, table)
277
+ @client = client
278
+ @table = table
279
+ @schema = nil
280
+ end
154
281
 
155
282
  def start
156
- @loop = Coolio::Loop.new
283
+ @schema = Schema.new(@client, @table)
157
284
  end
158
285
 
159
286
  def shutdown
160
287
  end
161
288
 
162
- def send(command)
163
- client = GroongaHTTPClient.connect(@host, @port)
164
- client.request("GET", command.to_uri_format)
165
- @loop.attach(client)
166
- @loop.run
289
+ def emit(chunk)
290
+ records = []
291
+ chunk.msgpack_each do |message|
292
+ tag, _, record = message
293
+ if /\Agroonga\.command\./ =~ tag
294
+ name = $POSTMATCH
295
+ unless records.empty?
296
+ store_records(records)
297
+ records.clear
298
+ end
299
+ @client.execute(name, record)
300
+ else
301
+ records << record
302
+ end
303
+ end
304
+ store_records(records) unless records.empty?
167
305
  end
168
306
 
169
- class GroongaHTTPClient < Coolio::HttpClient
170
- def on_body_data(data)
171
- end
307
+ private
308
+ def store_records(records)
309
+ return if @table.nil?
310
+
311
+ @schema.update(records)
312
+
313
+ arguments = {
314
+ "table" => @table,
315
+ "values" => Yajl::Encoder.encode(records),
316
+ }
317
+ @client.execute("load", arguments)
318
+ end
319
+ end
320
+
321
+ class BaseClient
322
+ private
323
+ def build_command(name, arguments={})
324
+ command_class = Groonga::Command.find(name)
325
+ command_class.new(name, arguments)
172
326
  end
173
327
  end
174
328
 
175
- class GQTPClient
329
+ class NetworkClient < BaseClient
176
330
  include Configurable
177
331
 
178
- config_param :host, :string, :default => "localhost"
179
- config_param :port, :integer, :default => 10041
332
+ config_param :host, :string, :default => nil
333
+ config_param :port, :integer, :default => nil
334
+
335
+ def initialize(protocol)
336
+ super()
337
+ @protocol = protocol
338
+ end
180
339
 
181
340
  def start
182
- @loop = Coolio::Loop.new
183
341
  @client = nil
184
342
  end
185
343
 
186
344
  def shutdown
187
345
  return if @client.nil?
188
- @client.close do
189
- @loop.stop
190
- end
191
- @loop.run
346
+ @client.close
192
347
  end
193
348
 
194
- def send(command)
195
- @client ||= GQTP::Client.new(:address => @host,
196
- :port => @port,
197
- :connection => :coolio,
198
- :loop => @loop)
199
- @client.send(command.to_command_format) do |header, body|
200
- @loop.stop
201
- end
202
- @loop.run
349
+ def execute(name, arguments={})
350
+ command = build_command(name, arguments)
351
+ @client ||= Groonga::Client.new(:protocol => @protocol,
352
+ :host => @host,
353
+ :port => @port,
354
+ :backend => :synchronous)
355
+ @client.execute(command)
203
356
  end
204
357
  end
205
358
 
206
- class CommandClient
359
+ class CommandClient < BaseClient
207
360
  include Configurable
208
361
 
209
362
  config_param :groonga, :string, :default => "groonga"
@@ -222,42 +375,45 @@ module Fluent
222
375
 
223
376
  def start
224
377
  run_groonga
225
- wrap_io
226
378
  end
227
379
 
228
380
  def shutdown
229
- @groonga_input.close
230
- @groonga_output.close
231
- @groonga_error.close
381
+ @input.close
382
+ read_output("shutdown")
383
+ @output.close
384
+ @error.close
232
385
  Process.waitpid(@pid)
233
386
  end
234
387
 
235
- def send(command)
388
+ def execute(name, arguments={})
389
+ command = build_command(name, arguments)
236
390
  body = nil
237
391
  if command.name == "load"
238
392
  body = command.arguments.delete(:values)
239
393
  end
240
- @groonga_input.write("#{command.to_uri_format}\n")
394
+ uri = command.to_uri_format
395
+ @input.write("#{uri}\n")
241
396
  if body
242
397
  body.each_line do |line|
243
- @groonga_input.write("#{line}\n")
398
+ @input.write("#{line}\n")
244
399
  end
245
400
  end
246
- @loop.run
401
+ @input.flush
402
+ read_output(uri)
247
403
  end
248
404
 
249
405
  private
250
406
  def run_groonga
251
407
  env = {}
252
- @input = IO.pipe("ASCII-8BIT")
253
- @output = IO.pipe("ASCII-8BIT")
254
- @error = IO.pipe("ASCII-8BIT")
255
- input_fd = @input[0].to_i
256
- output_fd = @output[1].to_i
408
+ input = IO.pipe("ASCII-8BIT")
409
+ output = IO.pipe("ASCII-8BIT")
410
+ error = IO.pipe("ASCII-8BIT")
411
+ input_fd = input[0].to_i
412
+ output_fd = output[1].to_i
257
413
  options = {
258
414
  input_fd => input_fd,
259
415
  output_fd => output_fd,
260
- :err => @error[1],
416
+ :err => error[1],
261
417
  }
262
418
  arguments = @arguments
263
419
  arguments += [
@@ -270,27 +426,42 @@ module Fluent
270
426
  end
271
427
  arguments << @database
272
428
  @pid = spawn(env, @groonga, *arguments, options)
273
- @input[0].close
274
- @output[1].close
275
- @error[1].close
429
+ input[0].close
430
+ @input = input[1]
431
+ output[1].close
432
+ @output = output[0]
433
+ error[1].close
434
+ @error = error[0]
276
435
  end
277
436
 
278
- def wrap_io
279
- @loop = Coolio::Loop.new
437
+ def read_output(context)
438
+ output_message = ""
439
+ error_message = ""
440
+
441
+ loop do
442
+ readables = IO.select([@output, @error], nil, nil, 0)
443
+ break if readables.nil?
444
+
445
+ readables.each do |readable|
446
+ case readable
447
+ when @output
448
+ output_message << @output.gets
449
+ when @error
450
+ error_message << @error.gets
451
+ end
452
+ end
453
+ end
280
454
 
281
- @groonga_input = Coolio::IO.new(@input[1])
282
- on_write_complete = lambda do
283
- @loop.stop
455
+ unless output_message.empty?
456
+ Engine.log.debug("[output][groonga][output]",
457
+ :context => context,
458
+ :message => output_message)
284
459
  end
285
- @groonga_input.on_write_complete do
286
- on_write_complete.call
460
+ unless error_message.empty?
461
+ Engine.log.error("[output][groonga][error]",
462
+ :context => context,
463
+ :message => error_message)
287
464
  end
288
- @groonga_output = Coolio::IO.new(@output[0])
289
- @groonga_error = Coolio::IO.new(@error[0])
290
-
291
- @loop.attach(@groonga_input)
292
- @loop.attach(@groonga_output)
293
- @loop.attach(@groonga_error)
294
465
  end
295
466
  end
296
467
  end