fluent-plugin-groonga 1.1.8 → 1.2.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: f225a28b0bd052b37c5180b7707ac898f550ca57
4
- data.tar.gz: 0c93017fa42a9a2edac9b50d97d0e3d3e5cf0a94
3
+ metadata.gz: 58980511ad7775baea84617dc9ab6ed0f93e29dc
4
+ data.tar.gz: 63193a59c766554172b3d34a4d3f40c47465d594
5
5
  SHA512:
6
- metadata.gz: deb8587173008cf6ddc636aebe1fd7c2eb41ed01b806e91c020af81c1dc38eeccdbac196391ecc2289354458d3172707db85f9d9865a7690bc18f8ff00e917df
7
- data.tar.gz: 737f5f9db324b14199f4e2b1734d7bc76e1008b80906eba78febe99f0aae91a428a1db69d809715661473bbc4e4252557be4f75512796f42df6f178195eb89ef
6
+ metadata.gz: c44c1ad1b43b1144ff703d61092a8ccf76735c5dfdfce75f5f33023b7e1969b78154629205d68b52d0b6ff9889f0b0a2a254a1e225777ded1cfcc23529293a71
7
+ data.tar.gz: 7a1b9fdef294db2a6ab661ad7d4619486a887e82e5e674f6e0d1243cb301fc7281a6df8c3463cad04bcab63cf359ff8bc9be6edcb773a61234243b551bd0c6a8
@@ -8,7 +8,25 @@ describes configuration parameters of them.
8
8
 
9
9
  ## The `groonga` input plugin
10
10
 
11
- Here are available parameters:
11
+ The behavior of `groonga` input plugin is customized in `system`
12
+ directive and `source` directive parameters.
13
+
14
+ Here is recommended parameter in `system` directive:
15
+
16
+ * `workers`: It specifies the number of workers. The point of view
17
+ in performance, the recommended value is greater than 1 to process
18
+ requests in parallel.
19
+
20
+ * recommended value: greater than `1`.
21
+
22
+ * default: `1`
23
+
24
+ Note that there is one exception about the number of recommended
25
+ workers. Users should use `1` worker when commands contain DDL such as
26
+ `table_create` and `column_create`. Because execution order of these
27
+ commands may be swapped. In such a case, replication will fail.
28
+
29
+ Here are available parameters in `source` directive:
12
30
 
13
31
  * `protocol`: It specifies protocol for receiving Groonga commands.
14
32
 
@@ -32,28 +50,14 @@ Here are available parameters:
32
50
 
33
51
  * default: `10041`
34
52
 
35
- * `command_name_position`: It specifies where Groonga command's
36
- name.
37
-
38
- If `tag` is specified, the plugin puts Groonga command's name to
39
- tag as `groonga.command.#{COMMAND_NAME}` format and record
40
- is the arguments of the command.
41
-
42
- If `record` is specified, the plugin puts both Groonga command's
43
- name and arguments to record as `{"name": "#{COMMAND_NAME}",
44
- "arguments": {...}}` format. Tag is always `groonga.command`.
45
-
46
- `record` is suitable when you want to implement replication. If
47
- you `tag` for replication, Groonga command's order may be changed.
48
-
49
- * Available values: `tag`, `record`
50
-
51
- * default: `tag`
52
-
53
53
  * `emit_commands`: TODO
54
54
 
55
55
  Here is an example:
56
56
 
57
+ <system>
58
+ workers 2
59
+ </system>
60
+
57
61
  <source>
58
62
  @type groonga
59
63
  protocol http
@@ -61,7 +65,6 @@ Here is an example:
61
65
  port 10041
62
66
  real_host 192.168.0.1
63
67
  real_port 10041
64
- command_name_position record
65
68
  </source>
66
69
 
67
70
  ## The `groonga` output plugin
@@ -66,21 +66,19 @@ Here is an example configuration file:
66
66
  # For master Groonga server
67
67
  <source>
68
68
  @type groonga
69
- protocol gqtp # Or use the below line
69
+ protocol gqtp # Or use the below line
70
70
  # protocol http
71
- bind 127.0.0.1 # For client side Fluentd
72
- # bind 192.168.0.1 # For master Groonga server side Fluentd
71
+ bind 127.0.0.1 # For client side Fluentd
72
+ # bind 192.168.0.1 # For master Groonga server side Fluentd
73
73
  port 10041
74
- real_host 192.168.29.1 # IP address of master Groonga server
75
- real_port 10041 # Port number of master Groonga server
76
- # real_port 20041 # Use different port number
77
- # for master Groonga server side Fluentd
78
-
79
- command_name_position record # To keep command order
74
+ real_host 192.168.29.1 # IP address of master Groonga server
75
+ real_port 10041 # Port number of master Groonga server
76
+ # real_port 20041 # Use different port number
77
+ # for master Groonga server side Fluentd
80
78
  </source>
81
79
 
82
80
  # For slave Groonga server
83
- <match groonga.command.**>
81
+ <match groonga.command.*>
84
82
  @type groonga
85
83
  protocol gqtp # Or use the below line
86
84
  # protocol http # You can use different protocol for
@@ -128,7 +126,7 @@ You cannot update data until fluentd is up.
128
126
  Here are recover steps when master Groonga server is down:
129
127
 
130
128
  1. Stop fluentd.
131
- 2. Run `groonga /PATH/TO/SLAVE/GROONGA/SERVER/DB dump >
129
+ 2. Run `grndump /PATH/TO/SLAVE/GROONGA/SERVER/DB >
132
130
  SLAVE_GROONGA_DUMP.grn` on slave Groonga server host.
133
131
  3. Run `groonga -n /PATH/TO/MASTER/GROONGA/SERVER/DB <
134
132
  SLAVE_GROONGA_DUMP.grn` on master Groonga server.
@@ -141,7 +139,7 @@ You cannot update data until you finish to recover.
141
139
 
142
140
  Here are recover steps when slave Groonga server is down:
143
141
 
144
- 1. Run `groonga /PATH/TO/MASTER/GROONGA/SERVER/DB dump >
142
+ 1. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
145
143
  MASTER_GROONGA_DUMP.grn` on master Groonga server host.
146
144
  2. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
147
145
  MASTER_GROONGA_DUMP.grn` on slave Groonga server.
@@ -157,7 +155,7 @@ is full (see `buffer_queue_limit`) or fluentd gives up retrying (see
157
155
  Groonga server before those situations:
158
156
 
159
157
  1. Stop fluentd.
160
- 2. Run `groonga /PATH/TO/MASTER/GROONGA/SERVER/DB dump >
158
+ 2. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
161
159
  MASTER_GROONGA_DUMP.grn` on master Groonga server host.
162
160
  3. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
163
161
  MASTER_GROONGA_DUMP.grn` on slave Groonga server host.
@@ -196,21 +194,19 @@ Here is an example configuration file:
196
194
  # For master Groonga server
197
195
  <source>
198
196
  @type groonga
199
- protocol gqtp # Or use the below line
197
+ protocol gqtp # Or use the below line
200
198
  # protocol http
201
- bind 127.0.0.1 # For client side Fluentd
202
- # bind 192.168.0.1 # For master Groonga server side Fluentd
199
+ bind 127.0.0.1 # For client side Fluentd
200
+ # bind 192.168.0.1 # For master Groonga server side Fluentd
203
201
  port 10041
204
- real_host 192.168.29.1 # IP address of master Groonga server
205
- real_port 10041 # Port number of master Groonga server
206
- # real_port 20041 # Use different port number
207
- # for master Groonga server side fluentd
208
-
209
- command_name_position record # To keep command order
202
+ real_host 192.168.29.1 # IP address of master Groonga server
203
+ real_port 10041 # Port number of master Groonga server
204
+ # real_port 20041 # Use different port number
205
+ # for master Groonga server side fluentd
210
206
  </source>
211
207
 
212
208
  # For slave Groonga servers
213
- <match groonga.command.**>
209
+ <match groonga.command.*>
214
210
  @type copy
215
211
 
216
212
  # The first slave Groonga server
@@ -2,12 +2,18 @@
2
2
 
3
3
  # News
4
4
 
5
- ## 1.1.8: 2018-01-16
5
+ ## 1.2.0: 2017-04-26
6
6
 
7
7
  ### Improvements
8
8
 
9
- * in: `command_name_position`: Added a new parameter to control
10
- command format. The default behavior isn't changed.
9
+ * Supported recent fluentd v0.14 API.
10
+ Since fluentd 0.14.12, compatibility layer is unexpectedly broken,
11
+ fluent-plugin-groonga had been also affected because
12
+ fluent-plugin-groonga relied on it. Note that fluent-plugin-groonga
13
+ does not work with fluentd v0.12 because it does not use
14
+ compatibility layer anymore. We recommends to use latest release,
15
+ but if you still stay with fluentd v0.12, you need to use
16
+ fluent-plugin-groonga 1.1.7.
11
17
 
12
18
  ## 1.1.7: 2017-04-04
13
19
 
@@ -1,7 +1,6 @@
1
1
  # -*- mode: ruby; coding: utf-8 -*-
2
2
  #
3
- # Copyright (C) 2018 Yasuhiro Horimoto <horimoto@clear-code.com>
4
- # Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
3
+ # Copyright (C) 2012-2016 Kouhei Sutou <kou@clear-code.com>
5
4
  #
6
5
  # This library is free software; you can redistribute it and/or
7
6
  # modify it under the terms of the GNU Lesser General Public
@@ -18,7 +17,7 @@
18
17
 
19
18
  Gem::Specification.new do |spec|
20
19
  spec.name = "fluent-plugin-groonga"
21
- spec.version = "1.1.8"
20
+ spec.version = "1.2.0"
22
21
  spec.authors = ["Kouhei Sutou"]
23
22
  spec.email = ["kou@clear-code.com"]
24
23
  spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
@@ -35,7 +34,7 @@ Gem::Specification.new do |spec|
35
34
  spec.test_files += Dir.glob("test/**/*")
36
35
  spec.require_paths = ["lib"]
37
36
 
38
- spec.add_runtime_dependency("fluentd", ">= 0.12.20", "< 0.14.0")
37
+ spec.add_runtime_dependency("fluentd", ">= 0.14.0")
39
38
  spec.add_runtime_dependency("groonga-client", ">= 0.1.0")
40
39
  spec.add_runtime_dependency("groonga-command-parser")
41
40
 
@@ -1,4 +1,3 @@
1
- # Copyright (C) 2018 Yasuhiro Horimoto <horimoto@clear-code.com>
2
1
  # Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
3
2
  #
4
3
  # This library is free software; you can redistribute it and/or
@@ -23,460 +22,448 @@ require "http_parser"
23
22
  require "gqtp"
24
23
  require "groonga/command/parser"
25
24
 
26
- require "fluent/input"
27
- require "fluent/process"
25
+ require "fluent/plugin/input"
28
26
 
29
27
  module Fluent
30
- class GroongaInput < Input
31
- Plugin.register_input("groonga", self)
28
+ module Plugin
29
+ class GroongaInput < Input
30
+ Plugin.register_input("groonga", self)
32
31
 
33
- def initialize
34
- super
35
- end
36
-
37
- config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
38
- config_param :command_name_position, :enum, :list => [:tag, :record], :default => :tag
39
-
40
- def configure(conf)
41
- super
42
- case @protocol
43
- when :http
44
- @input = HTTPInput.new(self)
45
- when :gqtp
46
- @input = GQTPInput.new(self)
47
- end
48
- @input.configure(conf)
49
- end
50
-
51
- def start
52
- super
53
- @input.start
54
- end
55
-
56
- def shutdown
57
- super
58
- @input.shutdown
59
- end
60
-
61
- class Repeater < Coolio::TCPSocket
62
- def initialize(socket, handler)
63
- super(socket)
64
- @handler = handler
65
- end
66
-
67
- def on_read(data)
68
- @handler.write_back(data)
69
- end
70
-
71
- def on_close
72
- @handler.close
73
- end
74
- end
32
+ helpers :server
75
33
 
76
- class BaseInput
77
- include Configurable
78
- include DetachMultiProcessMixin
79
-
80
- config_param :bind, :string, :default => "0.0.0.0"
81
- config_param :port, :integer, :default => nil
82
- config_param :real_host, :string
83
- config_param :real_port, :integer, :default => nil
84
- DEFAULT_EMIT_COMMANDS = [
85
- "clearlock",
86
- "column_copy",
87
- "column_create",
88
- "column_remove",
89
- "column_rename",
90
- "config_delete",
91
- "config_set",
92
- "delete",
93
- "load",
94
- "lock_acquire",
95
- "lock_clear",
96
- "lock_release",
97
- "logical_table_remove",
98
- "object_remove",
99
- "plugin_register",
100
- "plugin_unregister",
101
- "register",
102
- "reindex",
103
- "table_copy",
104
- "table_create",
105
- "table_remove",
106
- "table_rename",
107
- "truncate",
108
- ]
109
- config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
110
- commands = value.split(/\s*,\s*/)
111
- commands.collect do |command|
112
- if /\A\/(.*)\/(i)?\z/ =~ command
113
- pattern = $1
114
- flag_mark = $2
115
- flag = 0
116
- flag |= Regexp::IGNORECASE if flag_mark == "i"
117
- Regexp.new(pattern, flag)
118
- else
119
- command
120
- end
121
- end
34
+ def initialize
35
+ super
122
36
  end
123
37
 
124
- def initialize(input_plugin)
125
- @input_plugin = input_plugin
126
- end
38
+ config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
127
39
 
128
40
  def configure(conf)
129
41
  super
130
-
131
- @port ||= default_port
132
- @real_port ||= default_port
42
+ case @protocol
43
+ when :http
44
+ @input = HTTPInput.new(self)
45
+ when :gqtp
46
+ @input = GQTPInput.new(self)
47
+ end
48
+ @input.configure(conf)
133
49
  end
134
50
 
135
51
  def start
136
- listen_socket = TCPServer.new(@bind, @port)
137
- detach_multi_process do
138
- @loop = Coolio::Loop.new
139
-
140
- @socket = Coolio::TCPServer.new(listen_socket, nil,
141
- handler_class, self)
142
- @loop.attach(@socket)
52
+ super
143
53
 
144
- @shutdown_notifier = Coolio::AsyncWatcher.new
145
- @loop.attach(@shutdown_notifier)
54
+ port = @input.port
55
+ bind = @input.bind
56
+ log.info("[input][groonga][connect] listening port",
57
+ :port => port, :bind => bind)
58
+ server_create_connection(:groonga_input,
59
+ port,
60
+ :proto => :tcp,
61
+ :shared => system_config.workers > 1,
62
+ :bind => bind) do |connection|
63
+ handler = nil
64
+ real_host = @input.real_host
65
+ real_port = @input.real_port
66
+ repeater = Coolio::TCPSocket.connect(real_host, real_port)
67
+ repeater.on_connect_failed do
68
+ log.error("[input][groonga][connect][error] " +
69
+ "failed to connect to Groonga:",
70
+ :real_host => real_host,
71
+ :real_port => real_port)
72
+ connection.close
73
+ end
74
+ repeater.on_read do |data|
75
+ handler.write_back(data)
76
+ end
77
+ repeater.on_close do
78
+ handler.close
79
+ end
80
+ event_loop_attach(repeater)
146
81
 
147
- @thread = Thread.new do
148
- run
82
+ handler = @input.create_handler(connection, repeater)
83
+ connection.data do |data|
84
+ handler.on_read(data)
149
85
  end
150
86
  end
151
87
  end
152
88
 
153
89
  def shutdown
154
- @loop.stop
155
- @socket.close
156
- @shutdown_notifier.signal
157
- @thread.join
90
+ super
158
91
  end
159
92
 
160
- def create_repeater(client)
161
- repeater = Repeater.connect(@real_host, @real_port, client)
162
- repeater.attach(@loop)
163
- repeater
93
+ def multi_workers_ready?
94
+ true
164
95
  end
165
96
 
166
- def emit(command, params)
167
- normalized_command = command.split(".")[0]
168
- return unless emit_command?(normalized_command)
169
- case @input_plugin.command_name_position
170
- when :tag
171
- tag = "groonga.command.#{normalized_command}"
172
- record = params
173
- else
174
- tag = "groonga.command"
175
- record = {
176
- "name" => normalized_command,
177
- "arguments" => params
178
- }
97
+ class Repeater < Coolio::TCPSocket
98
+ def initialize(socket, handler)
99
+ super(socket)
100
+ @handler = handler
179
101
  end
180
- @input_plugin.router.emit(tag,
181
- Engine.now,
182
- record)
183
- end
184
102
 
185
- private
186
- def run
187
- @loop.run
188
- rescue
189
- $log.error("[input][groonga][error] unexpected error",
190
- :error => "#{$!.class}: #{$!}")
191
- $log.error_backtrace
192
- end
193
-
194
- def emit_command?(command)
195
- return true if @emit_commands.empty?
196
- @emit_commands.any? do |pattern|
197
- pattern === command
103
+ def on_read(data)
104
+ @handler.write_back(data)
198
105
  end
199
- end
200
- end
201
106
 
202
- class HTTPInput < BaseInput
203
- private
204
- def default_port
205
- 10041
206
- end
207
-
208
- def handler_class
209
- Handler
107
+ def on_close
108
+ @handler.close
109
+ end
210
110
  end
211
111
 
212
- class Handler < Coolio::Socket
213
- def initialize(socket, input)
214
- super(socket)
215
- @input = input
112
+ class BaseInput
113
+ include Configurable
114
+
115
+ config_param :bind, :string, :default => "0.0.0.0"
116
+ config_param :port, :integer, :default => nil
117
+ config_param :real_host, :string
118
+ config_param :real_port, :integer, :default => nil
119
+ DEFAULT_EMIT_COMMANDS = [
120
+ "clearlock",
121
+ "column_copy",
122
+ "column_create",
123
+ "column_remove",
124
+ "column_rename",
125
+ "config_delete",
126
+ "config_set",
127
+ "delete",
128
+ "load",
129
+ "lock_acquire",
130
+ "lock_clear",
131
+ "lock_release",
132
+ "logical_table_remove",
133
+ "object_remove",
134
+ "plugin_register",
135
+ "plugin_unregister",
136
+ "register",
137
+ "reindex",
138
+ "table_copy",
139
+ "table_create",
140
+ "table_remove",
141
+ "table_rename",
142
+ "truncate",
143
+ ]
144
+ config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
145
+ commands = value.split(/\s*,\s*/)
146
+ commands.collect do |command|
147
+ if /\A\/(.*)\/(i)?\z/ =~ command
148
+ pattern = $1
149
+ flag_mark = $2
150
+ flag = 0
151
+ flag |= Regexp::IGNORECASE if flag_mark == "i"
152
+ Regexp.new(pattern, flag)
153
+ else
154
+ command
155
+ end
156
+ end
216
157
  end
217
158
 
218
- def on_connect
219
- @repeater = @input.create_repeater(self)
220
- @repeater.on_connect_failed do
221
- $log.error("[input][groonga][connect][error] " +
222
- "failed to connect to Groonga:",
223
- :real_host => @input.real_host,
224
- :real_port => @input.real_port)
225
- close
226
- end
227
- @request_handler = RequestHandler.new(@input, @repeater)
228
- @response_handler = ResponseHandler.new(self)
159
+ def initialize(input_plugin)
160
+ @input_plugin = input_plugin
229
161
  end
230
162
 
231
- def on_read(data)
232
- begin
233
- @request_handler << data
234
- rescue HTTP::Parser::Error, URI::InvalidURIError
235
- $log.error("[input][groonga][request][error] " +
236
- "failed to parse HTTP request:",
237
- :error => "#{$!.class}: #{$!}")
238
- $log.error_backtrace
239
- reply_error_response("400 Bad Request")
240
- rescue
241
- $log.error("[input][groonga][request][error] " +
242
- "failed to handle HTTP request:",
243
- :error => "#{$!.class}: #{$!}")
244
- $log.error_backtrace
245
- reply_error_response("500 Internal Server Error")
246
- end
247
- end
163
+ def configure(conf)
164
+ super
248
165
 
249
- def write_back(data)
250
- begin
251
- @response_handler << data
252
- rescue
253
- $log.error("[input][groonga][response][error] " +
254
- "failed to handle HTTP response from Groonga:",
255
- :error => "#{$!.class}: #{$!}")
256
- $log.error_backtrace
257
- reply_error_response("500 Internal Server Error")
258
- return
259
- end
260
- write(data)
166
+ @port ||= default_port
167
+ @real_port ||= default_port
261
168
  end
262
169
 
263
- def on_response_complete(response)
264
- if need_emit?(response)
265
- @input.emit(@request_handler.command,
266
- @request_handler.params)
267
- end
268
- on_write_complete do
269
- @repeater.close
270
- end
170
+ def emit(command, params)
171
+ normalized_command = command.split(".")[0]
172
+ return unless emit_command?(normalized_command)
173
+ @input_plugin.router.emit("groonga.command.#{normalized_command}",
174
+ Engine.now,
175
+ params)
271
176
  end
272
177
 
273
- private
274
- def need_emit?(response)
275
- case @request_handler.command
276
- when "load", "object_remove"
277
- return true
278
- end
279
-
280
- case response
281
- when Array
282
- return_code = response[0][0]
283
- return_code.zero?
284
- else
285
- false
286
- end
178
+ def log
179
+ @input_plugin.log
287
180
  end
288
181
 
289
- def reply_error_response(status)
290
- write("HTTP1.1 #{status}\r\n")
291
- write("Server: fluent-plugin-groonga\r\n")
292
- write("Connection: close\r\n")
293
- write("Content-Length: 0\r\n")
294
- write("\r\n")
295
- disable
296
- on_write_complete do
297
- @repeater.close
182
+ private
183
+ def emit_command?(command)
184
+ return true if @emit_commands.empty?
185
+ @emit_commands.any? do |pattern|
186
+ pattern === command
298
187
  end
299
188
  end
300
189
  end
301
190
 
302
- class RequestHandler
303
- attr_reader :command
304
- attr_reader :params
305
- def initialize(input, repeater)
306
- @input = input
307
- @repeater = repeater
308
- @parser = Http::Parser.new(self)
191
+ class HTTPInput < BaseInput
192
+ def create_handler(connection, repeater)
193
+ Handler.new(self, connection, repeater)
309
194
  end
310
195
 
311
- def <<(chunk)
312
- @parser << chunk
196
+ private
197
+ def default_port
198
+ 10041
313
199
  end
314
200
 
315
- def on_message_begin
316
- @body = ""
317
- @command = nil
318
- @params = nil
319
- end
201
+ class Handler
202
+ def initialize(input, connection, repeater)
203
+ @input = input
204
+ @connection = connection
205
+ @repeater = repeater
206
+ @request_handler = RequestHandler.new(@input, @repeater)
207
+ @response_handler = ResponseHandler.new(self, @input)
208
+ end
320
209
 
321
- def on_headers_complete(headers)
322
- method = @parser.http_method
323
- url = @parser.request_url
324
- http_version = @parser.http_version.join(".")
325
- @repeater.write("#{method} #{url} HTTP/#{http_version}\r\n")
326
- headers.each do |name, value|
327
- case name
328
- when /\AHost\z/i
329
- real_host = @input.real_host
330
- real_port = @input.real_port
331
- @repeater.write("#{name}: #{real_host}:#{real_port}\r\n")
332
- else
333
- @repeater.write("#{name}: #{value}\r\n")
210
+ def on_read(data)
211
+ begin
212
+ @request_handler << data
213
+ rescue HTTP::Parser::Error, URI::InvalidURIError
214
+ @input.log.error("[input][groonga][request][error] " +
215
+ "failed to parse HTTP request:",
216
+ :error => "#{$!.class}: #{$!}")
217
+ @input.log.error_backtrace
218
+ reply_error_response("400 Bad Request")
219
+ rescue
220
+ @input.log.error("[input][groonga][request][error] " +
221
+ "failed to handle HTTP request:",
222
+ :error => "#{$!.class}: #{$!}")
223
+ @input.log.error_backtrace
224
+ reply_error_response("500 Internal Server Error")
334
225
  end
335
226
  end
336
- @repeater.write("\r\n")
337
- end
338
227
 
339
- def on_body(chunk)
340
- @body << chunk
341
- @repeater.write(chunk)
342
- end
228
+ def write_back(data)
229
+ begin
230
+ @response_handler << data
231
+ rescue
232
+ @input.log.error("[input][groonga][response][error] " +
233
+ "failed to handle HTTP response from Groonga:",
234
+ :error => "#{$!.class}: #{$!}")
235
+ @input.log.error_backtrace
236
+ reply_error_response("500 Internal Server Error")
237
+ return
238
+ end
239
+ @connection.write(data)
240
+ end
343
241
 
344
- def on_message_complete
345
- uri = URI.parse(@parser.request_url)
346
- params = WEBrick::HTTPUtils.parse_query(uri.query)
347
- path_info = uri.path
348
- case path_info
349
- when /\A\/d\//
350
- command = $POSTMATCH
351
- if command == "load"
352
- params["values"] = @body unless @body.empty?
242
+ def on_response_complete(response)
243
+ if need_emit?(response)
244
+ @input.emit(@request_handler.command,
245
+ @request_handler.params)
246
+ end
247
+ @connection.on(:write_complete) do
248
+ @repeater.close
353
249
  end
354
- @command = command
355
- @params = params
356
250
  end
357
- end
358
- end
359
251
 
360
- class ResponseHandler
361
- def initialize(handler)
362
- @handler = handler
363
- @parser = Http::Parser.new(self)
364
- end
252
+ def close
253
+ @connection.close
254
+ end
365
255
 
366
- def <<(chunk)
367
- @parser << chunk
368
- end
256
+ private
257
+ def need_emit?(response)
258
+ case @request_handler.command
259
+ when "load", "object_remove"
260
+ return true
261
+ end
369
262
 
370
- def on_message_begin
371
- @body = ""
372
- @content_type = nil
373
- end
263
+ case response
264
+ when Array
265
+ return_code = response[0][0]
266
+ return_code.zero?
267
+ else
268
+ false
269
+ end
270
+ end
374
271
 
375
- def on_headers_complete(headers)
376
- headers.each do |name, value|
377
- case name
378
- when /\AContent-Type\z/i
379
- @content_type = value
272
+ def reply_error_response(status)
273
+ @connection.write("HTTP1.1 #{status}\r\n")
274
+ @connection.write("Server: fluent-plugin-groonga\r\n")
275
+ @connection.write("Connection: close\r\n")
276
+ @connection.write("Content-Length: 0\r\n")
277
+ @connection.write("\r\n")
278
+ @connection.on(:write_complete) do
279
+ @repeater.close
380
280
  end
381
281
  end
382
282
  end
383
283
 
384
- def on_body(chunk)
385
- @body << chunk
386
- end
284
+ class RequestHandler
285
+ attr_reader :command
286
+ attr_reader :params
287
+ def initialize(input, repeater)
288
+ @input = input
289
+ @repeater = repeater
290
+ @parser = Http::Parser.new(self)
291
+ end
387
292
 
388
- def on_message_complete
389
- return if @parser.status_code == 100
293
+ def <<(chunk)
294
+ @parser << chunk
295
+ end
390
296
 
391
- response = nil
392
- case @content_type
393
- when /\Aapplication\/json\z/i
394
- begin
395
- response = JSON.parse(@body)
396
- rescue JSON::ParserError
397
- $log.warn("[input][groonga][response][warn] " +
398
- "failed to parse response JSON:",
399
- :error => "#{$!.class}: #{$!}",
400
- :json => @body)
297
+ def on_message_begin
298
+ @body = ""
299
+ @command = nil
300
+ @params = nil
301
+ end
302
+
303
+ def on_headers_complete(headers)
304
+ method = @parser.http_method
305
+ url = @parser.request_url
306
+ http_version = @parser.http_version.join(".")
307
+ @repeater.write("#{method} #{url} HTTP/#{http_version}\r\n")
308
+ headers.each do |name, value|
309
+ case name
310
+ when /\AHost\z/i
311
+ real_host = @input.real_host
312
+ real_port = @input.real_port
313
+ @repeater.write("#{name}: #{real_host}:#{real_port}\r\n")
314
+ else
315
+ @repeater.write("#{name}: #{value}\r\n")
316
+ end
401
317
  end
402
- when /\Aapplication\/x-msgpack\z/i
403
- begin
404
- response = MessagePack.unpack(@body)
405
- rescue MessagePack::UnpackError, EOFError
406
- $log.warn("[input][groonga][response][warn] " +
407
- "failed to parse response MessagePack",
408
- :error => "#{$!.class}: #{$!}",
409
- :msgpack => @body)
318
+ @repeater.write("\r\n")
319
+ end
320
+
321
+ def on_body(chunk)
322
+ @body << chunk
323
+ @repeater.write(chunk)
324
+ end
325
+
326
+ def on_message_complete
327
+ uri = URI.parse(@parser.request_url)
328
+ params = WEBrick::HTTPUtils.parse_query(uri.query)
329
+ path_info = uri.path
330
+ case path_info
331
+ when /\A\/d\//
332
+ command = $POSTMATCH
333
+ if command == "load"
334
+ params["values"] = @body unless @body.empty?
335
+ end
336
+ @command = command
337
+ @params = params
410
338
  end
411
- when /\Atext\/x-groonga-command-list\z/i
412
- response = @body
413
339
  end
414
- @handler.on_response_complete(response)
415
340
  end
416
- end
417
- end
418
341
 
419
- class GQTPInput < BaseInput
420
- private
421
- def default_port
422
- 10043
423
- end
342
+ class ResponseHandler
343
+ def initialize(handler, input)
344
+ @handler = handler
345
+ @input = input
346
+ @parser = Http::Parser.new(self)
347
+ end
424
348
 
425
- def handler_class
426
- Handler
427
- end
349
+ def <<(chunk)
350
+ @parser << chunk
351
+ end
428
352
 
429
- class Handler < Coolio::Socket
430
- def initialize(socket, input)
431
- super(socket)
432
- @input = input
433
- end
353
+ def on_message_begin
354
+ @body = ""
355
+ @content_type = nil
356
+ end
434
357
 
435
- def on_connect
436
- @parser = Parser.new(@input)
437
- @repeater = @input.create_repeater(self)
438
- end
358
+ def on_headers_complete(headers)
359
+ headers.each do |name, value|
360
+ case name
361
+ when /\AContent-Type\z/i
362
+ @content_type = value
363
+ end
364
+ end
365
+ end
439
366
 
440
- def on_read(data)
441
- @parser << data
442
- @repeater.write(data)
443
- end
367
+ def on_body(chunk)
368
+ @body << chunk
369
+ end
444
370
 
445
- def on_close
446
- @parser.close
371
+ def on_message_complete
372
+ return if @parser.status_code == 100
373
+
374
+ response = nil
375
+ case @content_type
376
+ when /\Aapplication\/json\z/i
377
+ begin
378
+ response = JSON.parse(@body)
379
+ rescue JSON::ParserError
380
+ @input.log.warn("[input][groonga][response][warn] " +
381
+ "failed to parse response JSON:",
382
+ :error => "#{$!.class}: #{$!}",
383
+ :json => @body)
384
+ end
385
+ when /\Aapplication\/x-msgpack\z/i
386
+ begin
387
+ response = MessagePack.unpack(@body)
388
+ rescue MessagePack::UnpackError, EOFError
389
+ @input.log.warn("[input][groonga][response][warn] " +
390
+ "failed to parse response MessagePack",
391
+ :error => "#{$!.class}: #{$!}",
392
+ :msgpack => @body)
393
+ end
394
+ when /\Atext\/x-groonga-command-list\z/i
395
+ response = @body
396
+ end
397
+ @handler.on_response_complete(response)
398
+ end
447
399
  end
448
400
  end
449
401
 
450
- class Parser < GQTP::Parser
451
- def initialize(input)
452
- super()
453
- @input = input
454
- initialize_command_parser
402
+ class GQTPInput < BaseInput
403
+ def create_handler(connection, repeater)
404
+ Handler.new(self, connection, repeater)
455
405
  end
456
406
 
457
- def on_body(chunk)
458
- @command_parser << chunk
407
+ private
408
+ def default_port
409
+ 10043
459
410
  end
460
411
 
461
- def on_complete
462
- @command_parser << "\n"
463
- end
412
+ class Handler
413
+ def initialize(input, connection, repeater)
414
+ @input = input
415
+ @connection = connection
416
+ @repeater = repeater
417
+
418
+ @request_parser = RequestParser.new(@input)
419
+ end
464
420
 
465
- def close
466
- @command_parser.finish
421
+ def on_read(data)
422
+ @request_parser << data
423
+ @repeater.write(data)
424
+ end
425
+
426
+ def write_back(data)
427
+ @connection.write(data)
428
+ end
429
+
430
+ def close
431
+ @request_parser.close
432
+ @connection.close
433
+ end
467
434
  end
468
435
 
469
- private
470
- def initialize_command_parser
471
- @command_parser = Groonga::Command::Parser.new
472
- @command_parser.on_command do |command|
473
- @input.emit(command.name, command.arguments)
474
- end
475
- @command_parser.on_load_value do |command, value|
476
- arguments = command.arguments.dup
477
- arguments[:columns] = command.columns.join(", ")
478
- arguments[:values] = Yajl::Encoder.encode([value])
479
- @input.emit(command.name, arguments)
436
+ class RequestParser < GQTP::Parser
437
+ def initialize(input)
438
+ super()
439
+ @input = input
440
+ initialize_command_parser
441
+ end
442
+
443
+ def on_body(chunk)
444
+ @command_parser << chunk
445
+ end
446
+
447
+ def on_complete
448
+ @command_parser << "\n"
449
+ end
450
+
451
+ def close
452
+ @command_parser.finish
453
+ end
454
+
455
+ private
456
+ def initialize_command_parser
457
+ @command_parser = Groonga::Command::Parser.new
458
+ @command_parser.on_command do |command|
459
+ @input.emit(command.command_name, command.arguments)
460
+ end
461
+ @command_parser.on_load_value do |command, value|
462
+ arguments = command.arguments.dup
463
+ arguments[:columns] = command.columns.join(", ")
464
+ arguments[:values] = Yajl::Encoder.encode([value])
465
+ @input.emit(command.command_name, arguments)
466
+ end
480
467
  end
481
468
  end
482
469
  end