fluent-plugin-groonga 1.1.8 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: f225a28b0bd052b37c5180b7707ac898f550ca57
4
- data.tar.gz: 0c93017fa42a9a2edac9b50d97d0e3d3e5cf0a94
3
+ metadata.gz: 58980511ad7775baea84617dc9ab6ed0f93e29dc
4
+ data.tar.gz: 63193a59c766554172b3d34a4d3f40c47465d594
5
5
  SHA512:
6
- metadata.gz: deb8587173008cf6ddc636aebe1fd7c2eb41ed01b806e91c020af81c1dc38eeccdbac196391ecc2289354458d3172707db85f9d9865a7690bc18f8ff00e917df
7
- data.tar.gz: 737f5f9db324b14199f4e2b1734d7bc76e1008b80906eba78febe99f0aae91a428a1db69d809715661473bbc4e4252557be4f75512796f42df6f178195eb89ef
6
+ metadata.gz: c44c1ad1b43b1144ff703d61092a8ccf76735c5dfdfce75f5f33023b7e1969b78154629205d68b52d0b6ff9889f0b0a2a254a1e225777ded1cfcc23529293a71
7
+ data.tar.gz: 7a1b9fdef294db2a6ab661ad7d4619486a887e82e5e674f6e0d1243cb301fc7281a6df8c3463cad04bcab63cf359ff8bc9be6edcb773a61234243b551bd0c6a8
@@ -8,7 +8,25 @@ describes configuration parameters of them.
8
8
 
9
9
  ## The `groonga` input plugin
10
10
 
11
- Here are available parameters:
11
+ The behavior of `groonga` input plugin is customized in `system`
12
+ directive and `source` directive parameters.
13
+
14
+ Here is recommended parameter in `system` directive:
15
+
16
+ * `workers`: It specifies the number of workers. The point of view
17
+ in performance, the recommended value is greater than 1 to process
18
+ requests in parallel.
19
+
20
+ * recommended value: greater than `1`.
21
+
22
+ * default: `1`
23
+
24
+ Note that there is one exception about the number of recommended
25
+ workers. Users should use `1` worker when commands contain DDL such as
26
+ `table_create` and `column_create`. Because execution order of these
27
+ commands may be swapped. In such a case, replication will fail.
28
+
29
+ Here are available parameters in `source` directive:
12
30
 
13
31
  * `protocol`: It specifies protocol for receiving Groonga commands.
14
32
 
@@ -32,28 +50,14 @@ Here are available parameters:
32
50
 
33
51
  * default: `10041`
34
52
 
35
- * `command_name_position`: It specifies where Groonga command's
36
- name.
37
-
38
- If `tag` is specified, the plugin puts Groonga command's name to
39
- tag as `groonga.command.#{COMMAND_NAME}` format and record
40
- is the arguments of the command.
41
-
42
- If `record` is specified, the plugin puts both Groonga command's
43
- name and arguments to record as `{"name": "#{COMMAND_NAME}",
44
- "arguments": {...}}` format. Tag is always `groonga.command`.
45
-
46
- `record` is suitable when you want to implement replication. If
47
- you `tag` for replication, Groonga command's order may be changed.
48
-
49
- * Available values: `tag`, `record`
50
-
51
- * default: `tag`
52
-
53
53
  * `emit_commands`: TODO
54
54
 
55
55
  Here is an example:
56
56
 
57
+ <system>
58
+ workers 2
59
+ </system>
60
+
57
61
  <source>
58
62
  @type groonga
59
63
  protocol http
@@ -61,7 +65,6 @@ Here is an example:
61
65
  port 10041
62
66
  real_host 192.168.0.1
63
67
  real_port 10041
64
- command_name_position record
65
68
  </source>
66
69
 
67
70
  ## The `groonga` output plugin
@@ -66,21 +66,19 @@ Here is an example configuration file:
66
66
  # For master Groonga server
67
67
  <source>
68
68
  @type groonga
69
- protocol gqtp # Or use the below line
69
+ protocol gqtp # Or use the below line
70
70
  # protocol http
71
- bind 127.0.0.1 # For client side Fluentd
72
- # bind 192.168.0.1 # For master Groonga server side Fluentd
71
+ bind 127.0.0.1 # For client side Fluentd
72
+ # bind 192.168.0.1 # For master Groonga server side Fluentd
73
73
  port 10041
74
- real_host 192.168.29.1 # IP address of master Groonga server
75
- real_port 10041 # Port number of master Groonga server
76
- # real_port 20041 # Use different port number
77
- # for master Groonga server side Fluentd
78
-
79
- command_name_position record # To keep command order
74
+ real_host 192.168.29.1 # IP address of master Groonga server
75
+ real_port 10041 # Port number of master Groonga server
76
+ # real_port 20041 # Use different port number
77
+ # for master Groonga server side Fluentd
80
78
  </source>
81
79
 
82
80
  # For slave Groonga server
83
- <match groonga.command.**>
81
+ <match groonga.command.*>
84
82
  @type groonga
85
83
  protocol gqtp # Or use the below line
86
84
  # protocol http # You can use different protocol for
@@ -128,7 +126,7 @@ You cannot update data until fluentd is up.
128
126
  Here are recover steps when master Groonga server is down:
129
127
 
130
128
  1. Stop fluentd.
131
- 2. Run `groonga /PATH/TO/SLAVE/GROONGA/SERVER/DB dump >
129
+ 2. Run `grndump /PATH/TO/SLAVE/GROONGA/SERVER/DB >
132
130
  SLAVE_GROONGA_DUMP.grn` on slave Groonga server host.
133
131
  3. Run `groonga -n /PATH/TO/MASTER/GROONGA/SERVER/DB <
134
132
  SLAVE_GROONGA_DUMP.grn` on master Groonga server.
@@ -141,7 +139,7 @@ You cannot update data until you finish to recover.
141
139
 
142
140
  Here are recover steps when slave Groonga server is down:
143
141
 
144
- 1. Run `groonga /PATH/TO/MASTER/GROONGA/SERVER/DB dump >
142
+ 1. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
145
143
  MASTER_GROONGA_DUMP.grn` on master Groonga server host.
146
144
  2. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
147
145
  MASTER_GROONGA_DUMP.grn` on slave Groonga server.
@@ -157,7 +155,7 @@ is full (see `buffer_queue_limit`) or fluentd gives up retrying (see
157
155
  Groonga server before those situations:
158
156
 
159
157
  1. Stop fluentd.
160
- 2. Run `groonga /PATH/TO/MASTER/GROONGA/SERVER/DB dump >
158
+ 2. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
161
159
  MASTER_GROONGA_DUMP.grn` on master Groonga server host.
162
160
  3. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
163
161
  MASTER_GROONGA_DUMP.grn` on slave Groonga server host.
@@ -196,21 +194,19 @@ Here is an example configuration file:
196
194
  # For master Groonga server
197
195
  <source>
198
196
  @type groonga
199
- protocol gqtp # Or use the below line
197
+ protocol gqtp # Or use the below line
200
198
  # protocol http
201
- bind 127.0.0.1 # For client side Fluentd
202
- # bind 192.168.0.1 # For master Groonga server side Fluentd
199
+ bind 127.0.0.1 # For client side Fluentd
200
+ # bind 192.168.0.1 # For master Groonga server side Fluentd
203
201
  port 10041
204
- real_host 192.168.29.1 # IP address of master Groonga server
205
- real_port 10041 # Port number of master Groonga server
206
- # real_port 20041 # Use different port number
207
- # for master Groonga server side fluentd
208
-
209
- command_name_position record # To keep command order
202
+ real_host 192.168.29.1 # IP address of master Groonga server
203
+ real_port 10041 # Port number of master Groonga server
204
+ # real_port 20041 # Use different port number
205
+ # for master Groonga server side fluentd
210
206
  </source>
211
207
 
212
208
  # For slave Groonga servers
213
- <match groonga.command.**>
209
+ <match groonga.command.*>
214
210
  @type copy
215
211
 
216
212
  # The first slave Groonga server
@@ -2,12 +2,18 @@
2
2
 
3
3
  # News
4
4
 
5
- ## 1.1.8: 2018-01-16
5
+ ## 1.2.0: 2017-04-26
6
6
 
7
7
  ### Improvements
8
8
 
9
- * in: `command_name_position`: Added a new parameter to control
10
- command format. The default behavior isn't changed.
9
+ * Supported recent fluentd v0.14 API.
10
+ Since fluentd 0.14.12, compatibility layer is unexpectedly broken,
11
+ fluent-plugin-groonga had been also affected because
12
+ fluent-plugin-groonga relied on it. Note that fluent-plugin-groonga
13
+ does not work with fluentd v0.12 because it does not use
14
+ compatibility layer anymore. We recommends to use latest release,
15
+ but if you still stay with fluentd v0.12, you need to use
16
+ fluent-plugin-groonga 1.1.7.
11
17
 
12
18
  ## 1.1.7: 2017-04-04
13
19
 
@@ -1,7 +1,6 @@
1
1
  # -*- mode: ruby; coding: utf-8 -*-
2
2
  #
3
- # Copyright (C) 2018 Yasuhiro Horimoto <horimoto@clear-code.com>
4
- # Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
3
+ # Copyright (C) 2012-2016 Kouhei Sutou <kou@clear-code.com>
5
4
  #
6
5
  # This library is free software; you can redistribute it and/or
7
6
  # modify it under the terms of the GNU Lesser General Public
@@ -18,7 +17,7 @@
18
17
 
19
18
  Gem::Specification.new do |spec|
20
19
  spec.name = "fluent-plugin-groonga"
21
- spec.version = "1.1.8"
20
+ spec.version = "1.2.0"
22
21
  spec.authors = ["Kouhei Sutou"]
23
22
  spec.email = ["kou@clear-code.com"]
24
23
  spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
@@ -35,7 +34,7 @@ Gem::Specification.new do |spec|
35
34
  spec.test_files += Dir.glob("test/**/*")
36
35
  spec.require_paths = ["lib"]
37
36
 
38
- spec.add_runtime_dependency("fluentd", ">= 0.12.20", "< 0.14.0")
37
+ spec.add_runtime_dependency("fluentd", ">= 0.14.0")
39
38
  spec.add_runtime_dependency("groonga-client", ">= 0.1.0")
40
39
  spec.add_runtime_dependency("groonga-command-parser")
41
40
 
@@ -1,4 +1,3 @@
1
- # Copyright (C) 2018 Yasuhiro Horimoto <horimoto@clear-code.com>
2
1
  # Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
3
2
  #
4
3
  # This library is free software; you can redistribute it and/or
@@ -23,460 +22,448 @@ require "http_parser"
23
22
  require "gqtp"
24
23
  require "groonga/command/parser"
25
24
 
26
- require "fluent/input"
27
- require "fluent/process"
25
+ require "fluent/plugin/input"
28
26
 
29
27
  module Fluent
30
- class GroongaInput < Input
31
- Plugin.register_input("groonga", self)
28
+ module Plugin
29
+ class GroongaInput < Input
30
+ Plugin.register_input("groonga", self)
32
31
 
33
- def initialize
34
- super
35
- end
36
-
37
- config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
38
- config_param :command_name_position, :enum, :list => [:tag, :record], :default => :tag
39
-
40
- def configure(conf)
41
- super
42
- case @protocol
43
- when :http
44
- @input = HTTPInput.new(self)
45
- when :gqtp
46
- @input = GQTPInput.new(self)
47
- end
48
- @input.configure(conf)
49
- end
50
-
51
- def start
52
- super
53
- @input.start
54
- end
55
-
56
- def shutdown
57
- super
58
- @input.shutdown
59
- end
60
-
61
- class Repeater < Coolio::TCPSocket
62
- def initialize(socket, handler)
63
- super(socket)
64
- @handler = handler
65
- end
66
-
67
- def on_read(data)
68
- @handler.write_back(data)
69
- end
70
-
71
- def on_close
72
- @handler.close
73
- end
74
- end
32
+ helpers :server
75
33
 
76
- class BaseInput
77
- include Configurable
78
- include DetachMultiProcessMixin
79
-
80
- config_param :bind, :string, :default => "0.0.0.0"
81
- config_param :port, :integer, :default => nil
82
- config_param :real_host, :string
83
- config_param :real_port, :integer, :default => nil
84
- DEFAULT_EMIT_COMMANDS = [
85
- "clearlock",
86
- "column_copy",
87
- "column_create",
88
- "column_remove",
89
- "column_rename",
90
- "config_delete",
91
- "config_set",
92
- "delete",
93
- "load",
94
- "lock_acquire",
95
- "lock_clear",
96
- "lock_release",
97
- "logical_table_remove",
98
- "object_remove",
99
- "plugin_register",
100
- "plugin_unregister",
101
- "register",
102
- "reindex",
103
- "table_copy",
104
- "table_create",
105
- "table_remove",
106
- "table_rename",
107
- "truncate",
108
- ]
109
- config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
110
- commands = value.split(/\s*,\s*/)
111
- commands.collect do |command|
112
- if /\A\/(.*)\/(i)?\z/ =~ command
113
- pattern = $1
114
- flag_mark = $2
115
- flag = 0
116
- flag |= Regexp::IGNORECASE if flag_mark == "i"
117
- Regexp.new(pattern, flag)
118
- else
119
- command
120
- end
121
- end
34
+ def initialize
35
+ super
122
36
  end
123
37
 
124
- def initialize(input_plugin)
125
- @input_plugin = input_plugin
126
- end
38
+ config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
127
39
 
128
40
  def configure(conf)
129
41
  super
130
-
131
- @port ||= default_port
132
- @real_port ||= default_port
42
+ case @protocol
43
+ when :http
44
+ @input = HTTPInput.new(self)
45
+ when :gqtp
46
+ @input = GQTPInput.new(self)
47
+ end
48
+ @input.configure(conf)
133
49
  end
134
50
 
135
51
  def start
136
- listen_socket = TCPServer.new(@bind, @port)
137
- detach_multi_process do
138
- @loop = Coolio::Loop.new
139
-
140
- @socket = Coolio::TCPServer.new(listen_socket, nil,
141
- handler_class, self)
142
- @loop.attach(@socket)
52
+ super
143
53
 
144
- @shutdown_notifier = Coolio::AsyncWatcher.new
145
- @loop.attach(@shutdown_notifier)
54
+ port = @input.port
55
+ bind = @input.bind
56
+ log.info("[input][groonga][connect] listening port",
57
+ :port => port, :bind => bind)
58
+ server_create_connection(:groonga_input,
59
+ port,
60
+ :proto => :tcp,
61
+ :shared => system_config.workers > 1,
62
+ :bind => bind) do |connection|
63
+ handler = nil
64
+ real_host = @input.real_host
65
+ real_port = @input.real_port
66
+ repeater = Coolio::TCPSocket.connect(real_host, real_port)
67
+ repeater.on_connect_failed do
68
+ log.error("[input][groonga][connect][error] " +
69
+ "failed to connect to Groonga:",
70
+ :real_host => real_host,
71
+ :real_port => real_port)
72
+ connection.close
73
+ end
74
+ repeater.on_read do |data|
75
+ handler.write_back(data)
76
+ end
77
+ repeater.on_close do
78
+ handler.close
79
+ end
80
+ event_loop_attach(repeater)
146
81
 
147
- @thread = Thread.new do
148
- run
82
+ handler = @input.create_handler(connection, repeater)
83
+ connection.data do |data|
84
+ handler.on_read(data)
149
85
  end
150
86
  end
151
87
  end
152
88
 
153
89
  def shutdown
154
- @loop.stop
155
- @socket.close
156
- @shutdown_notifier.signal
157
- @thread.join
90
+ super
158
91
  end
159
92
 
160
- def create_repeater(client)
161
- repeater = Repeater.connect(@real_host, @real_port, client)
162
- repeater.attach(@loop)
163
- repeater
93
+ def multi_workers_ready?
94
+ true
164
95
  end
165
96
 
166
- def emit(command, params)
167
- normalized_command = command.split(".")[0]
168
- return unless emit_command?(normalized_command)
169
- case @input_plugin.command_name_position
170
- when :tag
171
- tag = "groonga.command.#{normalized_command}"
172
- record = params
173
- else
174
- tag = "groonga.command"
175
- record = {
176
- "name" => normalized_command,
177
- "arguments" => params
178
- }
97
+ class Repeater < Coolio::TCPSocket
98
+ def initialize(socket, handler)
99
+ super(socket)
100
+ @handler = handler
179
101
  end
180
- @input_plugin.router.emit(tag,
181
- Engine.now,
182
- record)
183
- end
184
102
 
185
- private
186
- def run
187
- @loop.run
188
- rescue
189
- $log.error("[input][groonga][error] unexpected error",
190
- :error => "#{$!.class}: #{$!}")
191
- $log.error_backtrace
192
- end
193
-
194
- def emit_command?(command)
195
- return true if @emit_commands.empty?
196
- @emit_commands.any? do |pattern|
197
- pattern === command
103
+ def on_read(data)
104
+ @handler.write_back(data)
198
105
  end
199
- end
200
- end
201
106
 
202
- class HTTPInput < BaseInput
203
- private
204
- def default_port
205
- 10041
206
- end
207
-
208
- def handler_class
209
- Handler
107
+ def on_close
108
+ @handler.close
109
+ end
210
110
  end
211
111
 
212
- class Handler < Coolio::Socket
213
- def initialize(socket, input)
214
- super(socket)
215
- @input = input
112
+ class BaseInput
113
+ include Configurable
114
+
115
+ config_param :bind, :string, :default => "0.0.0.0"
116
+ config_param :port, :integer, :default => nil
117
+ config_param :real_host, :string
118
+ config_param :real_port, :integer, :default => nil
119
+ DEFAULT_EMIT_COMMANDS = [
120
+ "clearlock",
121
+ "column_copy",
122
+ "column_create",
123
+ "column_remove",
124
+ "column_rename",
125
+ "config_delete",
126
+ "config_set",
127
+ "delete",
128
+ "load",
129
+ "lock_acquire",
130
+ "lock_clear",
131
+ "lock_release",
132
+ "logical_table_remove",
133
+ "object_remove",
134
+ "plugin_register",
135
+ "plugin_unregister",
136
+ "register",
137
+ "reindex",
138
+ "table_copy",
139
+ "table_create",
140
+ "table_remove",
141
+ "table_rename",
142
+ "truncate",
143
+ ]
144
+ config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
145
+ commands = value.split(/\s*,\s*/)
146
+ commands.collect do |command|
147
+ if /\A\/(.*)\/(i)?\z/ =~ command
148
+ pattern = $1
149
+ flag_mark = $2
150
+ flag = 0
151
+ flag |= Regexp::IGNORECASE if flag_mark == "i"
152
+ Regexp.new(pattern, flag)
153
+ else
154
+ command
155
+ end
156
+ end
216
157
  end
217
158
 
218
- def on_connect
219
- @repeater = @input.create_repeater(self)
220
- @repeater.on_connect_failed do
221
- $log.error("[input][groonga][connect][error] " +
222
- "failed to connect to Groonga:",
223
- :real_host => @input.real_host,
224
- :real_port => @input.real_port)
225
- close
226
- end
227
- @request_handler = RequestHandler.new(@input, @repeater)
228
- @response_handler = ResponseHandler.new(self)
159
+ def initialize(input_plugin)
160
+ @input_plugin = input_plugin
229
161
  end
230
162
 
231
- def on_read(data)
232
- begin
233
- @request_handler << data
234
- rescue HTTP::Parser::Error, URI::InvalidURIError
235
- $log.error("[input][groonga][request][error] " +
236
- "failed to parse HTTP request:",
237
- :error => "#{$!.class}: #{$!}")
238
- $log.error_backtrace
239
- reply_error_response("400 Bad Request")
240
- rescue
241
- $log.error("[input][groonga][request][error] " +
242
- "failed to handle HTTP request:",
243
- :error => "#{$!.class}: #{$!}")
244
- $log.error_backtrace
245
- reply_error_response("500 Internal Server Error")
246
- end
247
- end
163
+ def configure(conf)
164
+ super
248
165
 
249
- def write_back(data)
250
- begin
251
- @response_handler << data
252
- rescue
253
- $log.error("[input][groonga][response][error] " +
254
- "failed to handle HTTP response from Groonga:",
255
- :error => "#{$!.class}: #{$!}")
256
- $log.error_backtrace
257
- reply_error_response("500 Internal Server Error")
258
- return
259
- end
260
- write(data)
166
+ @port ||= default_port
167
+ @real_port ||= default_port
261
168
  end
262
169
 
263
- def on_response_complete(response)
264
- if need_emit?(response)
265
- @input.emit(@request_handler.command,
266
- @request_handler.params)
267
- end
268
- on_write_complete do
269
- @repeater.close
270
- end
170
+ def emit(command, params)
171
+ normalized_command = command.split(".")[0]
172
+ return unless emit_command?(normalized_command)
173
+ @input_plugin.router.emit("groonga.command.#{normalized_command}",
174
+ Engine.now,
175
+ params)
271
176
  end
272
177
 
273
- private
274
- def need_emit?(response)
275
- case @request_handler.command
276
- when "load", "object_remove"
277
- return true
278
- end
279
-
280
- case response
281
- when Array
282
- return_code = response[0][0]
283
- return_code.zero?
284
- else
285
- false
286
- end
178
+ def log
179
+ @input_plugin.log
287
180
  end
288
181
 
289
- def reply_error_response(status)
290
- write("HTTP1.1 #{status}\r\n")
291
- write("Server: fluent-plugin-groonga\r\n")
292
- write("Connection: close\r\n")
293
- write("Content-Length: 0\r\n")
294
- write("\r\n")
295
- disable
296
- on_write_complete do
297
- @repeater.close
182
+ private
183
+ def emit_command?(command)
184
+ return true if @emit_commands.empty?
185
+ @emit_commands.any? do |pattern|
186
+ pattern === command
298
187
  end
299
188
  end
300
189
  end
301
190
 
302
- class RequestHandler
303
- attr_reader :command
304
- attr_reader :params
305
- def initialize(input, repeater)
306
- @input = input
307
- @repeater = repeater
308
- @parser = Http::Parser.new(self)
191
+ class HTTPInput < BaseInput
192
+ def create_handler(connection, repeater)
193
+ Handler.new(self, connection, repeater)
309
194
  end
310
195
 
311
- def <<(chunk)
312
- @parser << chunk
196
+ private
197
+ def default_port
198
+ 10041
313
199
  end
314
200
 
315
- def on_message_begin
316
- @body = ""
317
- @command = nil
318
- @params = nil
319
- end
201
+ class Handler
202
+ def initialize(input, connection, repeater)
203
+ @input = input
204
+ @connection = connection
205
+ @repeater = repeater
206
+ @request_handler = RequestHandler.new(@input, @repeater)
207
+ @response_handler = ResponseHandler.new(self, @input)
208
+ end
320
209
 
321
- def on_headers_complete(headers)
322
- method = @parser.http_method
323
- url = @parser.request_url
324
- http_version = @parser.http_version.join(".")
325
- @repeater.write("#{method} #{url} HTTP/#{http_version}\r\n")
326
- headers.each do |name, value|
327
- case name
328
- when /\AHost\z/i
329
- real_host = @input.real_host
330
- real_port = @input.real_port
331
- @repeater.write("#{name}: #{real_host}:#{real_port}\r\n")
332
- else
333
- @repeater.write("#{name}: #{value}\r\n")
210
+ def on_read(data)
211
+ begin
212
+ @request_handler << data
213
+ rescue HTTP::Parser::Error, URI::InvalidURIError
214
+ @input.log.error("[input][groonga][request][error] " +
215
+ "failed to parse HTTP request:",
216
+ :error => "#{$!.class}: #{$!}")
217
+ @input.log.error_backtrace
218
+ reply_error_response("400 Bad Request")
219
+ rescue
220
+ @input.log.error("[input][groonga][request][error] " +
221
+ "failed to handle HTTP request:",
222
+ :error => "#{$!.class}: #{$!}")
223
+ @input.log.error_backtrace
224
+ reply_error_response("500 Internal Server Error")
334
225
  end
335
226
  end
336
- @repeater.write("\r\n")
337
- end
338
227
 
339
- def on_body(chunk)
340
- @body << chunk
341
- @repeater.write(chunk)
342
- end
228
+ def write_back(data)
229
+ begin
230
+ @response_handler << data
231
+ rescue
232
+ @input.log.error("[input][groonga][response][error] " +
233
+ "failed to handle HTTP response from Groonga:",
234
+ :error => "#{$!.class}: #{$!}")
235
+ @input.log.error_backtrace
236
+ reply_error_response("500 Internal Server Error")
237
+ return
238
+ end
239
+ @connection.write(data)
240
+ end
343
241
 
344
- def on_message_complete
345
- uri = URI.parse(@parser.request_url)
346
- params = WEBrick::HTTPUtils.parse_query(uri.query)
347
- path_info = uri.path
348
- case path_info
349
- when /\A\/d\//
350
- command = $POSTMATCH
351
- if command == "load"
352
- params["values"] = @body unless @body.empty?
242
+ def on_response_complete(response)
243
+ if need_emit?(response)
244
+ @input.emit(@request_handler.command,
245
+ @request_handler.params)
246
+ end
247
+ @connection.on(:write_complete) do
248
+ @repeater.close
353
249
  end
354
- @command = command
355
- @params = params
356
250
  end
357
- end
358
- end
359
251
 
360
- class ResponseHandler
361
- def initialize(handler)
362
- @handler = handler
363
- @parser = Http::Parser.new(self)
364
- end
252
+ def close
253
+ @connection.close
254
+ end
365
255
 
366
- def <<(chunk)
367
- @parser << chunk
368
- end
256
+ private
257
+ def need_emit?(response)
258
+ case @request_handler.command
259
+ when "load", "object_remove"
260
+ return true
261
+ end
369
262
 
370
- def on_message_begin
371
- @body = ""
372
- @content_type = nil
373
- end
263
+ case response
264
+ when Array
265
+ return_code = response[0][0]
266
+ return_code.zero?
267
+ else
268
+ false
269
+ end
270
+ end
374
271
 
375
- def on_headers_complete(headers)
376
- headers.each do |name, value|
377
- case name
378
- when /\AContent-Type\z/i
379
- @content_type = value
272
+ def reply_error_response(status)
273
+ @connection.write("HTTP1.1 #{status}\r\n")
274
+ @connection.write("Server: fluent-plugin-groonga\r\n")
275
+ @connection.write("Connection: close\r\n")
276
+ @connection.write("Content-Length: 0\r\n")
277
+ @connection.write("\r\n")
278
+ @connection.on(:write_complete) do
279
+ @repeater.close
380
280
  end
381
281
  end
382
282
  end
383
283
 
384
- def on_body(chunk)
385
- @body << chunk
386
- end
284
+ class RequestHandler
285
+ attr_reader :command
286
+ attr_reader :params
287
+ def initialize(input, repeater)
288
+ @input = input
289
+ @repeater = repeater
290
+ @parser = Http::Parser.new(self)
291
+ end
387
292
 
388
- def on_message_complete
389
- return if @parser.status_code == 100
293
+ def <<(chunk)
294
+ @parser << chunk
295
+ end
390
296
 
391
- response = nil
392
- case @content_type
393
- when /\Aapplication\/json\z/i
394
- begin
395
- response = JSON.parse(@body)
396
- rescue JSON::ParserError
397
- $log.warn("[input][groonga][response][warn] " +
398
- "failed to parse response JSON:",
399
- :error => "#{$!.class}: #{$!}",
400
- :json => @body)
297
+ def on_message_begin
298
+ @body = ""
299
+ @command = nil
300
+ @params = nil
301
+ end
302
+
303
+ def on_headers_complete(headers)
304
+ method = @parser.http_method
305
+ url = @parser.request_url
306
+ http_version = @parser.http_version.join(".")
307
+ @repeater.write("#{method} #{url} HTTP/#{http_version}\r\n")
308
+ headers.each do |name, value|
309
+ case name
310
+ when /\AHost\z/i
311
+ real_host = @input.real_host
312
+ real_port = @input.real_port
313
+ @repeater.write("#{name}: #{real_host}:#{real_port}\r\n")
314
+ else
315
+ @repeater.write("#{name}: #{value}\r\n")
316
+ end
401
317
  end
402
- when /\Aapplication\/x-msgpack\z/i
403
- begin
404
- response = MessagePack.unpack(@body)
405
- rescue MessagePack::UnpackError, EOFError
406
- $log.warn("[input][groonga][response][warn] " +
407
- "failed to parse response MessagePack",
408
- :error => "#{$!.class}: #{$!}",
409
- :msgpack => @body)
318
+ @repeater.write("\r\n")
319
+ end
320
+
321
+ def on_body(chunk)
322
+ @body << chunk
323
+ @repeater.write(chunk)
324
+ end
325
+
326
+ def on_message_complete
327
+ uri = URI.parse(@parser.request_url)
328
+ params = WEBrick::HTTPUtils.parse_query(uri.query)
329
+ path_info = uri.path
330
+ case path_info
331
+ when /\A\/d\//
332
+ command = $POSTMATCH
333
+ if command == "load"
334
+ params["values"] = @body unless @body.empty?
335
+ end
336
+ @command = command
337
+ @params = params
410
338
  end
411
- when /\Atext\/x-groonga-command-list\z/i
412
- response = @body
413
339
  end
414
- @handler.on_response_complete(response)
415
340
  end
416
- end
417
- end
418
341
 
419
- class GQTPInput < BaseInput
420
- private
421
- def default_port
422
- 10043
423
- end
342
+ class ResponseHandler
343
+ def initialize(handler, input)
344
+ @handler = handler
345
+ @input = input
346
+ @parser = Http::Parser.new(self)
347
+ end
424
348
 
425
- def handler_class
426
- Handler
427
- end
349
+ def <<(chunk)
350
+ @parser << chunk
351
+ end
428
352
 
429
- class Handler < Coolio::Socket
430
- def initialize(socket, input)
431
- super(socket)
432
- @input = input
433
- end
353
+ def on_message_begin
354
+ @body = ""
355
+ @content_type = nil
356
+ end
434
357
 
435
- def on_connect
436
- @parser = Parser.new(@input)
437
- @repeater = @input.create_repeater(self)
438
- end
358
+ def on_headers_complete(headers)
359
+ headers.each do |name, value|
360
+ case name
361
+ when /\AContent-Type\z/i
362
+ @content_type = value
363
+ end
364
+ end
365
+ end
439
366
 
440
- def on_read(data)
441
- @parser << data
442
- @repeater.write(data)
443
- end
367
+ def on_body(chunk)
368
+ @body << chunk
369
+ end
444
370
 
445
- def on_close
446
- @parser.close
371
+ def on_message_complete
372
+ return if @parser.status_code == 100
373
+
374
+ response = nil
375
+ case @content_type
376
+ when /\Aapplication\/json\z/i
377
+ begin
378
+ response = JSON.parse(@body)
379
+ rescue JSON::ParserError
380
+ @input.log.warn("[input][groonga][response][warn] " +
381
+ "failed to parse response JSON:",
382
+ :error => "#{$!.class}: #{$!}",
383
+ :json => @body)
384
+ end
385
+ when /\Aapplication\/x-msgpack\z/i
386
+ begin
387
+ response = MessagePack.unpack(@body)
388
+ rescue MessagePack::UnpackError, EOFError
389
+ @input.log.warn("[input][groonga][response][warn] " +
390
+ "failed to parse response MessagePack",
391
+ :error => "#{$!.class}: #{$!}",
392
+ :msgpack => @body)
393
+ end
394
+ when /\Atext\/x-groonga-command-list\z/i
395
+ response = @body
396
+ end
397
+ @handler.on_response_complete(response)
398
+ end
447
399
  end
448
400
  end
449
401
 
450
- class Parser < GQTP::Parser
451
- def initialize(input)
452
- super()
453
- @input = input
454
- initialize_command_parser
402
+ class GQTPInput < BaseInput
403
+ def create_handler(connection, repeater)
404
+ Handler.new(self, connection, repeater)
455
405
  end
456
406
 
457
- def on_body(chunk)
458
- @command_parser << chunk
407
+ private
408
+ def default_port
409
+ 10043
459
410
  end
460
411
 
461
- def on_complete
462
- @command_parser << "\n"
463
- end
412
+ class Handler
413
+ def initialize(input, connection, repeater)
414
+ @input = input
415
+ @connection = connection
416
+ @repeater = repeater
417
+
418
+ @request_parser = RequestParser.new(@input)
419
+ end
464
420
 
465
- def close
466
- @command_parser.finish
421
+ def on_read(data)
422
+ @request_parser << data
423
+ @repeater.write(data)
424
+ end
425
+
426
+ def write_back(data)
427
+ @connection.write(data)
428
+ end
429
+
430
+ def close
431
+ @request_parser.close
432
+ @connection.close
433
+ end
467
434
  end
468
435
 
469
- private
470
- def initialize_command_parser
471
- @command_parser = Groonga::Command::Parser.new
472
- @command_parser.on_command do |command|
473
- @input.emit(command.name, command.arguments)
474
- end
475
- @command_parser.on_load_value do |command, value|
476
- arguments = command.arguments.dup
477
- arguments[:columns] = command.columns.join(", ")
478
- arguments[:values] = Yajl::Encoder.encode([value])
479
- @input.emit(command.name, arguments)
436
+ class RequestParser < GQTP::Parser
437
+ def initialize(input)
438
+ super()
439
+ @input = input
440
+ initialize_command_parser
441
+ end
442
+
443
+ def on_body(chunk)
444
+ @command_parser << chunk
445
+ end
446
+
447
+ def on_complete
448
+ @command_parser << "\n"
449
+ end
450
+
451
+ def close
452
+ @command_parser.finish
453
+ end
454
+
455
+ private
456
+ def initialize_command_parser
457
+ @command_parser = Groonga::Command::Parser.new
458
+ @command_parser.on_command do |command|
459
+ @input.emit(command.command_name, command.arguments)
460
+ end
461
+ @command_parser.on_load_value do |command, value|
462
+ arguments = command.arguments.dup
463
+ arguments[:columns] = command.columns.join(", ")
464
+ arguments[:values] = Yajl::Encoder.encode([value])
465
+ @input.emit(command.command_name, arguments)
466
+ end
480
467
  end
481
468
  end
482
469
  end