fluent-plugin-groonga 1.1.8 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/doc/text/configuration.md +23 -20
- data/doc/text/constitution.md +19 -23
- data/doc/text/news.md +9 -3
- data/fluent-plugin-groonga.gemspec +3 -4
- data/lib/fluent/plugin/in_groonga.rb +357 -370
- data/lib/fluent/plugin/out_groonga.rb +599 -584
- data/sample/command.conf +5 -4
- data/sample/gqtp.conf +5 -4
- data/sample/http.conf +5 -4
- data/sample/store-apache.conf +5 -3
- data/sample/store-syslog.conf +5 -3
- data/sample/store.conf +5 -3
- data/test/output/test_table_definition.rb +1 -1
- data/test/output/test_table_index_definition.rb +2 -2
- data/test/output/test_type_guesser.rb +1 -1
- data/test/run-test.rb +4 -1
- data/test/test_input.rb +27 -75
- data/test/test_output.rb +27 -65
- metadata +6 -12
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 58980511ad7775baea84617dc9ab6ed0f93e29dc
|
4
|
+
data.tar.gz: 63193a59c766554172b3d34a4d3f40c47465d594
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c44c1ad1b43b1144ff703d61092a8ccf76735c5dfdfce75f5f33023b7e1969b78154629205d68b52d0b6ff9889f0b0a2a254a1e225777ded1cfcc23529293a71
|
7
|
+
data.tar.gz: 7a1b9fdef294db2a6ab661ad7d4619486a887e82e5e674f6e0d1243cb301fc7281a6df8c3463cad04bcab63cf359ff8bc9be6edcb773a61234243b551bd0c6a8
|
data/doc/text/configuration.md
CHANGED
@@ -8,7 +8,25 @@ describes configuration parameters of them.
|
|
8
8
|
|
9
9
|
## The `groonga` input plugin
|
10
10
|
|
11
|
-
|
11
|
+
The behavior of `groonga` input plugin is customized in `system`
|
12
|
+
directive and `source` directive parameters.
|
13
|
+
|
14
|
+
Here is recommended parameter in `system` directive:
|
15
|
+
|
16
|
+
* `workers`: It specifies the number of workers. The point of view
|
17
|
+
in performance, the recommended value is greater than 1 to process
|
18
|
+
requests in parallel.
|
19
|
+
|
20
|
+
* recommended value: greater than `1`.
|
21
|
+
|
22
|
+
* default: `1`
|
23
|
+
|
24
|
+
Note that there is one exception about the number of recommended
|
25
|
+
workers. Users should use `1` worker when commands contain DDL such as
|
26
|
+
`table_create` and `column_create`. Because execution order of these
|
27
|
+
commands may be swapped. In such a case, replication will fail.
|
28
|
+
|
29
|
+
Here are available parameters in `source` directive:
|
12
30
|
|
13
31
|
* `protocol`: It specifies protocol for receiving Groonga commands.
|
14
32
|
|
@@ -32,28 +50,14 @@ Here are available parameters:
|
|
32
50
|
|
33
51
|
* default: `10041`
|
34
52
|
|
35
|
-
* `command_name_position`: It specifies where Groonga command's
|
36
|
-
name.
|
37
|
-
|
38
|
-
If `tag` is specified, the plugin puts Groonga command's name to
|
39
|
-
tag as `groonga.command.#{COMMAND_NAME}` format and record
|
40
|
-
is the arguments of the command.
|
41
|
-
|
42
|
-
If `record` is specified, the plugin puts both Groonga command's
|
43
|
-
name and arguments to record as `{"name": "#{COMMAND_NAME}",
|
44
|
-
"arguments": {...}}` format. Tag is always `groonga.command`.
|
45
|
-
|
46
|
-
`record` is suitable when you want to implement replication. If
|
47
|
-
you `tag` for replication, Groonga command's order may be changed.
|
48
|
-
|
49
|
-
* Available values: `tag`, `record`
|
50
|
-
|
51
|
-
* default: `tag`
|
52
|
-
|
53
53
|
* `emit_commands`: TODO
|
54
54
|
|
55
55
|
Here is an example:
|
56
56
|
|
57
|
+
<system>
|
58
|
+
workers 2
|
59
|
+
</system>
|
60
|
+
|
57
61
|
<source>
|
58
62
|
@type groonga
|
59
63
|
protocol http
|
@@ -61,7 +65,6 @@ Here is an example:
|
|
61
65
|
port 10041
|
62
66
|
real_host 192.168.0.1
|
63
67
|
real_port 10041
|
64
|
-
command_name_position record
|
65
68
|
</source>
|
66
69
|
|
67
70
|
## The `groonga` output plugin
|
data/doc/text/constitution.md
CHANGED
@@ -66,21 +66,19 @@ Here is an example configuration file:
|
|
66
66
|
# For master Groonga server
|
67
67
|
<source>
|
68
68
|
@type groonga
|
69
|
-
protocol gqtp
|
69
|
+
protocol gqtp # Or use the below line
|
70
70
|
# protocol http
|
71
|
-
bind 127.0.0.1
|
72
|
-
# bind 192.168.0.1
|
71
|
+
bind 127.0.0.1 # For client side Fluentd
|
72
|
+
# bind 192.168.0.1 # For master Groonga server side Fluentd
|
73
73
|
port 10041
|
74
|
-
real_host 192.168.29.1
|
75
|
-
real_port 10041
|
76
|
-
# real_port 20041
|
77
|
-
|
78
|
-
|
79
|
-
command_name_position record # To keep command order
|
74
|
+
real_host 192.168.29.1 # IP address of master Groonga server
|
75
|
+
real_port 10041 # Port number of master Groonga server
|
76
|
+
# real_port 20041 # Use different port number
|
77
|
+
# for master Groonga server side Fluentd
|
80
78
|
</source>
|
81
79
|
|
82
80
|
# For slave Groonga server
|
83
|
-
<match groonga.command
|
81
|
+
<match groonga.command.*>
|
84
82
|
@type groonga
|
85
83
|
protocol gqtp # Or use the below line
|
86
84
|
# protocol http # You can use different protocol for
|
@@ -128,7 +126,7 @@ You cannot update data until fluentd is up.
|
|
128
126
|
Here are recover steps when master Groonga server is down:
|
129
127
|
|
130
128
|
1. Stop fluentd.
|
131
|
-
2. Run `
|
129
|
+
2. Run `grndump /PATH/TO/SLAVE/GROONGA/SERVER/DB >
|
132
130
|
SLAVE_GROONGA_DUMP.grn` on slave Groonga server host.
|
133
131
|
3. Run `groonga -n /PATH/TO/MASTER/GROONGA/SERVER/DB <
|
134
132
|
SLAVE_GROONGA_DUMP.grn` on master Groonga server.
|
@@ -141,7 +139,7 @@ You cannot update data until you finish to recover.
|
|
141
139
|
|
142
140
|
Here are recover steps when slave Groonga server is down:
|
143
141
|
|
144
|
-
1. Run `
|
142
|
+
1. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
|
145
143
|
MASTER_GROONGA_DUMP.grn` on master Groonga server host.
|
146
144
|
2. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
|
147
145
|
MASTER_GROONGA_DUMP.grn` on slave Groonga server.
|
@@ -157,7 +155,7 @@ is full (see `buffer_queue_limit`) or fluentd gives up retrying (see
|
|
157
155
|
Groonga server before those situations:
|
158
156
|
|
159
157
|
1. Stop fluentd.
|
160
|
-
2. Run `
|
158
|
+
2. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
|
161
159
|
MASTER_GROONGA_DUMP.grn` on master Groonga server host.
|
162
160
|
3. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
|
163
161
|
MASTER_GROONGA_DUMP.grn` on slave Groonga server host.
|
@@ -196,21 +194,19 @@ Here is an example configuration file:
|
|
196
194
|
# For master Groonga server
|
197
195
|
<source>
|
198
196
|
@type groonga
|
199
|
-
protocol gqtp
|
197
|
+
protocol gqtp # Or use the below line
|
200
198
|
# protocol http
|
201
|
-
bind 127.0.0.1
|
202
|
-
# bind 192.168.0.1
|
199
|
+
bind 127.0.0.1 # For client side Fluentd
|
200
|
+
# bind 192.168.0.1 # For master Groonga server side Fluentd
|
203
201
|
port 10041
|
204
|
-
real_host 192.168.29.1
|
205
|
-
real_port 10041
|
206
|
-
# real_port 20041
|
207
|
-
|
208
|
-
|
209
|
-
command_name_position record # To keep command order
|
202
|
+
real_host 192.168.29.1 # IP address of master Groonga server
|
203
|
+
real_port 10041 # Port number of master Groonga server
|
204
|
+
# real_port 20041 # Use different port number
|
205
|
+
# for master Groonga server side fluentd
|
210
206
|
</source>
|
211
207
|
|
212
208
|
# For slave Groonga servers
|
213
|
-
<match groonga.command
|
209
|
+
<match groonga.command.*>
|
214
210
|
@type copy
|
215
211
|
|
216
212
|
# The first slave Groonga server
|
data/doc/text/news.md
CHANGED
@@ -2,12 +2,18 @@
|
|
2
2
|
|
3
3
|
# News
|
4
4
|
|
5
|
-
## 1.
|
5
|
+
## 1.2.0: 2017-04-26
|
6
6
|
|
7
7
|
### Improvements
|
8
8
|
|
9
|
-
*
|
10
|
-
|
9
|
+
* Supported recent fluentd v0.14 API.
|
10
|
+
Since fluentd 0.14.12, compatibility layer is unexpectedly broken,
|
11
|
+
fluent-plugin-groonga had been also affected because
|
12
|
+
fluent-plugin-groonga relied on it. Note that fluent-plugin-groonga
|
13
|
+
does not work with fluentd v0.12 because it does not use
|
14
|
+
compatibility layer anymore. We recommends to use latest release,
|
15
|
+
but if you still stay with fluentd v0.12, you need to use
|
16
|
+
fluent-plugin-groonga 1.1.7.
|
11
17
|
|
12
18
|
## 1.1.7: 2017-04-04
|
13
19
|
|
@@ -1,7 +1,6 @@
|
|
1
1
|
# -*- mode: ruby; coding: utf-8 -*-
|
2
2
|
#
|
3
|
-
# Copyright (C)
|
4
|
-
# Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
|
3
|
+
# Copyright (C) 2012-2016 Kouhei Sutou <kou@clear-code.com>
|
5
4
|
#
|
6
5
|
# This library is free software; you can redistribute it and/or
|
7
6
|
# modify it under the terms of the GNU Lesser General Public
|
@@ -18,7 +17,7 @@
|
|
18
17
|
|
19
18
|
Gem::Specification.new do |spec|
|
20
19
|
spec.name = "fluent-plugin-groonga"
|
21
|
-
spec.version = "1.
|
20
|
+
spec.version = "1.2.0"
|
22
21
|
spec.authors = ["Kouhei Sutou"]
|
23
22
|
spec.email = ["kou@clear-code.com"]
|
24
23
|
spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
|
@@ -35,7 +34,7 @@ Gem::Specification.new do |spec|
|
|
35
34
|
spec.test_files += Dir.glob("test/**/*")
|
36
35
|
spec.require_paths = ["lib"]
|
37
36
|
|
38
|
-
spec.add_runtime_dependency("fluentd", ">= 0.
|
37
|
+
spec.add_runtime_dependency("fluentd", ">= 0.14.0")
|
39
38
|
spec.add_runtime_dependency("groonga-client", ">= 0.1.0")
|
40
39
|
spec.add_runtime_dependency("groonga-command-parser")
|
41
40
|
|
@@ -1,4 +1,3 @@
|
|
1
|
-
# Copyright (C) 2018 Yasuhiro Horimoto <horimoto@clear-code.com>
|
2
1
|
# Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
|
3
2
|
#
|
4
3
|
# This library is free software; you can redistribute it and/or
|
@@ -23,460 +22,448 @@ require "http_parser"
|
|
23
22
|
require "gqtp"
|
24
23
|
require "groonga/command/parser"
|
25
24
|
|
26
|
-
require "fluent/input"
|
27
|
-
require "fluent/process"
|
25
|
+
require "fluent/plugin/input"
|
28
26
|
|
29
27
|
module Fluent
|
30
|
-
|
31
|
-
|
28
|
+
module Plugin
|
29
|
+
class GroongaInput < Input
|
30
|
+
Plugin.register_input("groonga", self)
|
32
31
|
|
33
|
-
|
34
|
-
super
|
35
|
-
end
|
36
|
-
|
37
|
-
config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
|
38
|
-
config_param :command_name_position, :enum, :list => [:tag, :record], :default => :tag
|
39
|
-
|
40
|
-
def configure(conf)
|
41
|
-
super
|
42
|
-
case @protocol
|
43
|
-
when :http
|
44
|
-
@input = HTTPInput.new(self)
|
45
|
-
when :gqtp
|
46
|
-
@input = GQTPInput.new(self)
|
47
|
-
end
|
48
|
-
@input.configure(conf)
|
49
|
-
end
|
50
|
-
|
51
|
-
def start
|
52
|
-
super
|
53
|
-
@input.start
|
54
|
-
end
|
55
|
-
|
56
|
-
def shutdown
|
57
|
-
super
|
58
|
-
@input.shutdown
|
59
|
-
end
|
60
|
-
|
61
|
-
class Repeater < Coolio::TCPSocket
|
62
|
-
def initialize(socket, handler)
|
63
|
-
super(socket)
|
64
|
-
@handler = handler
|
65
|
-
end
|
66
|
-
|
67
|
-
def on_read(data)
|
68
|
-
@handler.write_back(data)
|
69
|
-
end
|
70
|
-
|
71
|
-
def on_close
|
72
|
-
@handler.close
|
73
|
-
end
|
74
|
-
end
|
32
|
+
helpers :server
|
75
33
|
|
76
|
-
|
77
|
-
|
78
|
-
include DetachMultiProcessMixin
|
79
|
-
|
80
|
-
config_param :bind, :string, :default => "0.0.0.0"
|
81
|
-
config_param :port, :integer, :default => nil
|
82
|
-
config_param :real_host, :string
|
83
|
-
config_param :real_port, :integer, :default => nil
|
84
|
-
DEFAULT_EMIT_COMMANDS = [
|
85
|
-
"clearlock",
|
86
|
-
"column_copy",
|
87
|
-
"column_create",
|
88
|
-
"column_remove",
|
89
|
-
"column_rename",
|
90
|
-
"config_delete",
|
91
|
-
"config_set",
|
92
|
-
"delete",
|
93
|
-
"load",
|
94
|
-
"lock_acquire",
|
95
|
-
"lock_clear",
|
96
|
-
"lock_release",
|
97
|
-
"logical_table_remove",
|
98
|
-
"object_remove",
|
99
|
-
"plugin_register",
|
100
|
-
"plugin_unregister",
|
101
|
-
"register",
|
102
|
-
"reindex",
|
103
|
-
"table_copy",
|
104
|
-
"table_create",
|
105
|
-
"table_remove",
|
106
|
-
"table_rename",
|
107
|
-
"truncate",
|
108
|
-
]
|
109
|
-
config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
|
110
|
-
commands = value.split(/\s*,\s*/)
|
111
|
-
commands.collect do |command|
|
112
|
-
if /\A\/(.*)\/(i)?\z/ =~ command
|
113
|
-
pattern = $1
|
114
|
-
flag_mark = $2
|
115
|
-
flag = 0
|
116
|
-
flag |= Regexp::IGNORECASE if flag_mark == "i"
|
117
|
-
Regexp.new(pattern, flag)
|
118
|
-
else
|
119
|
-
command
|
120
|
-
end
|
121
|
-
end
|
34
|
+
def initialize
|
35
|
+
super
|
122
36
|
end
|
123
37
|
|
124
|
-
|
125
|
-
@input_plugin = input_plugin
|
126
|
-
end
|
38
|
+
config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
|
127
39
|
|
128
40
|
def configure(conf)
|
129
41
|
super
|
130
|
-
|
131
|
-
|
132
|
-
|
42
|
+
case @protocol
|
43
|
+
when :http
|
44
|
+
@input = HTTPInput.new(self)
|
45
|
+
when :gqtp
|
46
|
+
@input = GQTPInput.new(self)
|
47
|
+
end
|
48
|
+
@input.configure(conf)
|
133
49
|
end
|
134
50
|
|
135
51
|
def start
|
136
|
-
|
137
|
-
detach_multi_process do
|
138
|
-
@loop = Coolio::Loop.new
|
139
|
-
|
140
|
-
@socket = Coolio::TCPServer.new(listen_socket, nil,
|
141
|
-
handler_class, self)
|
142
|
-
@loop.attach(@socket)
|
52
|
+
super
|
143
53
|
|
144
|
-
|
145
|
-
|
54
|
+
port = @input.port
|
55
|
+
bind = @input.bind
|
56
|
+
log.info("[input][groonga][connect] listening port",
|
57
|
+
:port => port, :bind => bind)
|
58
|
+
server_create_connection(:groonga_input,
|
59
|
+
port,
|
60
|
+
:proto => :tcp,
|
61
|
+
:shared => system_config.workers > 1,
|
62
|
+
:bind => bind) do |connection|
|
63
|
+
handler = nil
|
64
|
+
real_host = @input.real_host
|
65
|
+
real_port = @input.real_port
|
66
|
+
repeater = Coolio::TCPSocket.connect(real_host, real_port)
|
67
|
+
repeater.on_connect_failed do
|
68
|
+
log.error("[input][groonga][connect][error] " +
|
69
|
+
"failed to connect to Groonga:",
|
70
|
+
:real_host => real_host,
|
71
|
+
:real_port => real_port)
|
72
|
+
connection.close
|
73
|
+
end
|
74
|
+
repeater.on_read do |data|
|
75
|
+
handler.write_back(data)
|
76
|
+
end
|
77
|
+
repeater.on_close do
|
78
|
+
handler.close
|
79
|
+
end
|
80
|
+
event_loop_attach(repeater)
|
146
81
|
|
147
|
-
|
148
|
-
|
82
|
+
handler = @input.create_handler(connection, repeater)
|
83
|
+
connection.data do |data|
|
84
|
+
handler.on_read(data)
|
149
85
|
end
|
150
86
|
end
|
151
87
|
end
|
152
88
|
|
153
89
|
def shutdown
|
154
|
-
|
155
|
-
@socket.close
|
156
|
-
@shutdown_notifier.signal
|
157
|
-
@thread.join
|
90
|
+
super
|
158
91
|
end
|
159
92
|
|
160
|
-
def
|
161
|
-
|
162
|
-
repeater.attach(@loop)
|
163
|
-
repeater
|
93
|
+
def multi_workers_ready?
|
94
|
+
true
|
164
95
|
end
|
165
96
|
|
166
|
-
|
167
|
-
|
168
|
-
|
169
|
-
|
170
|
-
when :tag
|
171
|
-
tag = "groonga.command.#{normalized_command}"
|
172
|
-
record = params
|
173
|
-
else
|
174
|
-
tag = "groonga.command"
|
175
|
-
record = {
|
176
|
-
"name" => normalized_command,
|
177
|
-
"arguments" => params
|
178
|
-
}
|
97
|
+
class Repeater < Coolio::TCPSocket
|
98
|
+
def initialize(socket, handler)
|
99
|
+
super(socket)
|
100
|
+
@handler = handler
|
179
101
|
end
|
180
|
-
@input_plugin.router.emit(tag,
|
181
|
-
Engine.now,
|
182
|
-
record)
|
183
|
-
end
|
184
102
|
|
185
|
-
|
186
|
-
|
187
|
-
@loop.run
|
188
|
-
rescue
|
189
|
-
$log.error("[input][groonga][error] unexpected error",
|
190
|
-
:error => "#{$!.class}: #{$!}")
|
191
|
-
$log.error_backtrace
|
192
|
-
end
|
193
|
-
|
194
|
-
def emit_command?(command)
|
195
|
-
return true if @emit_commands.empty?
|
196
|
-
@emit_commands.any? do |pattern|
|
197
|
-
pattern === command
|
103
|
+
def on_read(data)
|
104
|
+
@handler.write_back(data)
|
198
105
|
end
|
199
|
-
end
|
200
|
-
end
|
201
106
|
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
10041
|
206
|
-
end
|
207
|
-
|
208
|
-
def handler_class
|
209
|
-
Handler
|
107
|
+
def on_close
|
108
|
+
@handler.close
|
109
|
+
end
|
210
110
|
end
|
211
111
|
|
212
|
-
class
|
213
|
-
|
214
|
-
|
215
|
-
|
112
|
+
class BaseInput
|
113
|
+
include Configurable
|
114
|
+
|
115
|
+
config_param :bind, :string, :default => "0.0.0.0"
|
116
|
+
config_param :port, :integer, :default => nil
|
117
|
+
config_param :real_host, :string
|
118
|
+
config_param :real_port, :integer, :default => nil
|
119
|
+
DEFAULT_EMIT_COMMANDS = [
|
120
|
+
"clearlock",
|
121
|
+
"column_copy",
|
122
|
+
"column_create",
|
123
|
+
"column_remove",
|
124
|
+
"column_rename",
|
125
|
+
"config_delete",
|
126
|
+
"config_set",
|
127
|
+
"delete",
|
128
|
+
"load",
|
129
|
+
"lock_acquire",
|
130
|
+
"lock_clear",
|
131
|
+
"lock_release",
|
132
|
+
"logical_table_remove",
|
133
|
+
"object_remove",
|
134
|
+
"plugin_register",
|
135
|
+
"plugin_unregister",
|
136
|
+
"register",
|
137
|
+
"reindex",
|
138
|
+
"table_copy",
|
139
|
+
"table_create",
|
140
|
+
"table_remove",
|
141
|
+
"table_rename",
|
142
|
+
"truncate",
|
143
|
+
]
|
144
|
+
config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
|
145
|
+
commands = value.split(/\s*,\s*/)
|
146
|
+
commands.collect do |command|
|
147
|
+
if /\A\/(.*)\/(i)?\z/ =~ command
|
148
|
+
pattern = $1
|
149
|
+
flag_mark = $2
|
150
|
+
flag = 0
|
151
|
+
flag |= Regexp::IGNORECASE if flag_mark == "i"
|
152
|
+
Regexp.new(pattern, flag)
|
153
|
+
else
|
154
|
+
command
|
155
|
+
end
|
156
|
+
end
|
216
157
|
end
|
217
158
|
|
218
|
-
def
|
219
|
-
@
|
220
|
-
@repeater.on_connect_failed do
|
221
|
-
$log.error("[input][groonga][connect][error] " +
|
222
|
-
"failed to connect to Groonga:",
|
223
|
-
:real_host => @input.real_host,
|
224
|
-
:real_port => @input.real_port)
|
225
|
-
close
|
226
|
-
end
|
227
|
-
@request_handler = RequestHandler.new(@input, @repeater)
|
228
|
-
@response_handler = ResponseHandler.new(self)
|
159
|
+
def initialize(input_plugin)
|
160
|
+
@input_plugin = input_plugin
|
229
161
|
end
|
230
162
|
|
231
|
-
def
|
232
|
-
|
233
|
-
@request_handler << data
|
234
|
-
rescue HTTP::Parser::Error, URI::InvalidURIError
|
235
|
-
$log.error("[input][groonga][request][error] " +
|
236
|
-
"failed to parse HTTP request:",
|
237
|
-
:error => "#{$!.class}: #{$!}")
|
238
|
-
$log.error_backtrace
|
239
|
-
reply_error_response("400 Bad Request")
|
240
|
-
rescue
|
241
|
-
$log.error("[input][groonga][request][error] " +
|
242
|
-
"failed to handle HTTP request:",
|
243
|
-
:error => "#{$!.class}: #{$!}")
|
244
|
-
$log.error_backtrace
|
245
|
-
reply_error_response("500 Internal Server Error")
|
246
|
-
end
|
247
|
-
end
|
163
|
+
def configure(conf)
|
164
|
+
super
|
248
165
|
|
249
|
-
|
250
|
-
|
251
|
-
@response_handler << data
|
252
|
-
rescue
|
253
|
-
$log.error("[input][groonga][response][error] " +
|
254
|
-
"failed to handle HTTP response from Groonga:",
|
255
|
-
:error => "#{$!.class}: #{$!}")
|
256
|
-
$log.error_backtrace
|
257
|
-
reply_error_response("500 Internal Server Error")
|
258
|
-
return
|
259
|
-
end
|
260
|
-
write(data)
|
166
|
+
@port ||= default_port
|
167
|
+
@real_port ||= default_port
|
261
168
|
end
|
262
169
|
|
263
|
-
def
|
264
|
-
|
265
|
-
|
266
|
-
|
267
|
-
|
268
|
-
|
269
|
-
@repeater.close
|
270
|
-
end
|
170
|
+
def emit(command, params)
|
171
|
+
normalized_command = command.split(".")[0]
|
172
|
+
return unless emit_command?(normalized_command)
|
173
|
+
@input_plugin.router.emit("groonga.command.#{normalized_command}",
|
174
|
+
Engine.now,
|
175
|
+
params)
|
271
176
|
end
|
272
177
|
|
273
|
-
|
274
|
-
|
275
|
-
case @request_handler.command
|
276
|
-
when "load", "object_remove"
|
277
|
-
return true
|
278
|
-
end
|
279
|
-
|
280
|
-
case response
|
281
|
-
when Array
|
282
|
-
return_code = response[0][0]
|
283
|
-
return_code.zero?
|
284
|
-
else
|
285
|
-
false
|
286
|
-
end
|
178
|
+
def log
|
179
|
+
@input_plugin.log
|
287
180
|
end
|
288
181
|
|
289
|
-
|
290
|
-
|
291
|
-
|
292
|
-
|
293
|
-
|
294
|
-
write("\r\n")
|
295
|
-
disable
|
296
|
-
on_write_complete do
|
297
|
-
@repeater.close
|
182
|
+
private
|
183
|
+
def emit_command?(command)
|
184
|
+
return true if @emit_commands.empty?
|
185
|
+
@emit_commands.any? do |pattern|
|
186
|
+
pattern === command
|
298
187
|
end
|
299
188
|
end
|
300
189
|
end
|
301
190
|
|
302
|
-
class
|
303
|
-
|
304
|
-
|
305
|
-
def initialize(input, repeater)
|
306
|
-
@input = input
|
307
|
-
@repeater = repeater
|
308
|
-
@parser = Http::Parser.new(self)
|
191
|
+
class HTTPInput < BaseInput
|
192
|
+
def create_handler(connection, repeater)
|
193
|
+
Handler.new(self, connection, repeater)
|
309
194
|
end
|
310
195
|
|
311
|
-
|
312
|
-
|
196
|
+
private
|
197
|
+
def default_port
|
198
|
+
10041
|
313
199
|
end
|
314
200
|
|
315
|
-
|
316
|
-
|
317
|
-
|
318
|
-
|
319
|
-
|
201
|
+
class Handler
|
202
|
+
def initialize(input, connection, repeater)
|
203
|
+
@input = input
|
204
|
+
@connection = connection
|
205
|
+
@repeater = repeater
|
206
|
+
@request_handler = RequestHandler.new(@input, @repeater)
|
207
|
+
@response_handler = ResponseHandler.new(self, @input)
|
208
|
+
end
|
320
209
|
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
|
328
|
-
|
329
|
-
|
330
|
-
|
331
|
-
@
|
332
|
-
|
333
|
-
|
210
|
+
def on_read(data)
|
211
|
+
begin
|
212
|
+
@request_handler << data
|
213
|
+
rescue HTTP::Parser::Error, URI::InvalidURIError
|
214
|
+
@input.log.error("[input][groonga][request][error] " +
|
215
|
+
"failed to parse HTTP request:",
|
216
|
+
:error => "#{$!.class}: #{$!}")
|
217
|
+
@input.log.error_backtrace
|
218
|
+
reply_error_response("400 Bad Request")
|
219
|
+
rescue
|
220
|
+
@input.log.error("[input][groonga][request][error] " +
|
221
|
+
"failed to handle HTTP request:",
|
222
|
+
:error => "#{$!.class}: #{$!}")
|
223
|
+
@input.log.error_backtrace
|
224
|
+
reply_error_response("500 Internal Server Error")
|
334
225
|
end
|
335
226
|
end
|
336
|
-
@repeater.write("\r\n")
|
337
|
-
end
|
338
227
|
|
339
|
-
|
340
|
-
|
341
|
-
|
342
|
-
|
228
|
+
def write_back(data)
|
229
|
+
begin
|
230
|
+
@response_handler << data
|
231
|
+
rescue
|
232
|
+
@input.log.error("[input][groonga][response][error] " +
|
233
|
+
"failed to handle HTTP response from Groonga:",
|
234
|
+
:error => "#{$!.class}: #{$!}")
|
235
|
+
@input.log.error_backtrace
|
236
|
+
reply_error_response("500 Internal Server Error")
|
237
|
+
return
|
238
|
+
end
|
239
|
+
@connection.write(data)
|
240
|
+
end
|
343
241
|
|
344
|
-
|
345
|
-
|
346
|
-
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
|
351
|
-
if command == "load"
|
352
|
-
params["values"] = @body unless @body.empty?
|
242
|
+
def on_response_complete(response)
|
243
|
+
if need_emit?(response)
|
244
|
+
@input.emit(@request_handler.command,
|
245
|
+
@request_handler.params)
|
246
|
+
end
|
247
|
+
@connection.on(:write_complete) do
|
248
|
+
@repeater.close
|
353
249
|
end
|
354
|
-
@command = command
|
355
|
-
@params = params
|
356
250
|
end
|
357
|
-
end
|
358
|
-
end
|
359
251
|
|
360
|
-
|
361
|
-
|
362
|
-
|
363
|
-
@parser = Http::Parser.new(self)
|
364
|
-
end
|
252
|
+
def close
|
253
|
+
@connection.close
|
254
|
+
end
|
365
255
|
|
366
|
-
|
367
|
-
|
368
|
-
|
256
|
+
private
|
257
|
+
def need_emit?(response)
|
258
|
+
case @request_handler.command
|
259
|
+
when "load", "object_remove"
|
260
|
+
return true
|
261
|
+
end
|
369
262
|
|
370
|
-
|
371
|
-
|
372
|
-
|
373
|
-
|
263
|
+
case response
|
264
|
+
when Array
|
265
|
+
return_code = response[0][0]
|
266
|
+
return_code.zero?
|
267
|
+
else
|
268
|
+
false
|
269
|
+
end
|
270
|
+
end
|
374
271
|
|
375
|
-
|
376
|
-
|
377
|
-
|
378
|
-
|
379
|
-
|
272
|
+
def reply_error_response(status)
|
273
|
+
@connection.write("HTTP1.1 #{status}\r\n")
|
274
|
+
@connection.write("Server: fluent-plugin-groonga\r\n")
|
275
|
+
@connection.write("Connection: close\r\n")
|
276
|
+
@connection.write("Content-Length: 0\r\n")
|
277
|
+
@connection.write("\r\n")
|
278
|
+
@connection.on(:write_complete) do
|
279
|
+
@repeater.close
|
380
280
|
end
|
381
281
|
end
|
382
282
|
end
|
383
283
|
|
384
|
-
|
385
|
-
|
386
|
-
|
284
|
+
class RequestHandler
|
285
|
+
attr_reader :command
|
286
|
+
attr_reader :params
|
287
|
+
def initialize(input, repeater)
|
288
|
+
@input = input
|
289
|
+
@repeater = repeater
|
290
|
+
@parser = Http::Parser.new(self)
|
291
|
+
end
|
387
292
|
|
388
|
-
|
389
|
-
|
293
|
+
def <<(chunk)
|
294
|
+
@parser << chunk
|
295
|
+
end
|
390
296
|
|
391
|
-
|
392
|
-
|
393
|
-
|
394
|
-
|
395
|
-
|
396
|
-
|
397
|
-
|
398
|
-
|
399
|
-
|
400
|
-
|
297
|
+
def on_message_begin
|
298
|
+
@body = ""
|
299
|
+
@command = nil
|
300
|
+
@params = nil
|
301
|
+
end
|
302
|
+
|
303
|
+
def on_headers_complete(headers)
|
304
|
+
method = @parser.http_method
|
305
|
+
url = @parser.request_url
|
306
|
+
http_version = @parser.http_version.join(".")
|
307
|
+
@repeater.write("#{method} #{url} HTTP/#{http_version}\r\n")
|
308
|
+
headers.each do |name, value|
|
309
|
+
case name
|
310
|
+
when /\AHost\z/i
|
311
|
+
real_host = @input.real_host
|
312
|
+
real_port = @input.real_port
|
313
|
+
@repeater.write("#{name}: #{real_host}:#{real_port}\r\n")
|
314
|
+
else
|
315
|
+
@repeater.write("#{name}: #{value}\r\n")
|
316
|
+
end
|
401
317
|
end
|
402
|
-
|
403
|
-
|
404
|
-
|
405
|
-
|
406
|
-
|
407
|
-
|
408
|
-
|
409
|
-
|
318
|
+
@repeater.write("\r\n")
|
319
|
+
end
|
320
|
+
|
321
|
+
def on_body(chunk)
|
322
|
+
@body << chunk
|
323
|
+
@repeater.write(chunk)
|
324
|
+
end
|
325
|
+
|
326
|
+
def on_message_complete
|
327
|
+
uri = URI.parse(@parser.request_url)
|
328
|
+
params = WEBrick::HTTPUtils.parse_query(uri.query)
|
329
|
+
path_info = uri.path
|
330
|
+
case path_info
|
331
|
+
when /\A\/d\//
|
332
|
+
command = $POSTMATCH
|
333
|
+
if command == "load"
|
334
|
+
params["values"] = @body unless @body.empty?
|
335
|
+
end
|
336
|
+
@command = command
|
337
|
+
@params = params
|
410
338
|
end
|
411
|
-
when /\Atext\/x-groonga-command-list\z/i
|
412
|
-
response = @body
|
413
339
|
end
|
414
|
-
@handler.on_response_complete(response)
|
415
340
|
end
|
416
|
-
end
|
417
|
-
end
|
418
341
|
|
419
|
-
|
420
|
-
|
421
|
-
|
422
|
-
|
423
|
-
|
342
|
+
class ResponseHandler
|
343
|
+
def initialize(handler, input)
|
344
|
+
@handler = handler
|
345
|
+
@input = input
|
346
|
+
@parser = Http::Parser.new(self)
|
347
|
+
end
|
424
348
|
|
425
|
-
|
426
|
-
|
427
|
-
|
349
|
+
def <<(chunk)
|
350
|
+
@parser << chunk
|
351
|
+
end
|
428
352
|
|
429
|
-
|
430
|
-
|
431
|
-
|
432
|
-
|
433
|
-
end
|
353
|
+
def on_message_begin
|
354
|
+
@body = ""
|
355
|
+
@content_type = nil
|
356
|
+
end
|
434
357
|
|
435
|
-
|
436
|
-
|
437
|
-
|
438
|
-
|
358
|
+
def on_headers_complete(headers)
|
359
|
+
headers.each do |name, value|
|
360
|
+
case name
|
361
|
+
when /\AContent-Type\z/i
|
362
|
+
@content_type = value
|
363
|
+
end
|
364
|
+
end
|
365
|
+
end
|
439
366
|
|
440
|
-
|
441
|
-
|
442
|
-
|
443
|
-
end
|
367
|
+
def on_body(chunk)
|
368
|
+
@body << chunk
|
369
|
+
end
|
444
370
|
|
445
|
-
|
446
|
-
|
371
|
+
def on_message_complete
|
372
|
+
return if @parser.status_code == 100
|
373
|
+
|
374
|
+
response = nil
|
375
|
+
case @content_type
|
376
|
+
when /\Aapplication\/json\z/i
|
377
|
+
begin
|
378
|
+
response = JSON.parse(@body)
|
379
|
+
rescue JSON::ParserError
|
380
|
+
@input.log.warn("[input][groonga][response][warn] " +
|
381
|
+
"failed to parse response JSON:",
|
382
|
+
:error => "#{$!.class}: #{$!}",
|
383
|
+
:json => @body)
|
384
|
+
end
|
385
|
+
when /\Aapplication\/x-msgpack\z/i
|
386
|
+
begin
|
387
|
+
response = MessagePack.unpack(@body)
|
388
|
+
rescue MessagePack::UnpackError, EOFError
|
389
|
+
@input.log.warn("[input][groonga][response][warn] " +
|
390
|
+
"failed to parse response MessagePack",
|
391
|
+
:error => "#{$!.class}: #{$!}",
|
392
|
+
:msgpack => @body)
|
393
|
+
end
|
394
|
+
when /\Atext\/x-groonga-command-list\z/i
|
395
|
+
response = @body
|
396
|
+
end
|
397
|
+
@handler.on_response_complete(response)
|
398
|
+
end
|
447
399
|
end
|
448
400
|
end
|
449
401
|
|
450
|
-
class
|
451
|
-
def
|
452
|
-
|
453
|
-
@input = input
|
454
|
-
initialize_command_parser
|
402
|
+
class GQTPInput < BaseInput
|
403
|
+
def create_handler(connection, repeater)
|
404
|
+
Handler.new(self, connection, repeater)
|
455
405
|
end
|
456
406
|
|
457
|
-
|
458
|
-
|
407
|
+
private
|
408
|
+
def default_port
|
409
|
+
10043
|
459
410
|
end
|
460
411
|
|
461
|
-
|
462
|
-
|
463
|
-
|
412
|
+
class Handler
|
413
|
+
def initialize(input, connection, repeater)
|
414
|
+
@input = input
|
415
|
+
@connection = connection
|
416
|
+
@repeater = repeater
|
417
|
+
|
418
|
+
@request_parser = RequestParser.new(@input)
|
419
|
+
end
|
464
420
|
|
465
|
-
|
466
|
-
|
421
|
+
def on_read(data)
|
422
|
+
@request_parser << data
|
423
|
+
@repeater.write(data)
|
424
|
+
end
|
425
|
+
|
426
|
+
def write_back(data)
|
427
|
+
@connection.write(data)
|
428
|
+
end
|
429
|
+
|
430
|
+
def close
|
431
|
+
@request_parser.close
|
432
|
+
@connection.close
|
433
|
+
end
|
467
434
|
end
|
468
435
|
|
469
|
-
|
470
|
-
|
471
|
-
|
472
|
-
|
473
|
-
|
474
|
-
end
|
475
|
-
|
476
|
-
|
477
|
-
|
478
|
-
|
479
|
-
|
436
|
+
class RequestParser < GQTP::Parser
|
437
|
+
def initialize(input)
|
438
|
+
super()
|
439
|
+
@input = input
|
440
|
+
initialize_command_parser
|
441
|
+
end
|
442
|
+
|
443
|
+
def on_body(chunk)
|
444
|
+
@command_parser << chunk
|
445
|
+
end
|
446
|
+
|
447
|
+
def on_complete
|
448
|
+
@command_parser << "\n"
|
449
|
+
end
|
450
|
+
|
451
|
+
def close
|
452
|
+
@command_parser.finish
|
453
|
+
end
|
454
|
+
|
455
|
+
private
|
456
|
+
def initialize_command_parser
|
457
|
+
@command_parser = Groonga::Command::Parser.new
|
458
|
+
@command_parser.on_command do |command|
|
459
|
+
@input.emit(command.command_name, command.arguments)
|
460
|
+
end
|
461
|
+
@command_parser.on_load_value do |command, value|
|
462
|
+
arguments = command.arguments.dup
|
463
|
+
arguments[:columns] = command.columns.join(", ")
|
464
|
+
arguments[:values] = Yajl::Encoder.encode([value])
|
465
|
+
@input.emit(command.command_name, arguments)
|
466
|
+
end
|
480
467
|
end
|
481
468
|
end
|
482
469
|
end
|