fluent-plugin-groonga 1.1.8 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/doc/text/configuration.md +23 -20
- data/doc/text/constitution.md +19 -23
- data/doc/text/news.md +9 -3
- data/fluent-plugin-groonga.gemspec +3 -4
- data/lib/fluent/plugin/in_groonga.rb +357 -370
- data/lib/fluent/plugin/out_groonga.rb +599 -584
- data/sample/command.conf +5 -4
- data/sample/gqtp.conf +5 -4
- data/sample/http.conf +5 -4
- data/sample/store-apache.conf +5 -3
- data/sample/store-syslog.conf +5 -3
- data/sample/store.conf +5 -3
- data/test/output/test_table_definition.rb +1 -1
- data/test/output/test_table_index_definition.rb +2 -2
- data/test/output/test_type_guesser.rb +1 -1
- data/test/run-test.rb +4 -1
- data/test/test_input.rb +27 -75
- data/test/test_output.rb +27 -65
- metadata +6 -12
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 58980511ad7775baea84617dc9ab6ed0f93e29dc
|
4
|
+
data.tar.gz: 63193a59c766554172b3d34a4d3f40c47465d594
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c44c1ad1b43b1144ff703d61092a8ccf76735c5dfdfce75f5f33023b7e1969b78154629205d68b52d0b6ff9889f0b0a2a254a1e225777ded1cfcc23529293a71
|
7
|
+
data.tar.gz: 7a1b9fdef294db2a6ab661ad7d4619486a887e82e5e674f6e0d1243cb301fc7281a6df8c3463cad04bcab63cf359ff8bc9be6edcb773a61234243b551bd0c6a8
|
data/doc/text/configuration.md
CHANGED
@@ -8,7 +8,25 @@ describes configuration parameters of them.
|
|
8
8
|
|
9
9
|
## The `groonga` input plugin
|
10
10
|
|
11
|
-
|
11
|
+
The behavior of `groonga` input plugin is customized in `system`
|
12
|
+
directive and `source` directive parameters.
|
13
|
+
|
14
|
+
Here is recommended parameter in `system` directive:
|
15
|
+
|
16
|
+
* `workers`: It specifies the number of workers. The point of view
|
17
|
+
in performance, the recommended value is greater than 1 to process
|
18
|
+
requests in parallel.
|
19
|
+
|
20
|
+
* recommended value: greater than `1`.
|
21
|
+
|
22
|
+
* default: `1`
|
23
|
+
|
24
|
+
Note that there is one exception about the number of recommended
|
25
|
+
workers. Users should use `1` worker when commands contain DDL such as
|
26
|
+
`table_create` and `column_create`. Because execution order of these
|
27
|
+
commands may be swapped. In such a case, replication will fail.
|
28
|
+
|
29
|
+
Here are available parameters in `source` directive:
|
12
30
|
|
13
31
|
* `protocol`: It specifies protocol for receiving Groonga commands.
|
14
32
|
|
@@ -32,28 +50,14 @@ Here are available parameters:
|
|
32
50
|
|
33
51
|
* default: `10041`
|
34
52
|
|
35
|
-
* `command_name_position`: It specifies where Groonga command's
|
36
|
-
name.
|
37
|
-
|
38
|
-
If `tag` is specified, the plugin puts Groonga command's name to
|
39
|
-
tag as `groonga.command.#{COMMAND_NAME}` format and record
|
40
|
-
is the arguments of the command.
|
41
|
-
|
42
|
-
If `record` is specified, the plugin puts both Groonga command's
|
43
|
-
name and arguments to record as `{"name": "#{COMMAND_NAME}",
|
44
|
-
"arguments": {...}}` format. Tag is always `groonga.command`.
|
45
|
-
|
46
|
-
`record` is suitable when you want to implement replication. If
|
47
|
-
you `tag` for replication, Groonga command's order may be changed.
|
48
|
-
|
49
|
-
* Available values: `tag`, `record`
|
50
|
-
|
51
|
-
* default: `tag`
|
52
|
-
|
53
53
|
* `emit_commands`: TODO
|
54
54
|
|
55
55
|
Here is an example:
|
56
56
|
|
57
|
+
<system>
|
58
|
+
workers 2
|
59
|
+
</system>
|
60
|
+
|
57
61
|
<source>
|
58
62
|
@type groonga
|
59
63
|
protocol http
|
@@ -61,7 +65,6 @@ Here is an example:
|
|
61
65
|
port 10041
|
62
66
|
real_host 192.168.0.1
|
63
67
|
real_port 10041
|
64
|
-
command_name_position record
|
65
68
|
</source>
|
66
69
|
|
67
70
|
## The `groonga` output plugin
|
data/doc/text/constitution.md
CHANGED
@@ -66,21 +66,19 @@ Here is an example configuration file:
|
|
66
66
|
# For master Groonga server
|
67
67
|
<source>
|
68
68
|
@type groonga
|
69
|
-
protocol gqtp
|
69
|
+
protocol gqtp # Or use the below line
|
70
70
|
# protocol http
|
71
|
-
bind 127.0.0.1
|
72
|
-
# bind 192.168.0.1
|
71
|
+
bind 127.0.0.1 # For client side Fluentd
|
72
|
+
# bind 192.168.0.1 # For master Groonga server side Fluentd
|
73
73
|
port 10041
|
74
|
-
real_host 192.168.29.1
|
75
|
-
real_port 10041
|
76
|
-
# real_port 20041
|
77
|
-
|
78
|
-
|
79
|
-
command_name_position record # To keep command order
|
74
|
+
real_host 192.168.29.1 # IP address of master Groonga server
|
75
|
+
real_port 10041 # Port number of master Groonga server
|
76
|
+
# real_port 20041 # Use different port number
|
77
|
+
# for master Groonga server side Fluentd
|
80
78
|
</source>
|
81
79
|
|
82
80
|
# For slave Groonga server
|
83
|
-
<match groonga.command
|
81
|
+
<match groonga.command.*>
|
84
82
|
@type groonga
|
85
83
|
protocol gqtp # Or use the below line
|
86
84
|
# protocol http # You can use different protocol for
|
@@ -128,7 +126,7 @@ You cannot update data until fluentd is up.
|
|
128
126
|
Here are recover steps when master Groonga server is down:
|
129
127
|
|
130
128
|
1. Stop fluentd.
|
131
|
-
2. Run `
|
129
|
+
2. Run `grndump /PATH/TO/SLAVE/GROONGA/SERVER/DB >
|
132
130
|
SLAVE_GROONGA_DUMP.grn` on slave Groonga server host.
|
133
131
|
3. Run `groonga -n /PATH/TO/MASTER/GROONGA/SERVER/DB <
|
134
132
|
SLAVE_GROONGA_DUMP.grn` on master Groonga server.
|
@@ -141,7 +139,7 @@ You cannot update data until you finish to recover.
|
|
141
139
|
|
142
140
|
Here are recover steps when slave Groonga server is down:
|
143
141
|
|
144
|
-
1. Run `
|
142
|
+
1. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
|
145
143
|
MASTER_GROONGA_DUMP.grn` on master Groonga server host.
|
146
144
|
2. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
|
147
145
|
MASTER_GROONGA_DUMP.grn` on slave Groonga server.
|
@@ -157,7 +155,7 @@ is full (see `buffer_queue_limit`) or fluentd gives up retrying (see
|
|
157
155
|
Groonga server before those situations:
|
158
156
|
|
159
157
|
1. Stop fluentd.
|
160
|
-
2. Run `
|
158
|
+
2. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
|
161
159
|
MASTER_GROONGA_DUMP.grn` on master Groonga server host.
|
162
160
|
3. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
|
163
161
|
MASTER_GROONGA_DUMP.grn` on slave Groonga server host.
|
@@ -196,21 +194,19 @@ Here is an example configuration file:
|
|
196
194
|
# For master Groonga server
|
197
195
|
<source>
|
198
196
|
@type groonga
|
199
|
-
protocol gqtp
|
197
|
+
protocol gqtp # Or use the below line
|
200
198
|
# protocol http
|
201
|
-
bind 127.0.0.1
|
202
|
-
# bind 192.168.0.1
|
199
|
+
bind 127.0.0.1 # For client side Fluentd
|
200
|
+
# bind 192.168.0.1 # For master Groonga server side Fluentd
|
203
201
|
port 10041
|
204
|
-
real_host 192.168.29.1
|
205
|
-
real_port 10041
|
206
|
-
# real_port 20041
|
207
|
-
|
208
|
-
|
209
|
-
command_name_position record # To keep command order
|
202
|
+
real_host 192.168.29.1 # IP address of master Groonga server
|
203
|
+
real_port 10041 # Port number of master Groonga server
|
204
|
+
# real_port 20041 # Use different port number
|
205
|
+
# for master Groonga server side fluentd
|
210
206
|
</source>
|
211
207
|
|
212
208
|
# For slave Groonga servers
|
213
|
-
<match groonga.command
|
209
|
+
<match groonga.command.*>
|
214
210
|
@type copy
|
215
211
|
|
216
212
|
# The first slave Groonga server
|
data/doc/text/news.md
CHANGED
@@ -2,12 +2,18 @@
|
|
2
2
|
|
3
3
|
# News
|
4
4
|
|
5
|
-
## 1.
|
5
|
+
## 1.2.0: 2017-04-26
|
6
6
|
|
7
7
|
### Improvements
|
8
8
|
|
9
|
-
*
|
10
|
-
|
9
|
+
* Supported recent fluentd v0.14 API.
|
10
|
+
Since fluentd 0.14.12, compatibility layer is unexpectedly broken,
|
11
|
+
fluent-plugin-groonga had been also affected because
|
12
|
+
fluent-plugin-groonga relied on it. Note that fluent-plugin-groonga
|
13
|
+
does not work with fluentd v0.12 because it does not use
|
14
|
+
compatibility layer anymore. We recommends to use latest release,
|
15
|
+
but if you still stay with fluentd v0.12, you need to use
|
16
|
+
fluent-plugin-groonga 1.1.7.
|
11
17
|
|
12
18
|
## 1.1.7: 2017-04-04
|
13
19
|
|
@@ -1,7 +1,6 @@
|
|
1
1
|
# -*- mode: ruby; coding: utf-8 -*-
|
2
2
|
#
|
3
|
-
# Copyright (C)
|
4
|
-
# Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
|
3
|
+
# Copyright (C) 2012-2016 Kouhei Sutou <kou@clear-code.com>
|
5
4
|
#
|
6
5
|
# This library is free software; you can redistribute it and/or
|
7
6
|
# modify it under the terms of the GNU Lesser General Public
|
@@ -18,7 +17,7 @@
|
|
18
17
|
|
19
18
|
Gem::Specification.new do |spec|
|
20
19
|
spec.name = "fluent-plugin-groonga"
|
21
|
-
spec.version = "1.
|
20
|
+
spec.version = "1.2.0"
|
22
21
|
spec.authors = ["Kouhei Sutou"]
|
23
22
|
spec.email = ["kou@clear-code.com"]
|
24
23
|
spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
|
@@ -35,7 +34,7 @@ Gem::Specification.new do |spec|
|
|
35
34
|
spec.test_files += Dir.glob("test/**/*")
|
36
35
|
spec.require_paths = ["lib"]
|
37
36
|
|
38
|
-
spec.add_runtime_dependency("fluentd", ">= 0.
|
37
|
+
spec.add_runtime_dependency("fluentd", ">= 0.14.0")
|
39
38
|
spec.add_runtime_dependency("groonga-client", ">= 0.1.0")
|
40
39
|
spec.add_runtime_dependency("groonga-command-parser")
|
41
40
|
|
@@ -1,4 +1,3 @@
|
|
1
|
-
# Copyright (C) 2018 Yasuhiro Horimoto <horimoto@clear-code.com>
|
2
1
|
# Copyright (C) 2012-2017 Kouhei Sutou <kou@clear-code.com>
|
3
2
|
#
|
4
3
|
# This library is free software; you can redistribute it and/or
|
@@ -23,460 +22,448 @@ require "http_parser"
|
|
23
22
|
require "gqtp"
|
24
23
|
require "groonga/command/parser"
|
25
24
|
|
26
|
-
require "fluent/input"
|
27
|
-
require "fluent/process"
|
25
|
+
require "fluent/plugin/input"
|
28
26
|
|
29
27
|
module Fluent
|
30
|
-
|
31
|
-
|
28
|
+
module Plugin
|
29
|
+
class GroongaInput < Input
|
30
|
+
Plugin.register_input("groonga", self)
|
32
31
|
|
33
|
-
|
34
|
-
super
|
35
|
-
end
|
36
|
-
|
37
|
-
config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
|
38
|
-
config_param :command_name_position, :enum, :list => [:tag, :record], :default => :tag
|
39
|
-
|
40
|
-
def configure(conf)
|
41
|
-
super
|
42
|
-
case @protocol
|
43
|
-
when :http
|
44
|
-
@input = HTTPInput.new(self)
|
45
|
-
when :gqtp
|
46
|
-
@input = GQTPInput.new(self)
|
47
|
-
end
|
48
|
-
@input.configure(conf)
|
49
|
-
end
|
50
|
-
|
51
|
-
def start
|
52
|
-
super
|
53
|
-
@input.start
|
54
|
-
end
|
55
|
-
|
56
|
-
def shutdown
|
57
|
-
super
|
58
|
-
@input.shutdown
|
59
|
-
end
|
60
|
-
|
61
|
-
class Repeater < Coolio::TCPSocket
|
62
|
-
def initialize(socket, handler)
|
63
|
-
super(socket)
|
64
|
-
@handler = handler
|
65
|
-
end
|
66
|
-
|
67
|
-
def on_read(data)
|
68
|
-
@handler.write_back(data)
|
69
|
-
end
|
70
|
-
|
71
|
-
def on_close
|
72
|
-
@handler.close
|
73
|
-
end
|
74
|
-
end
|
32
|
+
helpers :server
|
75
33
|
|
76
|
-
|
77
|
-
|
78
|
-
include DetachMultiProcessMixin
|
79
|
-
|
80
|
-
config_param :bind, :string, :default => "0.0.0.0"
|
81
|
-
config_param :port, :integer, :default => nil
|
82
|
-
config_param :real_host, :string
|
83
|
-
config_param :real_port, :integer, :default => nil
|
84
|
-
DEFAULT_EMIT_COMMANDS = [
|
85
|
-
"clearlock",
|
86
|
-
"column_copy",
|
87
|
-
"column_create",
|
88
|
-
"column_remove",
|
89
|
-
"column_rename",
|
90
|
-
"config_delete",
|
91
|
-
"config_set",
|
92
|
-
"delete",
|
93
|
-
"load",
|
94
|
-
"lock_acquire",
|
95
|
-
"lock_clear",
|
96
|
-
"lock_release",
|
97
|
-
"logical_table_remove",
|
98
|
-
"object_remove",
|
99
|
-
"plugin_register",
|
100
|
-
"plugin_unregister",
|
101
|
-
"register",
|
102
|
-
"reindex",
|
103
|
-
"table_copy",
|
104
|
-
"table_create",
|
105
|
-
"table_remove",
|
106
|
-
"table_rename",
|
107
|
-
"truncate",
|
108
|
-
]
|
109
|
-
config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
|
110
|
-
commands = value.split(/\s*,\s*/)
|
111
|
-
commands.collect do |command|
|
112
|
-
if /\A\/(.*)\/(i)?\z/ =~ command
|
113
|
-
pattern = $1
|
114
|
-
flag_mark = $2
|
115
|
-
flag = 0
|
116
|
-
flag |= Regexp::IGNORECASE if flag_mark == "i"
|
117
|
-
Regexp.new(pattern, flag)
|
118
|
-
else
|
119
|
-
command
|
120
|
-
end
|
121
|
-
end
|
34
|
+
def initialize
|
35
|
+
super
|
122
36
|
end
|
123
37
|
|
124
|
-
|
125
|
-
@input_plugin = input_plugin
|
126
|
-
end
|
38
|
+
config_param :protocol, :enum, :list => [:http, :gqtp], :default => :http
|
127
39
|
|
128
40
|
def configure(conf)
|
129
41
|
super
|
130
|
-
|
131
|
-
|
132
|
-
|
42
|
+
case @protocol
|
43
|
+
when :http
|
44
|
+
@input = HTTPInput.new(self)
|
45
|
+
when :gqtp
|
46
|
+
@input = GQTPInput.new(self)
|
47
|
+
end
|
48
|
+
@input.configure(conf)
|
133
49
|
end
|
134
50
|
|
135
51
|
def start
|
136
|
-
|
137
|
-
detach_multi_process do
|
138
|
-
@loop = Coolio::Loop.new
|
139
|
-
|
140
|
-
@socket = Coolio::TCPServer.new(listen_socket, nil,
|
141
|
-
handler_class, self)
|
142
|
-
@loop.attach(@socket)
|
52
|
+
super
|
143
53
|
|
144
|
-
|
145
|
-
|
54
|
+
port = @input.port
|
55
|
+
bind = @input.bind
|
56
|
+
log.info("[input][groonga][connect] listening port",
|
57
|
+
:port => port, :bind => bind)
|
58
|
+
server_create_connection(:groonga_input,
|
59
|
+
port,
|
60
|
+
:proto => :tcp,
|
61
|
+
:shared => system_config.workers > 1,
|
62
|
+
:bind => bind) do |connection|
|
63
|
+
handler = nil
|
64
|
+
real_host = @input.real_host
|
65
|
+
real_port = @input.real_port
|
66
|
+
repeater = Coolio::TCPSocket.connect(real_host, real_port)
|
67
|
+
repeater.on_connect_failed do
|
68
|
+
log.error("[input][groonga][connect][error] " +
|
69
|
+
"failed to connect to Groonga:",
|
70
|
+
:real_host => real_host,
|
71
|
+
:real_port => real_port)
|
72
|
+
connection.close
|
73
|
+
end
|
74
|
+
repeater.on_read do |data|
|
75
|
+
handler.write_back(data)
|
76
|
+
end
|
77
|
+
repeater.on_close do
|
78
|
+
handler.close
|
79
|
+
end
|
80
|
+
event_loop_attach(repeater)
|
146
81
|
|
147
|
-
|
148
|
-
|
82
|
+
handler = @input.create_handler(connection, repeater)
|
83
|
+
connection.data do |data|
|
84
|
+
handler.on_read(data)
|
149
85
|
end
|
150
86
|
end
|
151
87
|
end
|
152
88
|
|
153
89
|
def shutdown
|
154
|
-
|
155
|
-
@socket.close
|
156
|
-
@shutdown_notifier.signal
|
157
|
-
@thread.join
|
90
|
+
super
|
158
91
|
end
|
159
92
|
|
160
|
-
def
|
161
|
-
|
162
|
-
repeater.attach(@loop)
|
163
|
-
repeater
|
93
|
+
def multi_workers_ready?
|
94
|
+
true
|
164
95
|
end
|
165
96
|
|
166
|
-
|
167
|
-
|
168
|
-
|
169
|
-
|
170
|
-
when :tag
|
171
|
-
tag = "groonga.command.#{normalized_command}"
|
172
|
-
record = params
|
173
|
-
else
|
174
|
-
tag = "groonga.command"
|
175
|
-
record = {
|
176
|
-
"name" => normalized_command,
|
177
|
-
"arguments" => params
|
178
|
-
}
|
97
|
+
class Repeater < Coolio::TCPSocket
|
98
|
+
def initialize(socket, handler)
|
99
|
+
super(socket)
|
100
|
+
@handler = handler
|
179
101
|
end
|
180
|
-
@input_plugin.router.emit(tag,
|
181
|
-
Engine.now,
|
182
|
-
record)
|
183
|
-
end
|
184
102
|
|
185
|
-
|
186
|
-
|
187
|
-
@loop.run
|
188
|
-
rescue
|
189
|
-
$log.error("[input][groonga][error] unexpected error",
|
190
|
-
:error => "#{$!.class}: #{$!}")
|
191
|
-
$log.error_backtrace
|
192
|
-
end
|
193
|
-
|
194
|
-
def emit_command?(command)
|
195
|
-
return true if @emit_commands.empty?
|
196
|
-
@emit_commands.any? do |pattern|
|
197
|
-
pattern === command
|
103
|
+
def on_read(data)
|
104
|
+
@handler.write_back(data)
|
198
105
|
end
|
199
|
-
end
|
200
|
-
end
|
201
106
|
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
10041
|
206
|
-
end
|
207
|
-
|
208
|
-
def handler_class
|
209
|
-
Handler
|
107
|
+
def on_close
|
108
|
+
@handler.close
|
109
|
+
end
|
210
110
|
end
|
211
111
|
|
212
|
-
class
|
213
|
-
|
214
|
-
|
215
|
-
|
112
|
+
class BaseInput
|
113
|
+
include Configurable
|
114
|
+
|
115
|
+
config_param :bind, :string, :default => "0.0.0.0"
|
116
|
+
config_param :port, :integer, :default => nil
|
117
|
+
config_param :real_host, :string
|
118
|
+
config_param :real_port, :integer, :default => nil
|
119
|
+
DEFAULT_EMIT_COMMANDS = [
|
120
|
+
"clearlock",
|
121
|
+
"column_copy",
|
122
|
+
"column_create",
|
123
|
+
"column_remove",
|
124
|
+
"column_rename",
|
125
|
+
"config_delete",
|
126
|
+
"config_set",
|
127
|
+
"delete",
|
128
|
+
"load",
|
129
|
+
"lock_acquire",
|
130
|
+
"lock_clear",
|
131
|
+
"lock_release",
|
132
|
+
"logical_table_remove",
|
133
|
+
"object_remove",
|
134
|
+
"plugin_register",
|
135
|
+
"plugin_unregister",
|
136
|
+
"register",
|
137
|
+
"reindex",
|
138
|
+
"table_copy",
|
139
|
+
"table_create",
|
140
|
+
"table_remove",
|
141
|
+
"table_rename",
|
142
|
+
"truncate",
|
143
|
+
]
|
144
|
+
config_param :emit_commands, :default => DEFAULT_EMIT_COMMANDS do |value|
|
145
|
+
commands = value.split(/\s*,\s*/)
|
146
|
+
commands.collect do |command|
|
147
|
+
if /\A\/(.*)\/(i)?\z/ =~ command
|
148
|
+
pattern = $1
|
149
|
+
flag_mark = $2
|
150
|
+
flag = 0
|
151
|
+
flag |= Regexp::IGNORECASE if flag_mark == "i"
|
152
|
+
Regexp.new(pattern, flag)
|
153
|
+
else
|
154
|
+
command
|
155
|
+
end
|
156
|
+
end
|
216
157
|
end
|
217
158
|
|
218
|
-
def
|
219
|
-
@
|
220
|
-
@repeater.on_connect_failed do
|
221
|
-
$log.error("[input][groonga][connect][error] " +
|
222
|
-
"failed to connect to Groonga:",
|
223
|
-
:real_host => @input.real_host,
|
224
|
-
:real_port => @input.real_port)
|
225
|
-
close
|
226
|
-
end
|
227
|
-
@request_handler = RequestHandler.new(@input, @repeater)
|
228
|
-
@response_handler = ResponseHandler.new(self)
|
159
|
+
def initialize(input_plugin)
|
160
|
+
@input_plugin = input_plugin
|
229
161
|
end
|
230
162
|
|
231
|
-
def
|
232
|
-
|
233
|
-
@request_handler << data
|
234
|
-
rescue HTTP::Parser::Error, URI::InvalidURIError
|
235
|
-
$log.error("[input][groonga][request][error] " +
|
236
|
-
"failed to parse HTTP request:",
|
237
|
-
:error => "#{$!.class}: #{$!}")
|
238
|
-
$log.error_backtrace
|
239
|
-
reply_error_response("400 Bad Request")
|
240
|
-
rescue
|
241
|
-
$log.error("[input][groonga][request][error] " +
|
242
|
-
"failed to handle HTTP request:",
|
243
|
-
:error => "#{$!.class}: #{$!}")
|
244
|
-
$log.error_backtrace
|
245
|
-
reply_error_response("500 Internal Server Error")
|
246
|
-
end
|
247
|
-
end
|
163
|
+
def configure(conf)
|
164
|
+
super
|
248
165
|
|
249
|
-
|
250
|
-
|
251
|
-
@response_handler << data
|
252
|
-
rescue
|
253
|
-
$log.error("[input][groonga][response][error] " +
|
254
|
-
"failed to handle HTTP response from Groonga:",
|
255
|
-
:error => "#{$!.class}: #{$!}")
|
256
|
-
$log.error_backtrace
|
257
|
-
reply_error_response("500 Internal Server Error")
|
258
|
-
return
|
259
|
-
end
|
260
|
-
write(data)
|
166
|
+
@port ||= default_port
|
167
|
+
@real_port ||= default_port
|
261
168
|
end
|
262
169
|
|
263
|
-
def
|
264
|
-
|
265
|
-
|
266
|
-
|
267
|
-
|
268
|
-
|
269
|
-
@repeater.close
|
270
|
-
end
|
170
|
+
def emit(command, params)
|
171
|
+
normalized_command = command.split(".")[0]
|
172
|
+
return unless emit_command?(normalized_command)
|
173
|
+
@input_plugin.router.emit("groonga.command.#{normalized_command}",
|
174
|
+
Engine.now,
|
175
|
+
params)
|
271
176
|
end
|
272
177
|
|
273
|
-
|
274
|
-
|
275
|
-
case @request_handler.command
|
276
|
-
when "load", "object_remove"
|
277
|
-
return true
|
278
|
-
end
|
279
|
-
|
280
|
-
case response
|
281
|
-
when Array
|
282
|
-
return_code = response[0][0]
|
283
|
-
return_code.zero?
|
284
|
-
else
|
285
|
-
false
|
286
|
-
end
|
178
|
+
def log
|
179
|
+
@input_plugin.log
|
287
180
|
end
|
288
181
|
|
289
|
-
|
290
|
-
|
291
|
-
|
292
|
-
|
293
|
-
|
294
|
-
write("\r\n")
|
295
|
-
disable
|
296
|
-
on_write_complete do
|
297
|
-
@repeater.close
|
182
|
+
private
|
183
|
+
def emit_command?(command)
|
184
|
+
return true if @emit_commands.empty?
|
185
|
+
@emit_commands.any? do |pattern|
|
186
|
+
pattern === command
|
298
187
|
end
|
299
188
|
end
|
300
189
|
end
|
301
190
|
|
302
|
-
class
|
303
|
-
|
304
|
-
|
305
|
-
def initialize(input, repeater)
|
306
|
-
@input = input
|
307
|
-
@repeater = repeater
|
308
|
-
@parser = Http::Parser.new(self)
|
191
|
+
class HTTPInput < BaseInput
|
192
|
+
def create_handler(connection, repeater)
|
193
|
+
Handler.new(self, connection, repeater)
|
309
194
|
end
|
310
195
|
|
311
|
-
|
312
|
-
|
196
|
+
private
|
197
|
+
def default_port
|
198
|
+
10041
|
313
199
|
end
|
314
200
|
|
315
|
-
|
316
|
-
|
317
|
-
|
318
|
-
|
319
|
-
|
201
|
+
class Handler
|
202
|
+
def initialize(input, connection, repeater)
|
203
|
+
@input = input
|
204
|
+
@connection = connection
|
205
|
+
@repeater = repeater
|
206
|
+
@request_handler = RequestHandler.new(@input, @repeater)
|
207
|
+
@response_handler = ResponseHandler.new(self, @input)
|
208
|
+
end
|
320
209
|
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
|
328
|
-
|
329
|
-
|
330
|
-
|
331
|
-
@
|
332
|
-
|
333
|
-
|
210
|
+
def on_read(data)
|
211
|
+
begin
|
212
|
+
@request_handler << data
|
213
|
+
rescue HTTP::Parser::Error, URI::InvalidURIError
|
214
|
+
@input.log.error("[input][groonga][request][error] " +
|
215
|
+
"failed to parse HTTP request:",
|
216
|
+
:error => "#{$!.class}: #{$!}")
|
217
|
+
@input.log.error_backtrace
|
218
|
+
reply_error_response("400 Bad Request")
|
219
|
+
rescue
|
220
|
+
@input.log.error("[input][groonga][request][error] " +
|
221
|
+
"failed to handle HTTP request:",
|
222
|
+
:error => "#{$!.class}: #{$!}")
|
223
|
+
@input.log.error_backtrace
|
224
|
+
reply_error_response("500 Internal Server Error")
|
334
225
|
end
|
335
226
|
end
|
336
|
-
@repeater.write("\r\n")
|
337
|
-
end
|
338
227
|
|
339
|
-
|
340
|
-
|
341
|
-
|
342
|
-
|
228
|
+
def write_back(data)
|
229
|
+
begin
|
230
|
+
@response_handler << data
|
231
|
+
rescue
|
232
|
+
@input.log.error("[input][groonga][response][error] " +
|
233
|
+
"failed to handle HTTP response from Groonga:",
|
234
|
+
:error => "#{$!.class}: #{$!}")
|
235
|
+
@input.log.error_backtrace
|
236
|
+
reply_error_response("500 Internal Server Error")
|
237
|
+
return
|
238
|
+
end
|
239
|
+
@connection.write(data)
|
240
|
+
end
|
343
241
|
|
344
|
-
|
345
|
-
|
346
|
-
|
347
|
-
|
348
|
-
|
349
|
-
|
350
|
-
|
351
|
-
if command == "load"
|
352
|
-
params["values"] = @body unless @body.empty?
|
242
|
+
def on_response_complete(response)
|
243
|
+
if need_emit?(response)
|
244
|
+
@input.emit(@request_handler.command,
|
245
|
+
@request_handler.params)
|
246
|
+
end
|
247
|
+
@connection.on(:write_complete) do
|
248
|
+
@repeater.close
|
353
249
|
end
|
354
|
-
@command = command
|
355
|
-
@params = params
|
356
250
|
end
|
357
|
-
end
|
358
|
-
end
|
359
251
|
|
360
|
-
|
361
|
-
|
362
|
-
|
363
|
-
@parser = Http::Parser.new(self)
|
364
|
-
end
|
252
|
+
def close
|
253
|
+
@connection.close
|
254
|
+
end
|
365
255
|
|
366
|
-
|
367
|
-
|
368
|
-
|
256
|
+
private
|
257
|
+
def need_emit?(response)
|
258
|
+
case @request_handler.command
|
259
|
+
when "load", "object_remove"
|
260
|
+
return true
|
261
|
+
end
|
369
262
|
|
370
|
-
|
371
|
-
|
372
|
-
|
373
|
-
|
263
|
+
case response
|
264
|
+
when Array
|
265
|
+
return_code = response[0][0]
|
266
|
+
return_code.zero?
|
267
|
+
else
|
268
|
+
false
|
269
|
+
end
|
270
|
+
end
|
374
271
|
|
375
|
-
|
376
|
-
|
377
|
-
|
378
|
-
|
379
|
-
|
272
|
+
def reply_error_response(status)
|
273
|
+
@connection.write("HTTP1.1 #{status}\r\n")
|
274
|
+
@connection.write("Server: fluent-plugin-groonga\r\n")
|
275
|
+
@connection.write("Connection: close\r\n")
|
276
|
+
@connection.write("Content-Length: 0\r\n")
|
277
|
+
@connection.write("\r\n")
|
278
|
+
@connection.on(:write_complete) do
|
279
|
+
@repeater.close
|
380
280
|
end
|
381
281
|
end
|
382
282
|
end
|
383
283
|
|
384
|
-
|
385
|
-
|
386
|
-
|
284
|
+
class RequestHandler
|
285
|
+
attr_reader :command
|
286
|
+
attr_reader :params
|
287
|
+
def initialize(input, repeater)
|
288
|
+
@input = input
|
289
|
+
@repeater = repeater
|
290
|
+
@parser = Http::Parser.new(self)
|
291
|
+
end
|
387
292
|
|
388
|
-
|
389
|
-
|
293
|
+
def <<(chunk)
|
294
|
+
@parser << chunk
|
295
|
+
end
|
390
296
|
|
391
|
-
|
392
|
-
|
393
|
-
|
394
|
-
|
395
|
-
|
396
|
-
|
397
|
-
|
398
|
-
|
399
|
-
|
400
|
-
|
297
|
+
def on_message_begin
|
298
|
+
@body = ""
|
299
|
+
@command = nil
|
300
|
+
@params = nil
|
301
|
+
end
|
302
|
+
|
303
|
+
def on_headers_complete(headers)
|
304
|
+
method = @parser.http_method
|
305
|
+
url = @parser.request_url
|
306
|
+
http_version = @parser.http_version.join(".")
|
307
|
+
@repeater.write("#{method} #{url} HTTP/#{http_version}\r\n")
|
308
|
+
headers.each do |name, value|
|
309
|
+
case name
|
310
|
+
when /\AHost\z/i
|
311
|
+
real_host = @input.real_host
|
312
|
+
real_port = @input.real_port
|
313
|
+
@repeater.write("#{name}: #{real_host}:#{real_port}\r\n")
|
314
|
+
else
|
315
|
+
@repeater.write("#{name}: #{value}\r\n")
|
316
|
+
end
|
401
317
|
end
|
402
|
-
|
403
|
-
|
404
|
-
|
405
|
-
|
406
|
-
|
407
|
-
|
408
|
-
|
409
|
-
|
318
|
+
@repeater.write("\r\n")
|
319
|
+
end
|
320
|
+
|
321
|
+
def on_body(chunk)
|
322
|
+
@body << chunk
|
323
|
+
@repeater.write(chunk)
|
324
|
+
end
|
325
|
+
|
326
|
+
def on_message_complete
|
327
|
+
uri = URI.parse(@parser.request_url)
|
328
|
+
params = WEBrick::HTTPUtils.parse_query(uri.query)
|
329
|
+
path_info = uri.path
|
330
|
+
case path_info
|
331
|
+
when /\A\/d\//
|
332
|
+
command = $POSTMATCH
|
333
|
+
if command == "load"
|
334
|
+
params["values"] = @body unless @body.empty?
|
335
|
+
end
|
336
|
+
@command = command
|
337
|
+
@params = params
|
410
338
|
end
|
411
|
-
when /\Atext\/x-groonga-command-list\z/i
|
412
|
-
response = @body
|
413
339
|
end
|
414
|
-
@handler.on_response_complete(response)
|
415
340
|
end
|
416
|
-
end
|
417
|
-
end
|
418
341
|
|
419
|
-
|
420
|
-
|
421
|
-
|
422
|
-
|
423
|
-
|
342
|
+
class ResponseHandler
|
343
|
+
def initialize(handler, input)
|
344
|
+
@handler = handler
|
345
|
+
@input = input
|
346
|
+
@parser = Http::Parser.new(self)
|
347
|
+
end
|
424
348
|
|
425
|
-
|
426
|
-
|
427
|
-
|
349
|
+
def <<(chunk)
|
350
|
+
@parser << chunk
|
351
|
+
end
|
428
352
|
|
429
|
-
|
430
|
-
|
431
|
-
|
432
|
-
|
433
|
-
end
|
353
|
+
def on_message_begin
|
354
|
+
@body = ""
|
355
|
+
@content_type = nil
|
356
|
+
end
|
434
357
|
|
435
|
-
|
436
|
-
|
437
|
-
|
438
|
-
|
358
|
+
def on_headers_complete(headers)
|
359
|
+
headers.each do |name, value|
|
360
|
+
case name
|
361
|
+
when /\AContent-Type\z/i
|
362
|
+
@content_type = value
|
363
|
+
end
|
364
|
+
end
|
365
|
+
end
|
439
366
|
|
440
|
-
|
441
|
-
|
442
|
-
|
443
|
-
end
|
367
|
+
def on_body(chunk)
|
368
|
+
@body << chunk
|
369
|
+
end
|
444
370
|
|
445
|
-
|
446
|
-
|
371
|
+
def on_message_complete
|
372
|
+
return if @parser.status_code == 100
|
373
|
+
|
374
|
+
response = nil
|
375
|
+
case @content_type
|
376
|
+
when /\Aapplication\/json\z/i
|
377
|
+
begin
|
378
|
+
response = JSON.parse(@body)
|
379
|
+
rescue JSON::ParserError
|
380
|
+
@input.log.warn("[input][groonga][response][warn] " +
|
381
|
+
"failed to parse response JSON:",
|
382
|
+
:error => "#{$!.class}: #{$!}",
|
383
|
+
:json => @body)
|
384
|
+
end
|
385
|
+
when /\Aapplication\/x-msgpack\z/i
|
386
|
+
begin
|
387
|
+
response = MessagePack.unpack(@body)
|
388
|
+
rescue MessagePack::UnpackError, EOFError
|
389
|
+
@input.log.warn("[input][groonga][response][warn] " +
|
390
|
+
"failed to parse response MessagePack",
|
391
|
+
:error => "#{$!.class}: #{$!}",
|
392
|
+
:msgpack => @body)
|
393
|
+
end
|
394
|
+
when /\Atext\/x-groonga-command-list\z/i
|
395
|
+
response = @body
|
396
|
+
end
|
397
|
+
@handler.on_response_complete(response)
|
398
|
+
end
|
447
399
|
end
|
448
400
|
end
|
449
401
|
|
450
|
-
class
|
451
|
-
def
|
452
|
-
|
453
|
-
@input = input
|
454
|
-
initialize_command_parser
|
402
|
+
class GQTPInput < BaseInput
|
403
|
+
def create_handler(connection, repeater)
|
404
|
+
Handler.new(self, connection, repeater)
|
455
405
|
end
|
456
406
|
|
457
|
-
|
458
|
-
|
407
|
+
private
|
408
|
+
def default_port
|
409
|
+
10043
|
459
410
|
end
|
460
411
|
|
461
|
-
|
462
|
-
|
463
|
-
|
412
|
+
class Handler
|
413
|
+
def initialize(input, connection, repeater)
|
414
|
+
@input = input
|
415
|
+
@connection = connection
|
416
|
+
@repeater = repeater
|
417
|
+
|
418
|
+
@request_parser = RequestParser.new(@input)
|
419
|
+
end
|
464
420
|
|
465
|
-
|
466
|
-
|
421
|
+
def on_read(data)
|
422
|
+
@request_parser << data
|
423
|
+
@repeater.write(data)
|
424
|
+
end
|
425
|
+
|
426
|
+
def write_back(data)
|
427
|
+
@connection.write(data)
|
428
|
+
end
|
429
|
+
|
430
|
+
def close
|
431
|
+
@request_parser.close
|
432
|
+
@connection.close
|
433
|
+
end
|
467
434
|
end
|
468
435
|
|
469
|
-
|
470
|
-
|
471
|
-
|
472
|
-
|
473
|
-
|
474
|
-
end
|
475
|
-
|
476
|
-
|
477
|
-
|
478
|
-
|
479
|
-
|
436
|
+
class RequestParser < GQTP::Parser
|
437
|
+
def initialize(input)
|
438
|
+
super()
|
439
|
+
@input = input
|
440
|
+
initialize_command_parser
|
441
|
+
end
|
442
|
+
|
443
|
+
def on_body(chunk)
|
444
|
+
@command_parser << chunk
|
445
|
+
end
|
446
|
+
|
447
|
+
def on_complete
|
448
|
+
@command_parser << "\n"
|
449
|
+
end
|
450
|
+
|
451
|
+
def close
|
452
|
+
@command_parser.finish
|
453
|
+
end
|
454
|
+
|
455
|
+
private
|
456
|
+
def initialize_command_parser
|
457
|
+
@command_parser = Groonga::Command::Parser.new
|
458
|
+
@command_parser.on_command do |command|
|
459
|
+
@input.emit(command.command_name, command.arguments)
|
460
|
+
end
|
461
|
+
@command_parser.on_load_value do |command, value|
|
462
|
+
arguments = command.arguments.dup
|
463
|
+
arguments[:columns] = command.columns.join(", ")
|
464
|
+
arguments[:values] = Yajl::Encoder.encode([value])
|
465
|
+
@input.emit(command.command_name, arguments)
|
466
|
+
end
|
480
467
|
end
|
481
468
|
end
|
482
469
|
end
|