fluent-plugin-groonga 1.0.3 → 1.0.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +120 -8
- data/doc/text/news.md +15 -0
- data/fluent-plugin-groonga.gemspec +5 -5
- data/lib/fluent/plugin/in_groonga.rb +32 -12
- data/lib/fluent/plugin/out_groonga.rb +300 -129
- data/sample/gqtp.conf +2 -2
- data/sample/store.conf +15 -0
- data/test/output/test_type_guesser.rb +127 -0
- data/test/test_output.rb +70 -30
- metadata +43 -38
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 91510d22d9e94a522c9ee54b3a4d0caf4c6f4cc9
|
4
|
+
data.tar.gz: 0e77c6fc21121575977a288c131272c780ae0ecb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 6d00c8ff58928fc8c94b1a6ab98f803cac53517a022bb968ecf1b7e4862442e89f6583c89cedcd2b3b6022e3ff6acb82bb9ad270ec17a98b14b998c00727425a
|
7
|
+
data.tar.gz: 1fb6b8d3faeae3bf096b1df0845f590620ecf08cc41c2e4234ade39297e42efecd2a908c61f64ad316096ddba98c9833c2a8b6439e83ed178133374617b06fc6
|
data/README.md
CHANGED
@@ -8,19 +8,59 @@ fluent-plugin-groonga
|
|
8
8
|
|
9
9
|
## Description
|
10
10
|
|
11
|
-
Fluent-plugin-groonga is
|
12
|
-
[
|
13
|
-
|
11
|
+
Fluent-plugin-groonga is a Fluentd plugin collection to use
|
12
|
+
[Groonga](http://groonga.org/) with Fluentd. Fluent-plugin-groonga
|
13
|
+
supports the following two usages:
|
14
|
+
|
15
|
+
* Store logs collected by Fluentd to Groonga.
|
16
|
+
* Implement replication system for Groonga.
|
17
|
+
|
18
|
+
The first usage is normal usage. You can store logs to Groonga and
|
19
|
+
find logs by full-text search.
|
20
|
+
|
21
|
+
The second usage is for Groonga users. Groonga itself doesn't support
|
22
|
+
replication. But Groonga users can replicate their data by
|
23
|
+
fluent-plugin-groonga.
|
14
24
|
|
15
25
|
Fluent-plugin-groonga includes an input plugin and an output
|
16
26
|
plugin. Both of them are named `groonga`.
|
17
27
|
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
28
|
+
If you want to use fluent-plugin-groonga to store logs to Groonga, you
|
29
|
+
need to use only `groonga` output plugin.
|
30
|
+
|
31
|
+
The following configuration stores all data in `/var/log/messages`
|
32
|
+
into Groonga:
|
33
|
+
|
34
|
+
<source>
|
35
|
+
type tail
|
36
|
+
format syslog
|
37
|
+
path /var/log/syslog.1
|
38
|
+
pos_file /tmp/messages.pos
|
39
|
+
tag log.messages
|
40
|
+
read_from_head true
|
41
|
+
</source>
|
42
|
+
|
43
|
+
<match log.**>
|
44
|
+
type groonga
|
45
|
+
table logs
|
46
|
+
|
47
|
+
protocol http
|
48
|
+
host 127.0.0.1
|
49
|
+
|
50
|
+
buffer_type file
|
51
|
+
buffer_path /tmp/buffer
|
52
|
+
flush_interval 1
|
53
|
+
</match>
|
22
54
|
|
23
|
-
|
55
|
+
If you want to use fluent-plugin-groonga to implement Groonga
|
56
|
+
replication system, you need to use both plugins.
|
57
|
+
|
58
|
+
The input plugin provides Groonga compatible interface. It means that
|
59
|
+
HTTP and GQTP interface. You can use the input plugin as Groonga
|
60
|
+
server. The input plugin receives Groonga commands and sends them to
|
61
|
+
the output plugin through zero or more Fluentds.
|
62
|
+
|
63
|
+
The output plugin sends received Groonga commands to Groonga. The
|
24
64
|
output plugin supports all interfaces, HTTP, GQTP and command
|
25
65
|
interface.
|
26
66
|
|
@@ -32,6 +72,78 @@ You can replicate your data by using `copy` output plugin.
|
|
32
72
|
|
33
73
|
## Usage
|
34
74
|
|
75
|
+
There are two usages:
|
76
|
+
|
77
|
+
* Store logs collected by Fluentd to Groonga.
|
78
|
+
* Implement replication system for Groonga.
|
79
|
+
|
80
|
+
They are described in other sections.
|
81
|
+
|
82
|
+
### Store logs into Groonga
|
83
|
+
|
84
|
+
You need to use `groonga` output plugin to store logs into Groonga.
|
85
|
+
|
86
|
+
The output plugin has auto schema define feature. So you don't need to
|
87
|
+
define schema in Groonga before running Fluentd. You just run Groonga.
|
88
|
+
|
89
|
+
There is one required parameter:
|
90
|
+
|
91
|
+
* `table`: It specifies table name for storing logs.
|
92
|
+
|
93
|
+
Here is a minimum configuration:
|
94
|
+
|
95
|
+
<match log.**>
|
96
|
+
type groonga
|
97
|
+
table logs
|
98
|
+
</match>
|
99
|
+
|
100
|
+
The configuration stores logs into `logs` table in Groonga that runs
|
101
|
+
on `localhost`.
|
102
|
+
|
103
|
+
There are optional parameters:
|
104
|
+
|
105
|
+
* `protocol`: It specifies protocol to communicate Groonga server.
|
106
|
+
* Available values: `http`, `gqtp`, `command`
|
107
|
+
* Default: `http`
|
108
|
+
* `host`: It specifies host name or IP address of Groonga server.
|
109
|
+
* Default: `127.0.0.1`
|
110
|
+
* `port`: It specifies port number of Groonga server.
|
111
|
+
* Default for `http` protocol: `10041`
|
112
|
+
* Default for `gqtp` protocol: `10043`
|
113
|
+
|
114
|
+
Here is a configuration that specifies optional parameters explicitly:
|
115
|
+
|
116
|
+
<match log.**>
|
117
|
+
type groonga
|
118
|
+
table logs
|
119
|
+
|
120
|
+
protocol http
|
121
|
+
host 127.0.0.1
|
122
|
+
port 10041
|
123
|
+
</match>
|
124
|
+
|
125
|
+
`groonga` output plugin supports buffer. So you can use buffer related
|
126
|
+
parameters. See
|
127
|
+
[Buffer Plugin Overview | Fluentd](http://docs.fluentd.org/articles/buffer-plugin-overview)
|
128
|
+
for details.
|
129
|
+
|
130
|
+
Note that there is special tag name. You can't use
|
131
|
+
`groonga.command.XXX` tag name for this usage. It means that you can't
|
132
|
+
use the following configuration:
|
133
|
+
|
134
|
+
<match groonga.command.*>
|
135
|
+
type groonga
|
136
|
+
# ...
|
137
|
+
</match>
|
138
|
+
|
139
|
+
`groonga.command.XXX` tag name is reserved for implementing
|
140
|
+
replication system for Groonga.
|
141
|
+
|
142
|
+
### Implement replication system for Groonga
|
143
|
+
|
144
|
+
See the following documents how to implement replication system for
|
145
|
+
Groonga:
|
146
|
+
|
35
147
|
* [Configuration](doc/text/configuration.md)
|
36
148
|
([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.configuration.html))
|
37
149
|
* [Constitution](doc/text/constitution.md)
|
data/doc/text/news.md
CHANGED
@@ -2,6 +2,21 @@
|
|
2
2
|
|
3
3
|
# News
|
4
4
|
|
5
|
+
## 1.0.4: 2014-10-20
|
6
|
+
|
7
|
+
### Improvements
|
8
|
+
|
9
|
+
* Supported the latest http_parser gem.
|
10
|
+
* Removed no buffer mode. Use `flush_interval 0` for no buffer like
|
11
|
+
behavior.
|
12
|
+
* Changed the default port number to `10043` for `gqtp` protocol usage.
|
13
|
+
Because Groonga changed the default port number for `gqtp` protocol.
|
14
|
+
* Reduced the number of `load` calls. It improves `load` performance.
|
15
|
+
* Supported auto schema define. You don't need to define schema in Groonga
|
16
|
+
before running Fluentd.
|
17
|
+
* Added document to use fluent-plugin-groonga to store logs into Groonga.
|
18
|
+
It fits normal Fluentd usage.
|
19
|
+
|
5
20
|
## 1.0.3: 2013-09-29
|
6
21
|
|
7
22
|
### Improvements
|
@@ -1,6 +1,6 @@
|
|
1
1
|
# -*- mode: ruby; coding: utf-8 -*-
|
2
2
|
#
|
3
|
-
# Copyright (C) 2012-
|
3
|
+
# Copyright (C) 2012-2014 Kouhei Sutou <kou@clear-code.com>
|
4
4
|
#
|
5
5
|
# This library is free software; you can redistribute it and/or
|
6
6
|
# modify it under the terms of the GNU Lesser General Public
|
@@ -17,12 +17,12 @@
|
|
17
17
|
|
18
18
|
Gem::Specification.new do |spec|
|
19
19
|
spec.name = "fluent-plugin-groonga"
|
20
|
-
spec.version = "1.0.
|
20
|
+
spec.version = "1.0.4"
|
21
21
|
spec.authors = ["Kouhei Sutou"]
|
22
22
|
spec.email = ["kou@clear-code.com"]
|
23
|
-
spec.summary = "Fluentd plugin
|
23
|
+
spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
|
24
24
|
spec.description =
|
25
|
-
"
|
25
|
+
"There are two usages. 1) Store data into Groonga. 2) Implement Groonga replication system. See documentation for details."
|
26
26
|
spec.homepage = "https://github.com/groonga/fluent-plugin-groonga"
|
27
27
|
spec.license = "LGPL-2.1"
|
28
28
|
|
@@ -35,7 +35,7 @@ Gem::Specification.new do |spec|
|
|
35
35
|
spec.require_paths = ["lib"]
|
36
36
|
|
37
37
|
spec.add_runtime_dependency("fluentd")
|
38
|
-
spec.add_runtime_dependency("
|
38
|
+
spec.add_runtime_dependency("groonga-client")
|
39
39
|
spec.add_runtime_dependency("groonga-command-parser")
|
40
40
|
|
41
41
|
spec.add_development_dependency("rake")
|
@@ -1,6 +1,6 @@
|
|
1
1
|
# -*- coding: utf-8 -*-
|
2
2
|
#
|
3
|
-
# Copyright (C) 2012-
|
3
|
+
# Copyright (C) 2012-2014 Kouhei Sutou <kou@clear-code.com>
|
4
4
|
#
|
5
5
|
# This library is free software; you can redistribute it and/or
|
6
6
|
# modify it under the terms of the GNU Lesser General Public
|
@@ -16,6 +16,7 @@
|
|
16
16
|
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
17
17
|
|
18
18
|
require "English"
|
19
|
+
require "uri"
|
19
20
|
require "webrick/httputils"
|
20
21
|
|
21
22
|
require "http_parser"
|
@@ -31,19 +32,22 @@ module Fluent
|
|
31
32
|
super
|
32
33
|
end
|
33
34
|
|
34
|
-
config_param :protocol, :
|
35
|
+
config_param :protocol, :defalut => :http do |value|
|
36
|
+
case value
|
37
|
+
when "http", "gqtp"
|
38
|
+
value.to_sym
|
39
|
+
else
|
40
|
+
raise ConfigError, "must be http or gqtp: <#{value}>"
|
41
|
+
end
|
42
|
+
end
|
35
43
|
|
36
44
|
def configure(conf)
|
37
45
|
super
|
38
46
|
case @protocol
|
39
|
-
when
|
47
|
+
when :http
|
40
48
|
@input = HTTPInput.new
|
41
|
-
when
|
49
|
+
when :gqtp
|
42
50
|
@input = GQTPInput.new
|
43
|
-
else
|
44
|
-
message = "unknown protocol: <#{@protocol.inspect}>"
|
45
|
-
$log.error message
|
46
|
-
raise ConfigError, message
|
47
51
|
end
|
48
52
|
@input.configure(conf)
|
49
53
|
end
|
@@ -76,9 +80,9 @@ module Fluent
|
|
76
80
|
include DetachMultiProcessMixin
|
77
81
|
|
78
82
|
config_param :bind, :string, :default => "0.0.0.0"
|
79
|
-
config_param :port, :integer, :default =>
|
83
|
+
config_param :port, :integer, :default => nil
|
80
84
|
config_param :real_host, :string
|
81
|
-
config_param :real_port, :integer, :default =>
|
85
|
+
config_param :real_port, :integer, :default => nil
|
82
86
|
DEFAULT_EMIT_COMMANDS = [
|
83
87
|
/\Atable_/,
|
84
88
|
/\Acolumn_/,
|
@@ -102,6 +106,13 @@ module Fluent
|
|
102
106
|
end
|
103
107
|
end
|
104
108
|
|
109
|
+
def configure(conf)
|
110
|
+
super
|
111
|
+
|
112
|
+
@port ||= default_port
|
113
|
+
@real_port ||= default_port
|
114
|
+
end
|
115
|
+
|
105
116
|
def start
|
106
117
|
listen_socket = TCPServer.new(@bind, @port)
|
107
118
|
detach_multi_process do
|
@@ -156,6 +167,10 @@ module Fluent
|
|
156
167
|
|
157
168
|
class HTTPInput < BaseInput
|
158
169
|
private
|
170
|
+
def default_port
|
171
|
+
10041
|
172
|
+
end
|
173
|
+
|
159
174
|
def handler_class
|
160
175
|
Handler
|
161
176
|
end
|
@@ -188,8 +203,9 @@ module Fluent
|
|
188
203
|
end
|
189
204
|
|
190
205
|
def on_message_complete
|
191
|
-
|
192
|
-
|
206
|
+
uri = URI.parse(@parser.request_url)
|
207
|
+
params = WEBrick::HTTPUtils.parse_query(uri.query)
|
208
|
+
path_info = uri.path
|
193
209
|
case path_info
|
194
210
|
when /\A\/d\//
|
195
211
|
command = $POSTMATCH
|
@@ -204,6 +220,10 @@ module Fluent
|
|
204
220
|
|
205
221
|
class GQTPInput < BaseInput
|
206
222
|
private
|
223
|
+
def default_port
|
224
|
+
10043
|
225
|
+
end
|
226
|
+
|
207
227
|
def handler_class
|
208
228
|
Handler
|
209
229
|
end
|
@@ -1,6 +1,6 @@
|
|
1
1
|
# -*- coding: utf-8 -*-
|
2
2
|
#
|
3
|
-
# Copyright (C) 2012 Kouhei Sutou <kou@clear-code.com>
|
3
|
+
# Copyright (C) 2012-2014 Kouhei Sutou <kou@clear-code.com>
|
4
4
|
#
|
5
5
|
# This library is free software; you can redistribute it and/or
|
6
6
|
# modify it under the terms of the GNU Lesser General Public
|
@@ -17,23 +17,26 @@
|
|
17
17
|
|
18
18
|
require "fileutils"
|
19
19
|
|
20
|
+
require "yajl"
|
21
|
+
|
22
|
+
require "groonga/client"
|
23
|
+
|
20
24
|
module Fluent
|
21
|
-
class GroongaOutput <
|
25
|
+
class GroongaOutput < BufferedOutput
|
22
26
|
Plugin.register_output("groonga", self)
|
23
27
|
|
24
28
|
def initialize
|
25
29
|
super
|
26
30
|
end
|
27
31
|
|
28
|
-
|
29
|
-
|
30
|
-
|
32
|
+
config_param :protocol, :default => :http do |value|
|
33
|
+
case value
|
34
|
+
when "http", "gqtp", "command"
|
35
|
+
value.to_sym
|
31
36
|
else
|
32
|
-
|
37
|
+
raise ConfigError, "must be http, gqtp or command: <#{value}>"
|
33
38
|
end
|
34
39
|
end
|
35
|
-
|
36
|
-
config_param :protocol, :string, :default => "http"
|
37
40
|
config_param :table, :string, :default => nil
|
38
41
|
|
39
42
|
def configure(conf)
|
@@ -42,168 +45,318 @@ module Fluent
|
|
42
45
|
@client.configure(conf)
|
43
46
|
|
44
47
|
@emitter = Emitter.new(@client, @table)
|
45
|
-
@output = create_output(@buffer_type, @emitter)
|
46
|
-
@output.configure(conf)
|
47
48
|
end
|
48
49
|
|
49
50
|
def start
|
50
51
|
super
|
51
52
|
@client.start
|
52
|
-
@
|
53
|
+
@emitter.start
|
53
54
|
end
|
54
55
|
|
55
56
|
def shutdown
|
56
57
|
super
|
57
|
-
@
|
58
|
+
@emitter.shutdown
|
58
59
|
@client.shutdown
|
59
60
|
end
|
60
61
|
|
61
|
-
def
|
62
|
-
|
62
|
+
def format(tag, time, record)
|
63
|
+
[tag, time, record].to_msgpack
|
63
64
|
end
|
64
65
|
|
66
|
+
def write(chunk)
|
67
|
+
@emitter.emit(chunk)
|
68
|
+
end
|
69
|
+
|
70
|
+
private
|
65
71
|
def create_client(protocol)
|
66
72
|
case protocol
|
67
|
-
when
|
68
|
-
|
69
|
-
when
|
70
|
-
GQTPClient.new
|
71
|
-
when "command"
|
73
|
+
when :http, :gqtp
|
74
|
+
NetworkClient.new(protocol)
|
75
|
+
when :command
|
72
76
|
CommandClient.new
|
73
77
|
end
|
74
78
|
end
|
75
79
|
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
|
80
|
-
|
80
|
+
class Schema
|
81
|
+
def initialize(client, table_name)
|
82
|
+
@client = client
|
83
|
+
@table_name = table_name
|
84
|
+
@table = nil
|
85
|
+
@columns = nil
|
81
86
|
end
|
82
|
-
end
|
83
87
|
|
84
|
-
|
85
|
-
|
86
|
-
@client = client
|
87
|
-
@table = table
|
88
|
+
def populate
|
89
|
+
# TODO
|
88
90
|
end
|
89
91
|
|
90
|
-
def
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
92
|
+
def update(records)
|
93
|
+
ensure_table
|
94
|
+
ensure_columns
|
95
|
+
|
96
|
+
nonexistent_columns = {}
|
97
|
+
records.each do |record|
|
98
|
+
record.each do |key, value|
|
99
|
+
column = @columns[key]
|
100
|
+
if column.nil?
|
101
|
+
nonexistent_columns[key] ||= []
|
102
|
+
nonexistent_columns[key] << value
|
103
|
+
end
|
104
|
+
end
|
105
|
+
end
|
106
|
+
|
107
|
+
nonexistent_columns.each do |name, values|
|
108
|
+
@columns[name] = create_column(name, values)
|
96
109
|
end
|
97
110
|
end
|
98
111
|
|
99
112
|
private
|
100
|
-
def
|
101
|
-
|
102
|
-
|
103
|
-
@client.
|
113
|
+
def ensure_table
|
114
|
+
return if @table
|
115
|
+
|
116
|
+
table_list = @client.execute("table_list")
|
117
|
+
target_table = table_list.find do |table|
|
118
|
+
table.name == @table_name
|
119
|
+
end
|
120
|
+
if target_table
|
121
|
+
@table = Table.new(@table_name, target_table.domain)
|
122
|
+
else
|
123
|
+
# TODO: Check response
|
124
|
+
@client.execute("table_create",
|
125
|
+
"name" => @table_name,
|
126
|
+
"flags" => "TABLE_NO_KEY")
|
127
|
+
@table = Table.new(@table_name, nil)
|
128
|
+
end
|
104
129
|
end
|
105
130
|
|
106
|
-
def
|
107
|
-
return if @
|
131
|
+
def ensure_columns
|
132
|
+
return if @columns
|
108
133
|
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
|
134
|
+
column_list = @client.execute("column_list", "table" => @table_name)
|
135
|
+
@columns = {}
|
136
|
+
column_list.each do |column|
|
137
|
+
vector_p = column.flags.split("|").include?("COLUMN_VECTOR")
|
138
|
+
@columns[column.name] = Column.new(column.name,
|
139
|
+
column.range,
|
140
|
+
vector_p)
|
141
|
+
end
|
115
142
|
end
|
116
|
-
end
|
117
143
|
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
144
|
+
def create_column(name, sample_values)
|
145
|
+
guesser = TypeGuesser.new(sample_values)
|
146
|
+
value_type = guesser.guess
|
147
|
+
vector_p = guesser.vector?
|
148
|
+
if vector_p
|
149
|
+
flags = "COLUMN_VECTOR"
|
150
|
+
else
|
151
|
+
flags = "COLUMN_SCALAR"
|
152
|
+
end
|
153
|
+
# TODO: Check response
|
154
|
+
@client.execute("column_create",
|
155
|
+
"table" => @table_name,
|
156
|
+
"name" => name,
|
157
|
+
"flags" => flags,
|
158
|
+
"type" => value_type)
|
159
|
+
Column.new(name, value_type, vector_p)
|
122
160
|
end
|
123
161
|
|
124
|
-
|
125
|
-
|
126
|
-
@
|
162
|
+
class TypeGuesser
|
163
|
+
def initialize(sample_values)
|
164
|
+
@sample_values = sample_values
|
127
165
|
end
|
128
|
-
chain.next
|
129
|
-
end
|
130
|
-
end
|
131
166
|
|
132
|
-
|
133
|
-
|
134
|
-
|
135
|
-
|
167
|
+
def guess
|
168
|
+
return "Time" if time_values?
|
169
|
+
return "Int32" if int32_values?
|
170
|
+
return "Int64" if int64_values?
|
171
|
+
return "Float" if float_values?
|
172
|
+
return "WGS84GeoPoint" if geo_point_values?
|
173
|
+
|
174
|
+
"Text"
|
175
|
+
end
|
176
|
+
|
177
|
+
def vector?
|
178
|
+
@sample_values.any? do |sample_value|
|
179
|
+
sample_value.is_a?(Array)
|
180
|
+
end
|
181
|
+
end
|
182
|
+
|
183
|
+
private
|
184
|
+
def time_values?
|
185
|
+
now = Time.now.to_i
|
186
|
+
year_in_seconds = 365 * 24 * 60 * 60
|
187
|
+
window = 10 * year_in_seconds
|
188
|
+
new = now + window
|
189
|
+
old = now - window
|
190
|
+
recent_range = old..new
|
191
|
+
@sample_values.all? do |sample_value|
|
192
|
+
sample_value.is_a?(Integer) and
|
193
|
+
recent_range.cover?(sample_value)
|
194
|
+
end
|
195
|
+
end
|
196
|
+
|
197
|
+
def integer_value?(value)
|
198
|
+
case value
|
199
|
+
when String
|
200
|
+
begin
|
201
|
+
Integer(value)
|
202
|
+
true
|
203
|
+
rescue ArgumentError
|
204
|
+
false
|
205
|
+
end
|
206
|
+
when Integer
|
207
|
+
true
|
208
|
+
else
|
209
|
+
false
|
210
|
+
end
|
211
|
+
end
|
212
|
+
|
213
|
+
def int32_values?
|
214
|
+
int32_min = -(2 ** 31)
|
215
|
+
int32_max = 2 ** 31 - 1
|
216
|
+
range = int32_min..int32_max
|
217
|
+
@sample_values.all? do |sample_value|
|
218
|
+
integer_value?(sample_value) and
|
219
|
+
range.cover?(Integer(sample_value))
|
220
|
+
end
|
221
|
+
end
|
222
|
+
|
223
|
+
def int64_values?
|
224
|
+
@sample_values.all? do |sample_value|
|
225
|
+
integer_value?(sample_value)
|
226
|
+
end
|
227
|
+
end
|
228
|
+
|
229
|
+
def float_value?(value)
|
230
|
+
case value
|
231
|
+
when String
|
232
|
+
begin
|
233
|
+
Float(value)
|
234
|
+
true
|
235
|
+
rescue ArgumentError
|
236
|
+
false
|
237
|
+
end
|
238
|
+
when Float
|
239
|
+
true
|
240
|
+
else
|
241
|
+
false
|
242
|
+
end
|
243
|
+
end
|
244
|
+
|
245
|
+
def float_values?
|
246
|
+
@sample_values.all? do |sample_value|
|
247
|
+
float_value?(sample_value)
|
248
|
+
end
|
249
|
+
end
|
250
|
+
|
251
|
+
def geo_point_values?
|
252
|
+
@sample_values.all? do |sample_value|
|
253
|
+
sample_value.is_a?(String) and
|
254
|
+
/\A-?\d+(?:\.\d+)[,x]-?\d+(?:\.\d+)\z/ =~ sample_value
|
255
|
+
end
|
256
|
+
end
|
136
257
|
end
|
137
258
|
|
138
|
-
|
139
|
-
|
259
|
+
class Table
|
260
|
+
def initialize(name, key_type)
|
261
|
+
@name = name
|
262
|
+
@key_type = key_type
|
263
|
+
end
|
140
264
|
end
|
141
265
|
|
142
|
-
|
143
|
-
|
144
|
-
@
|
266
|
+
class Column
|
267
|
+
def initialize(name, value_type, vector_p)
|
268
|
+
@name = name
|
269
|
+
@value_type = value_type
|
270
|
+
@vector_p = vector_p
|
145
271
|
end
|
146
272
|
end
|
147
273
|
end
|
148
274
|
|
149
|
-
class
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
275
|
+
class Emitter
|
276
|
+
def initialize(client, table)
|
277
|
+
@client = client
|
278
|
+
@table = table
|
279
|
+
@schema = nil
|
280
|
+
end
|
154
281
|
|
155
282
|
def start
|
156
|
-
@
|
283
|
+
@schema = Schema.new(@client, @table)
|
157
284
|
end
|
158
285
|
|
159
286
|
def shutdown
|
160
287
|
end
|
161
288
|
|
162
|
-
def
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
289
|
+
def emit(chunk)
|
290
|
+
records = []
|
291
|
+
chunk.msgpack_each do |message|
|
292
|
+
tag, _, record = message
|
293
|
+
if /\Agroonga\.command\./ =~ tag
|
294
|
+
name = $POSTMATCH
|
295
|
+
unless records.empty?
|
296
|
+
store_records(records)
|
297
|
+
records.clear
|
298
|
+
end
|
299
|
+
@client.execute(name, record)
|
300
|
+
else
|
301
|
+
records << record
|
302
|
+
end
|
303
|
+
end
|
304
|
+
store_records(records) unless records.empty?
|
167
305
|
end
|
168
306
|
|
169
|
-
|
170
|
-
|
171
|
-
|
307
|
+
private
|
308
|
+
def store_records(records)
|
309
|
+
return if @table.nil?
|
310
|
+
|
311
|
+
@schema.update(records)
|
312
|
+
|
313
|
+
arguments = {
|
314
|
+
"table" => @table,
|
315
|
+
"values" => Yajl::Encoder.encode(records),
|
316
|
+
}
|
317
|
+
@client.execute("load", arguments)
|
318
|
+
end
|
319
|
+
end
|
320
|
+
|
321
|
+
class BaseClient
|
322
|
+
private
|
323
|
+
def build_command(name, arguments={})
|
324
|
+
command_class = Groonga::Command.find(name)
|
325
|
+
command_class.new(name, arguments)
|
172
326
|
end
|
173
327
|
end
|
174
328
|
|
175
|
-
class
|
329
|
+
class NetworkClient < BaseClient
|
176
330
|
include Configurable
|
177
331
|
|
178
|
-
config_param :host, :string, :default =>
|
179
|
-
config_param :port, :integer, :default =>
|
332
|
+
config_param :host, :string, :default => nil
|
333
|
+
config_param :port, :integer, :default => nil
|
334
|
+
|
335
|
+
def initialize(protocol)
|
336
|
+
super()
|
337
|
+
@protocol = protocol
|
338
|
+
end
|
180
339
|
|
181
340
|
def start
|
182
|
-
@loop = Coolio::Loop.new
|
183
341
|
@client = nil
|
184
342
|
end
|
185
343
|
|
186
344
|
def shutdown
|
187
345
|
return if @client.nil?
|
188
|
-
@client.close
|
189
|
-
@loop.stop
|
190
|
-
end
|
191
|
-
@loop.run
|
346
|
+
@client.close
|
192
347
|
end
|
193
348
|
|
194
|
-
def
|
195
|
-
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
200
|
-
|
201
|
-
end
|
202
|
-
@loop.run
|
349
|
+
def execute(name, arguments={})
|
350
|
+
command = build_command(name, arguments)
|
351
|
+
@client ||= Groonga::Client.new(:protocol => @protocol,
|
352
|
+
:host => @host,
|
353
|
+
:port => @port,
|
354
|
+
:backend => :synchronous)
|
355
|
+
@client.execute(command)
|
203
356
|
end
|
204
357
|
end
|
205
358
|
|
206
|
-
class CommandClient
|
359
|
+
class CommandClient < BaseClient
|
207
360
|
include Configurable
|
208
361
|
|
209
362
|
config_param :groonga, :string, :default => "groonga"
|
@@ -222,42 +375,45 @@ module Fluent
|
|
222
375
|
|
223
376
|
def start
|
224
377
|
run_groonga
|
225
|
-
wrap_io
|
226
378
|
end
|
227
379
|
|
228
380
|
def shutdown
|
229
|
-
@
|
230
|
-
|
231
|
-
@
|
381
|
+
@input.close
|
382
|
+
read_output("shutdown")
|
383
|
+
@output.close
|
384
|
+
@error.close
|
232
385
|
Process.waitpid(@pid)
|
233
386
|
end
|
234
387
|
|
235
|
-
def
|
388
|
+
def execute(name, arguments={})
|
389
|
+
command = build_command(name, arguments)
|
236
390
|
body = nil
|
237
391
|
if command.name == "load"
|
238
392
|
body = command.arguments.delete(:values)
|
239
393
|
end
|
240
|
-
|
394
|
+
uri = command.to_uri_format
|
395
|
+
@input.write("#{uri}\n")
|
241
396
|
if body
|
242
397
|
body.each_line do |line|
|
243
|
-
@
|
398
|
+
@input.write("#{line}\n")
|
244
399
|
end
|
245
400
|
end
|
246
|
-
@
|
401
|
+
@input.flush
|
402
|
+
read_output(uri)
|
247
403
|
end
|
248
404
|
|
249
405
|
private
|
250
406
|
def run_groonga
|
251
407
|
env = {}
|
252
|
-
|
253
|
-
|
254
|
-
|
255
|
-
input_fd =
|
256
|
-
output_fd =
|
408
|
+
input = IO.pipe("ASCII-8BIT")
|
409
|
+
output = IO.pipe("ASCII-8BIT")
|
410
|
+
error = IO.pipe("ASCII-8BIT")
|
411
|
+
input_fd = input[0].to_i
|
412
|
+
output_fd = output[1].to_i
|
257
413
|
options = {
|
258
414
|
input_fd => input_fd,
|
259
415
|
output_fd => output_fd,
|
260
|
-
:err =>
|
416
|
+
:err => error[1],
|
261
417
|
}
|
262
418
|
arguments = @arguments
|
263
419
|
arguments += [
|
@@ -270,27 +426,42 @@ module Fluent
|
|
270
426
|
end
|
271
427
|
arguments << @database
|
272
428
|
@pid = spawn(env, @groonga, *arguments, options)
|
273
|
-
|
274
|
-
@
|
275
|
-
|
429
|
+
input[0].close
|
430
|
+
@input = input[1]
|
431
|
+
output[1].close
|
432
|
+
@output = output[0]
|
433
|
+
error[1].close
|
434
|
+
@error = error[0]
|
276
435
|
end
|
277
436
|
|
278
|
-
def
|
279
|
-
|
437
|
+
def read_output(context)
|
438
|
+
output_message = ""
|
439
|
+
error_message = ""
|
440
|
+
|
441
|
+
loop do
|
442
|
+
readables = IO.select([@output, @error], nil, nil, 0)
|
443
|
+
break if readables.nil?
|
444
|
+
|
445
|
+
readables.each do |readable|
|
446
|
+
case readable
|
447
|
+
when @output
|
448
|
+
output_message << @output.gets
|
449
|
+
when @error
|
450
|
+
error_message << @error.gets
|
451
|
+
end
|
452
|
+
end
|
453
|
+
end
|
280
454
|
|
281
|
-
|
282
|
-
|
283
|
-
|
455
|
+
unless output_message.empty?
|
456
|
+
Engine.log.debug("[output][groonga][output]",
|
457
|
+
:context => context,
|
458
|
+
:message => output_message)
|
284
459
|
end
|
285
|
-
|
286
|
-
|
460
|
+
unless error_message.empty?
|
461
|
+
Engine.log.error("[output][groonga][error]",
|
462
|
+
:context => context,
|
463
|
+
:message => error_message)
|
287
464
|
end
|
288
|
-
@groonga_output = Coolio::IO.new(@output[0])
|
289
|
-
@groonga_error = Coolio::IO.new(@error[0])
|
290
|
-
|
291
|
-
@loop.attach(@groonga_input)
|
292
|
-
@loop.attach(@groonga_output)
|
293
|
-
@loop.attach(@groonga_error)
|
294
465
|
end
|
295
466
|
end
|
296
467
|
end
|