fluent-plugin-groonga 1.1.0 → 1.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +24 -13
- data/doc/text/configuration.md +55 -30
- data/doc/text/constitution.md +83 -83
- data/doc/text/news.md +11 -0
- data/fluent-plugin-groonga.gemspec +2 -2
- data/lib/fluent/plugin/in_groonga.rb +11 -3
- data/sample/command.conf +2 -2
- data/sample/gqtp.conf +2 -2
- data/sample/http.conf +2 -2
- data/sample/store-apache.conf +5 -5
- data/sample/store-syslog.conf +6 -5
- data/sample/store.conf +2 -2
- metadata +3 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 409393e5f37e320bd7236591e119c552ddcc6490
|
4
|
+
data.tar.gz: f7070f11fb9c6126ca5ddd07786e510e6209cf6a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: dd76ea51915f8f84d6f2d5cfc2245fb02e05cee1eedcba341142ac020f43d72db488ac87aac1e40a7dc0d98674bb45124bb0c541e7f010a7dd0478ae857ae785
|
7
|
+
data.tar.gz: 7c29cbb8d38341cf3d36e0d5ff61626baf3e1a16aea1bef71a0ba5146495c6300e70936c3d62a208028cf9bf06766342d8c09aae2489a3cb7ff05fda18032947
|
data/README.md
CHANGED
@@ -13,6 +13,7 @@ Fluent-plugin-groonga is a Fluentd plugin collection to use
|
|
13
13
|
supports the following two usages:
|
14
14
|
|
15
15
|
* Store logs collected by Fluentd to Groonga.
|
16
|
+
|
16
17
|
* Implement replication system for Groonga.
|
17
18
|
|
18
19
|
The first usage is normal usage. You can store logs to Groonga and
|
@@ -32,7 +33,7 @@ The following configuration stores all data in `/var/log/messages`
|
|
32
33
|
into Groonga:
|
33
34
|
|
34
35
|
<source>
|
35
|
-
type tail
|
36
|
+
@type tail
|
36
37
|
format syslog
|
37
38
|
path /var/log/syslog.1
|
38
39
|
pos_file /tmp/messages.pos
|
@@ -41,7 +42,7 @@ into Groonga:
|
|
41
42
|
</source>
|
42
43
|
|
43
44
|
<match log.**>
|
44
|
-
type groonga
|
45
|
+
@type groonga
|
45
46
|
store_table logs
|
46
47
|
|
47
48
|
protocol http
|
@@ -75,6 +76,7 @@ You can replicate your data by using `copy` output plugin.
|
|
75
76
|
There are two usages:
|
76
77
|
|
77
78
|
* Store logs collected by Fluentd to Groonga.
|
79
|
+
|
78
80
|
* Implement replication system for Groonga.
|
79
81
|
|
80
82
|
They are described in other sections.
|
@@ -93,7 +95,7 @@ There is one required parameter:
|
|
93
95
|
Here is a minimum configuration:
|
94
96
|
|
95
97
|
<match log.**>
|
96
|
-
type groonga
|
98
|
+
@type groonga
|
97
99
|
store_table logs
|
98
100
|
</match>
|
99
101
|
|
@@ -103,18 +105,25 @@ on `localhost`.
|
|
103
105
|
There are optional parameters:
|
104
106
|
|
105
107
|
* `protocol`: It specifies protocol to communicate Groonga server.
|
108
|
+
|
106
109
|
* Available values: `http`, `gqtp`, `command`
|
110
|
+
|
107
111
|
* Default: `http`
|
112
|
+
|
108
113
|
* `host`: It specifies host name or IP address of Groonga server.
|
114
|
+
|
109
115
|
* Default: `127.0.0.1`
|
116
|
+
|
110
117
|
* `port`: It specifies port number of Groonga server.
|
118
|
+
|
111
119
|
* Default for `http` protocol: `10041`
|
120
|
+
|
112
121
|
* Default for `gqtp` protocol: `10043`
|
113
122
|
|
114
123
|
Here is a configuration that specifies optional parameters explicitly:
|
115
124
|
|
116
125
|
<match log.**>
|
117
|
-
type groonga
|
126
|
+
@type groonga
|
118
127
|
store_table logs
|
119
128
|
|
120
129
|
protocol http
|
@@ -132,7 +141,7 @@ Note that there is special tag name. You can't use
|
|
132
141
|
use the following configuration:
|
133
142
|
|
134
143
|
<match groonga.command.*>
|
135
|
-
type groonga
|
144
|
+
@type groonga
|
136
145
|
# ...
|
137
146
|
</match>
|
138
147
|
|
@@ -144,14 +153,15 @@ replication system for Groonga.
|
|
144
153
|
See the following documents how to implement replication system for
|
145
154
|
Groonga:
|
146
155
|
|
147
|
-
* [Configuration](doc/text/configuration.md)
|
148
|
-
|
149
|
-
|
150
|
-
|
156
|
+
* [Configuration](doc/text/configuration.md)
|
157
|
+
([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.configuration.html))
|
158
|
+
|
159
|
+
* [Constitution](doc/text/constitution.md)
|
160
|
+
([on the Web](http://groonga.org/fluent-plugin-groonga/en/file.constitution.html))
|
151
161
|
|
152
162
|
## Authors
|
153
163
|
|
154
|
-
* Kouhei Sutou `<kou@clear-code.com>`
|
164
|
+
* Kouhei Sutou `<kou@clear-code.com>`
|
155
165
|
|
156
166
|
## License
|
157
167
|
|
@@ -162,8 +172,9 @@ contributed patches.)
|
|
162
172
|
|
163
173
|
## Mailing list
|
164
174
|
|
165
|
-
* English: [groonga-talk](https://lists.sourceforge.net/lists/listinfo/groonga-talk)
|
166
|
-
|
175
|
+
* English: [groonga-talk](https://lists.sourceforge.net/lists/listinfo/groonga-talk)
|
176
|
+
|
177
|
+
* Japanese: [groonga-dev](http://lists.sourceforge.jp/mailman/listinfo/groonga-dev)
|
167
178
|
|
168
179
|
## Source
|
169
180
|
|
@@ -172,4 +183,4 @@ The repository for fluent-plugin-groonga is on
|
|
172
183
|
|
173
184
|
## Thanks
|
174
185
|
|
175
|
-
* ...
|
186
|
+
* ...
|
data/doc/text/configuration.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
# Configuration
|
4
4
|
|
5
|
-
Fluent-plugin-groonga includes two
|
5
|
+
Fluent-plugin-groonga includes two Fluentd plugins. They are the
|
6
6
|
`groonga` input plugin and the `groonga` output plugin. This documents
|
7
7
|
describes configuration parameters of them.
|
8
8
|
|
@@ -10,23 +10,34 @@ describes configuration parameters of them.
|
|
10
10
|
|
11
11
|
Here are available parameters:
|
12
12
|
|
13
|
-
* `protocol`: It specifies protocol for receiving
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
13
|
+
* `protocol`: It specifies protocol for receiving Groonga commands.
|
14
|
+
|
15
|
+
* available values: `http` and `gqtp`
|
16
|
+
|
17
|
+
* default: `http`
|
18
|
+
|
19
|
+
* `bind`: It specifies bind address.
|
20
|
+
|
21
|
+
* default: `0.0.0.0`
|
22
|
+
|
23
|
+
* `port`: It specifies port number.
|
24
|
+
|
25
|
+
* default: `10041`
|
26
|
+
|
27
|
+
* `real_host`: It specifies real Groonga server's address. It is required.
|
28
|
+
|
29
|
+
* default: no default.
|
30
|
+
|
31
|
+
* `real_port`: It specifies real Groonga server's port number.
|
32
|
+
|
33
|
+
* default: `10041`
|
34
|
+
|
35
|
+
* `emit_commands`: TODO
|
25
36
|
|
26
37
|
Here is an example:
|
27
38
|
|
28
39
|
<source>
|
29
|
-
type groonga
|
40
|
+
@type groonga
|
30
41
|
protocol http
|
31
42
|
bind 127.0.0.1
|
32
43
|
port 10041
|
@@ -36,26 +47,40 @@ Here is an example:
|
|
36
47
|
|
37
48
|
## The `groonga` output plugin
|
38
49
|
|
39
|
-
* `protocol`: It specifies protocol for sending
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
* `
|
46
|
-
|
47
|
-
*
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
50
|
+
* `protocol`: It specifies protocol for sending Groonga commands to Groonga.
|
51
|
+
|
52
|
+
* available values: `http`, `gqtp` and `command`
|
53
|
+
|
54
|
+
* default: `http`
|
55
|
+
|
56
|
+
* For `http` and `gqtp` use:
|
57
|
+
|
58
|
+
* `host`: It specifies Groonga server's address.
|
59
|
+
|
60
|
+
* default: `localhost`
|
61
|
+
|
62
|
+
* `port`: It specifies Groonga server's port number.
|
63
|
+
|
64
|
+
* default: `10041`
|
65
|
+
|
66
|
+
* For `command` use:
|
67
|
+
|
68
|
+
* `groonga`: It specifies path of `groonga` command.
|
69
|
+
|
70
|
+
* default: `groonga`
|
71
|
+
|
72
|
+
* `database`: It specifies path of Groonga database. It is required.
|
73
|
+
|
74
|
+
* default: no default.
|
75
|
+
|
76
|
+
* `arguments`: It specifies additional arguments for `groonga` command.
|
77
|
+
|
78
|
+
* default: no additional arguments.
|
54
79
|
|
55
80
|
Here is an example:
|
56
81
|
|
57
82
|
<match groonga.command.*>
|
58
|
-
type groonga
|
83
|
+
@type groonga
|
59
84
|
protocol command
|
60
85
|
database /tmp/groonga/db
|
61
86
|
</match>
|
data/doc/text/constitution.md
CHANGED
@@ -3,7 +3,7 @@
|
|
3
3
|
# Constitution
|
4
4
|
|
5
5
|
You can chose some system constitutions to implemented replication
|
6
|
-
ready
|
6
|
+
ready Groonga system. This document describes some patterns.
|
7
7
|
|
8
8
|
Here are available patterns:
|
9
9
|
|
@@ -25,15 +25,15 @@ configure your system constitution.
|
|
25
25
|
|
26
26
|
### Small system
|
27
27
|
|
28
|
-
In small system, you just has two servers. One is the master
|
29
|
-
server and the other is the slave
|
28
|
+
In small system, you just has two servers. One is the master Groonga
|
29
|
+
server and the other is the slave Groonga server. You send all update
|
30
30
|
commands (e.g. `table_create`, `column_create`, `load` and so on.) to
|
31
31
|
fluentd. In fluentd, the `groonga` input plugin receives commands from
|
32
|
-
client, passes through them to master
|
33
|
-
through responses from master
|
32
|
+
client, passes through them to master Groonga server and passes
|
33
|
+
through responses from master Groonga server to client. The `groonga`
|
34
34
|
input plugin converts update commands to fluent messages when the
|
35
35
|
`groonga` input plugin passes through comamands and responses. The
|
36
|
-
fluent messages are sent to slave
|
36
|
+
fluent messages are sent to slave Groonga server by the `groonga`
|
37
37
|
output plugin.
|
38
38
|
|
39
39
|
Here is a diagram of this constitution.
|
@@ -42,56 +42,56 @@ Here is a diagram of this constitution.
|
|
42
42
|
and and
|
43
43
|
search +---------+ search +---------+
|
44
44
|
+--------+ <------> | fluentd | <------> | master |
|
45
|
-
| | +---------+ |
|
45
|
+
| | +---------+ | Groonga |
|
46
46
|
| client | update | +---------+
|
47
47
|
| | \_/
|
48
48
|
| | search +---------+
|
49
49
|
+--------+ <------> | slave |
|
50
|
-
|
|
50
|
+
| Groonga |
|
51
51
|
+---------+
|
52
52
|
|
53
|
-
Fluentd should be placed at client or master
|
53
|
+
Fluentd should be placed at client or master Groonga server. If you
|
54
54
|
have only one client that updates data, client side is reasonable. If
|
55
|
-
you have multiple clients that update data, master
|
55
|
+
you have multiple clients that update data, master Groonga server side
|
56
56
|
is reasonable.
|
57
57
|
|
58
58
|
You can use replication for high performance by providing search
|
59
59
|
service with multi servers. You can't use replication for high
|
60
|
-
availability. If master
|
61
|
-
can't update data. (Searching is still available because
|
60
|
+
availability. If master Groonga server or fluentd is down, this system
|
61
|
+
can't update data. (Searching is still available because slave Groonga
|
62
62
|
server is alive.)
|
63
63
|
|
64
64
|
Here is an example configuration file:
|
65
65
|
|
66
|
-
# For master
|
66
|
+
# For master Groonga server
|
67
67
|
<source>
|
68
|
-
type groonga
|
68
|
+
@type groonga
|
69
69
|
protocol gqtp # Or use the below line
|
70
70
|
# protocol http
|
71
|
-
bind 127.0.0.1 # For client side
|
72
|
-
# bind 192.168.0.1 # For master
|
71
|
+
bind 127.0.0.1 # For client side Fluentd
|
72
|
+
# bind 192.168.0.1 # For master Groonga server side Fluentd
|
73
73
|
port 10041
|
74
|
-
real_host 192.168.29.1 # IP address of master
|
75
|
-
real_port 10041 # Port number of master
|
74
|
+
real_host 192.168.29.1 # IP address of master Groonga server
|
75
|
+
real_port 10041 # Port number of master Groonga server
|
76
76
|
# real_port 20041 # Use different port number
|
77
|
-
# for master
|
77
|
+
# for master Groonga server side Fluentd
|
78
78
|
</source>
|
79
79
|
|
80
|
-
# For slave
|
80
|
+
# For slave Groonga server
|
81
81
|
<match groonga.command.*>
|
82
|
-
type groonga
|
82
|
+
@type groonga
|
83
83
|
protocol gqtp # Or use the below line
|
84
84
|
# protocol http # You can use different protocol for
|
85
|
-
# master
|
86
|
-
host 192.168.29.29 # IP address of slave
|
87
|
-
port 10041 # Port number of slave
|
85
|
+
# master Groonga server and slave Groonga server
|
86
|
+
host 192.168.29.29 # IP address of slave Groonga server
|
87
|
+
port 10041 # Port number of slave Groonga server
|
88
88
|
|
89
89
|
# Buffer
|
90
90
|
flush_interval 1s # Use small value for less delay replication
|
91
91
|
|
92
92
|
## Use the following configurations to support resending data to
|
93
|
-
## recovered slave
|
94
|
-
##
|
93
|
+
## recovered slave Groonga server. If you don't care about slave
|
94
|
+
## Groonga server is down case, you don't need the following
|
95
95
|
## configuration.
|
96
96
|
|
97
97
|
## For supporting resending data after fluentd is restarted
|
@@ -101,7 +101,7 @@ Here is an example configuration file:
|
|
101
101
|
## A value in load command is a chunk.
|
102
102
|
# buffer_chunk_limit 256m
|
103
103
|
## Use large value if you want to support resending data after
|
104
|
-
## slave
|
104
|
+
## slave Groonga server is down long time.
|
105
105
|
## 17: about 1.5day =
|
106
106
|
## ((2 ** 0) + (2 ** 1) + ... + (2 ** 17)) / 60.0 / 60.0 / 24.0
|
107
107
|
## (default)
|
@@ -121,53 +121,53 @@ commands.
|
|
121
121
|
|
122
122
|
You cannot update data until fluentd is up.
|
123
123
|
|
124
|
-
#### How to recover from master
|
124
|
+
#### How to recover from master Groonga server down
|
125
125
|
|
126
|
-
Here are recover steps when master
|
126
|
+
Here are recover steps when master Groonga server is down:
|
127
127
|
|
128
128
|
1. Stop fluentd.
|
129
129
|
2. Run `grndump /PATH/TO/SLAVE/GROONGA/SERVER/DB >
|
130
|
-
SLAVE_GROONGA_DUMP.grn` on slave
|
130
|
+
SLAVE_GROONGA_DUMP.grn` on slave Groonga server host.
|
131
131
|
3. Run `groonga -n /PATH/TO/MASTER/GROONGA/SERVER/DB <
|
132
|
-
SLAVE_GROONGA_DUMP.grn` on master
|
133
|
-
4. Run master
|
132
|
+
SLAVE_GROONGA_DUMP.grn` on master Groonga server.
|
133
|
+
4. Run master Groonga server.
|
134
134
|
5. Start fluentd.
|
135
135
|
|
136
136
|
You cannot update data until you finish to recover.
|
137
137
|
|
138
|
-
#### How to recover from slave
|
138
|
+
#### How to recover from slave Groonga server down
|
139
139
|
|
140
|
-
Here are recover steps when slave
|
140
|
+
Here are recover steps when slave Groonga server is down:
|
141
141
|
|
142
142
|
1. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
|
143
|
-
MASTER_GROONGA_DUMP.grn` on master
|
143
|
+
MASTER_GROONGA_DUMP.grn` on master Groonga server host.
|
144
144
|
2. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
|
145
|
-
MASTER_GROONGA_DUMP.grn` on slave
|
146
|
-
3. Run slave
|
145
|
+
MASTER_GROONGA_DUMP.grn` on slave Groonga server.
|
146
|
+
3. Run slave Groonga server.
|
147
147
|
|
148
148
|
You can update data while you recover. If your system can't process
|
149
|
-
all search requests by only master
|
149
|
+
all search requests by only master Groonga server, your system will be
|
150
150
|
down.
|
151
151
|
|
152
|
-
You need to recover slave
|
152
|
+
You need to recover slave Groonga server before fluentd's buffer queue
|
153
153
|
is full (see `buffer_queue_limit`) or fluentd gives up retrying (see
|
154
154
|
`retry_limit`). Here are recover steps when you cannot recover slave
|
155
|
-
|
155
|
+
Groonga server before those situations:
|
156
156
|
|
157
157
|
1. Stop fluentd.
|
158
158
|
2. Run `grndump /PATH/TO/MASTER/GROONGA/SERVER/DB >
|
159
|
-
MASTER_GROONGA_DUMP.grn` on master
|
159
|
+
MASTER_GROONGA_DUMP.grn` on master Groonga server host.
|
160
160
|
3. Run `groonga -n /PATH/TO/SLAVE/GROONGA/SERVER/DB <
|
161
|
-
MASTER_GROONGA_DUMP.grn` on slave
|
162
|
-
4. Run slave
|
161
|
+
MASTER_GROONGA_DUMP.grn` on slave Groonga server host.
|
162
|
+
4. Run slave Groonga server.
|
163
163
|
5. Start fluentd.
|
164
164
|
|
165
165
|
You cannot update data until you finish to recover.
|
166
166
|
|
167
167
|
### Medium system
|
168
168
|
|
169
|
-
In medium system, you have three or more slave
|
170
|
-
updates two or more slave
|
169
|
+
In medium system, you have three or more slave Groonga servers. Fluentd
|
170
|
+
updates two or more slave Groonga servers with the `copy` output
|
171
171
|
plugin and the `groonga` output plugin.
|
172
172
|
|
173
173
|
Here is a diagram of this constitution.
|
@@ -176,64 +176,64 @@ Here is a diagram of this constitution.
|
|
176
176
|
and and
|
177
177
|
search +---------+ search +---------+
|
178
178
|
+--------+ <------> | fluentd | <------> | master |
|
179
|
-
| | +---------+ |
|
179
|
+
| | +---------+ | Groonga |
|
180
180
|
| client | +--------+ +---------+
|
181
181
|
| | |
|
182
182
|
+--------+ search +---------+ |
|
183
183
|
| | <------> | slave | <-+ update
|
184
|
-
| client | |
|
184
|
+
| client | | Groonga | |
|
185
185
|
| | +---------+ |
|
186
186
|
+--------+ search +---------+ |
|
187
187
|
| | <------> | slave | <-+ update
|
188
|
-
| client | |
|
188
|
+
| client | | Groonga | |
|
189
189
|
| | +---------+ |
|
190
190
|
+- ... -+ ... ... ...
|
191
191
|
|
192
192
|
Here is an example configuration file:
|
193
193
|
|
194
|
-
# For master
|
194
|
+
# For master Groonga server
|
195
195
|
<source>
|
196
|
-
type groonga
|
196
|
+
@type groonga
|
197
197
|
protocol gqtp # Or use the below line
|
198
198
|
# protocol http
|
199
|
-
bind 127.0.0.1 # For client side
|
200
|
-
# bind 192.168.0.1 # For master
|
199
|
+
bind 127.0.0.1 # For client side Fluentd
|
200
|
+
# bind 192.168.0.1 # For master Groonga server side Fluentd
|
201
201
|
port 10041
|
202
|
-
real_host 192.168.29.1 # IP address of master
|
203
|
-
real_port 10041 # Port number of master
|
202
|
+
real_host 192.168.29.1 # IP address of master Groonga server
|
203
|
+
real_port 10041 # Port number of master Groonga server
|
204
204
|
# real_port 20041 # Use different port number
|
205
|
-
# for master
|
205
|
+
# for master Groonga server side fluentd
|
206
206
|
</source>
|
207
207
|
|
208
|
-
# For slave
|
208
|
+
# For slave Groonga servers
|
209
209
|
<match groonga.command.*>
|
210
|
-
type copy
|
210
|
+
@type copy
|
211
211
|
|
212
|
-
# The first slave
|
212
|
+
# The first slave Groonga server
|
213
213
|
<store>
|
214
|
-
type groonga
|
214
|
+
@type groonga
|
215
215
|
protocol gqtp # Or use the below line
|
216
216
|
# protocol http # You can use different protocol for
|
217
|
-
# master
|
218
|
-
host 192.168.29.2 # IP address of slave
|
219
|
-
port 10041 # Port number of slave
|
217
|
+
# master Groonga server and slave Groonga server
|
218
|
+
host 192.168.29.2 # IP address of slave Groonga server
|
219
|
+
port 10041 # Port number of slave Groonga server
|
220
220
|
|
221
221
|
# Buffer
|
222
222
|
flush_interval 1s # Use small value for less delay replication
|
223
223
|
|
224
224
|
## Use the following configurations to support resending data to
|
225
|
-
## recovered slave
|
226
|
-
##
|
225
|
+
## recovered slave Groonga server. If you don't care about slave
|
226
|
+
## Groonga server is down case, you don't need the following
|
227
227
|
## configuration.
|
228
228
|
|
229
229
|
## For supporting resending data after fluentd is restarted
|
230
230
|
# buffer_type file
|
231
|
-
# buffer_path /var/log/fluent/
|
231
|
+
# buffer_path /var/log/fluent/groonga1.*.buffer
|
232
232
|
## Use large value if a record has many data in load command.
|
233
233
|
## A value in load command is a chunk.
|
234
234
|
# buffer_chunk_limit 256m
|
235
235
|
## Use large value if you want to support resending data after
|
236
|
-
## slave
|
236
|
+
## slave Groonga server is down long time.
|
237
237
|
## 17: about 1.5day =
|
238
238
|
## ((2 ** 0) + (2 ** 1) + ... + (2 ** 17)) / 60.0 / 60.0 / 24.0
|
239
239
|
## (default)
|
@@ -245,22 +245,22 @@ Here is an example configuration file:
|
|
245
245
|
# buffer_queue_limit 10000
|
246
246
|
</store>
|
247
247
|
|
248
|
-
# The second slave
|
248
|
+
# The second slave Groonga server
|
249
249
|
<store>
|
250
|
-
type groonga
|
250
|
+
@type groonga
|
251
251
|
protocol gqtp # Or use the below line
|
252
252
|
# protocol http # You can use different protocol for
|
253
|
-
# master
|
254
|
-
host 192.168.29.3 # IP address of slave
|
255
|
-
port 10041 # Port number of slave
|
253
|
+
# master Groonga server and slave Groonga server
|
254
|
+
host 192.168.29.3 # IP address of slave Groonga server
|
255
|
+
port 10041 # Port number of slave Groonga server
|
256
256
|
|
257
257
|
# Buffer
|
258
258
|
# ...
|
259
259
|
</store>
|
260
260
|
|
261
|
-
# More slave
|
261
|
+
# More slave Groonga servers
|
262
262
|
# <store>
|
263
|
-
# type groonga
|
263
|
+
# @type groonga
|
264
264
|
# ...
|
265
265
|
# </store>
|
266
266
|
</match>
|
@@ -269,12 +269,12 @@ TODO: ...
|
|
269
269
|
|
270
270
|
### Large system
|
271
271
|
|
272
|
-
In large system, you have two or more slave
|
273
|
-
Fluentd that connects with master
|
274
|
-
fluentds that are in slave
|
272
|
+
In large system, you have two or more slave Groonga server clusters.
|
273
|
+
Fluentd that connects with master Groonga server updates two or more
|
274
|
+
fluentds that are in slave Groonga server clusters with the `copy`
|
275
275
|
output plugin and the `forward` output plugin. A slave cluster has a
|
276
|
-
fluentd. Fluentd in slave
|
277
|
-
|
276
|
+
fluentd. Fluentd in slave Groonga server clusters updates slave
|
277
|
+
Groonga server in the same slave Groonga server cluster by the `copy`
|
278
278
|
output plugin and `groonga` output plugin.
|
279
279
|
|
280
280
|
Here is a diagram of this constitution.
|
@@ -283,18 +283,18 @@ Here is a diagram of this constitution.
|
|
283
283
|
and and
|
284
284
|
search +---------+ search +---------+
|
285
285
|
+--------+ <--------> | fluentd | <------> | master |
|
286
|
-
| | +---------+ |
|
286
|
+
| | +---------+ | Groonga |
|
287
287
|
| client | | +---------+
|
288
288
|
| | +------------------------------+
|
289
289
|
+--------+ +----------------------------------+ |
|
290
290
|
| | | slave cluster | |
|
291
291
|
| client | search | +---------+ update +---------+ | |
|
292
292
|
| | <------> | | slave | <------- | fluentd | <-+ update
|
293
|
-
+--------| | |
|
293
|
+
+--------| | | Groonga | +---------+ | |
|
294
294
|
| | | +---------+ +-----------+ | |
|
295
295
|
| client | search | +---------+ | | |
|
296
296
|
| | <------> | | slave | <-+ update | |
|
297
|
-
+--------| | |
|
297
|
+
+--------| | | Groonga | | | |
|
298
298
|
| | | +---------+ | | |
|
299
299
|
| ... | ... | ... ... | |
|
300
300
|
+----------------------------------+ |
|
@@ -302,11 +302,11 @@ Here is a diagram of this constitution.
|
|
302
302
|
| | | slave cluster | |
|
303
303
|
| client | search | +---------+ update +---------+ | |
|
304
304
|
| | <------> | | slave | <------- | fluentd | <-+ update
|
305
|
-
+--------| | |
|
305
|
+
+--------| | | Groonga | +---------+ | |
|
306
306
|
| | | +---------+ +-----------+ | |
|
307
307
|
| client | search | +---------+ | | |
|
308
308
|
| | <------> | | slave | <-+ update | |
|
309
|
-
+--------| | |
|
309
|
+
+--------| | | Groonga | | | |
|
310
310
|
| | | +---------+ | | |
|
311
311
|
| ... | ... | ... ... | |
|
312
312
|
+----------------------------------+ |
|
data/doc/text/news.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# -*- mode: ruby; coding: utf-8 -*-
|
2
2
|
#
|
3
|
-
# Copyright (C) 2012-
|
3
|
+
# Copyright (C) 2012-2016 Kouhei Sutou <kou@clear-code.com>
|
4
4
|
#
|
5
5
|
# This library is free software; you can redistribute it and/or
|
6
6
|
# modify it under the terms of the GNU Lesser General Public
|
@@ -17,7 +17,7 @@
|
|
17
17
|
|
18
18
|
Gem::Specification.new do |spec|
|
19
19
|
spec.name = "fluent-plugin-groonga"
|
20
|
-
spec.version = "1.1.
|
20
|
+
spec.version = "1.1.1"
|
21
21
|
spec.authors = ["Kouhei Sutou"]
|
22
22
|
spec.email = ["kou@clear-code.com"]
|
23
23
|
spec.summary = "Fluentd plugin to store data into Groonga and implement Groonga replication system."
|
@@ -24,6 +24,8 @@ require "http_parser"
|
|
24
24
|
require "gqtp"
|
25
25
|
require "groonga/command/parser"
|
26
26
|
|
27
|
+
require "fluent/process"
|
28
|
+
|
27
29
|
module Fluent
|
28
30
|
class GroongaInput < Input
|
29
31
|
Plugin.register_input("groonga", self)
|
@@ -45,9 +47,9 @@ module Fluent
|
|
45
47
|
super
|
46
48
|
case @protocol
|
47
49
|
when :http
|
48
|
-
@input = HTTPInput.new
|
50
|
+
@input = HTTPInput.new(self)
|
49
51
|
when :gqtp
|
50
|
-
@input = GQTPInput.new
|
52
|
+
@input = GQTPInput.new(self)
|
51
53
|
end
|
52
54
|
@input.configure(conf)
|
53
55
|
end
|
@@ -107,6 +109,10 @@ module Fluent
|
|
107
109
|
end
|
108
110
|
end
|
109
111
|
|
112
|
+
def initialize(input_plugin)
|
113
|
+
@input_plugin = input_plugin
|
114
|
+
end
|
115
|
+
|
110
116
|
def configure(conf)
|
111
117
|
super
|
112
118
|
|
@@ -147,7 +153,9 @@ module Fluent
|
|
147
153
|
|
148
154
|
def emit(command, params)
|
149
155
|
return unless emit_command?(command)
|
150
|
-
|
156
|
+
@input_plugin.router.emit("groonga.command.#{command}",
|
157
|
+
Engine.now,
|
158
|
+
params)
|
151
159
|
end
|
152
160
|
|
153
161
|
private
|
data/sample/command.conf
CHANGED
data/sample/gqtp.conf
CHANGED
data/sample/http.conf
CHANGED
data/sample/store-apache.conf
CHANGED
@@ -1,9 +1,9 @@
|
|
1
1
|
<source>
|
2
|
-
type forward
|
2
|
+
@type forward
|
3
3
|
</source>
|
4
4
|
|
5
5
|
<source>
|
6
|
-
type tail
|
6
|
+
@type tail
|
7
7
|
path /var/log/apache2/access.log
|
8
8
|
pos_file /tmp/apache_access.pos
|
9
9
|
tag apache.raw.log.apache.access
|
@@ -12,7 +12,7 @@
|
|
12
12
|
</source>
|
13
13
|
|
14
14
|
<match apache.**>
|
15
|
-
type record_reformer
|
15
|
+
@type record_reformer
|
16
16
|
enable_ruby false
|
17
17
|
|
18
18
|
tag ${tag_suffix[1]}
|
@@ -23,7 +23,7 @@
|
|
23
23
|
</match>
|
24
24
|
|
25
25
|
<match raw.log.**>
|
26
|
-
type record_reformer
|
26
|
+
@type record_reformer
|
27
27
|
enable_ruby false
|
28
28
|
|
29
29
|
tag ${tag_suffix[1]}
|
@@ -36,7 +36,7 @@
|
|
36
36
|
</match>
|
37
37
|
|
38
38
|
<match log.**>
|
39
|
-
type groonga
|
39
|
+
@type groonga
|
40
40
|
store_table Logs
|
41
41
|
|
42
42
|
protocol http
|
data/sample/store-syslog.conf
CHANGED
@@ -5,19 +5,20 @@
|
|
5
5
|
# * fluent-plugin-groonga
|
6
6
|
|
7
7
|
<source>
|
8
|
-
type forward
|
8
|
+
@type forward
|
9
9
|
</source>
|
10
10
|
|
11
11
|
<source>
|
12
|
-
type tail
|
12
|
+
@type tail
|
13
13
|
path /var/log/messages
|
14
14
|
pos_file /var/log/td-agent/messages.pos
|
15
15
|
tag "raw.messages.log.#{Socket.gethostname}"
|
16
|
+
read_from_head true
|
16
17
|
format none
|
17
18
|
</source>
|
18
19
|
|
19
20
|
<match raw.*.log.**>
|
20
|
-
type forest
|
21
|
+
@type forest
|
21
22
|
subtype parser
|
22
23
|
|
23
24
|
<template>
|
@@ -31,7 +32,7 @@
|
|
31
32
|
</match>
|
32
33
|
|
33
34
|
<match *.log.*.**>
|
34
|
-
type record_reformer
|
35
|
+
@type record_reformer
|
35
36
|
enable_ruby false
|
36
37
|
|
37
38
|
tag ${tag_parts[1]}
|
@@ -44,7 +45,7 @@
|
|
44
45
|
</match>
|
45
46
|
|
46
47
|
<match log>
|
47
|
-
type groonga
|
48
|
+
@type groonga
|
48
49
|
store_table Logs
|
49
50
|
|
50
51
|
protocol http
|
data/sample/store.conf
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-groonga
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.1.
|
4
|
+
version: 1.1.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Kouhei Sutou
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-05-27 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: fluentd
|
@@ -186,7 +186,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
186
186
|
version: '0'
|
187
187
|
requirements: []
|
188
188
|
rubyforge_project:
|
189
|
-
rubygems_version: 2.
|
189
|
+
rubygems_version: 2.5.1
|
190
190
|
signing_key:
|
191
191
|
specification_version: 4
|
192
192
|
summary: Fluentd plugin to store data into Groonga and implement Groonga replication
|