fluent-plugin-rewrite-tag-filter 2.0.0 → 2.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.travis.yml +4 -4
- data/README.md +100 -28
- data/example.conf +132 -96
- data/example2.conf +63 -37
- data/fluent-plugin-rewrite-tag-filter.gemspec +4 -3
- data/lib/fluent/plugin/out_rewrite_tag_filter.rb +53 -14
- data/test/helper.rb +4 -0
- data/test/plugin/test_out_rewrite_tag_filter.rb +224 -0
- metadata +39 -6
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: a12e8f6567cd07462a236f0cda38e98bac14b921a2c7b762e9851cb04e5bf119
|
4
|
+
data.tar.gz: b198b1e6b8a48d386f7cc62521f358cdff745754edc8dda753ac176ced1e3e67
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: cc556895e92c3f53ca3c41af6866072e48ac53b6b170cfe163f1563d04310c8b89cc802cabefd957dab95da29362ff76164621f0a83d806619d67df521e866b6
|
7
|
+
data.tar.gz: 1777b76d00afe128f20da230ca26d0c30d4f62e60f4800efec6b59450040e54de7b861ee7913ed82af8f91b9f3cf5e11344b5ec5b08e57af97d611a0060d0ddf
|
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -3,10 +3,12 @@
|
|
3
3
|
## Overview
|
4
4
|
|
5
5
|
Rewrite Tag Filter for [Fluentd](http://fluentd.org). It is designed to rewrite tags like mod_rewrite.
|
6
|
-
Re-emit the record with
|
6
|
+
Re-emit the record with rewritten tag when a value matches/unmatches with a regular expression.
|
7
7
|
Also you can change a tag from Apache log by domain, status code (ex. 500 error),
|
8
8
|
user-agent, request-uri, regex-backreference and so on with regular expression.
|
9
9
|
|
10
|
+
This is an output plugin because fluentd's `filter` doesn't allow tag rewrite.
|
11
|
+
|
10
12
|
## Requirements
|
11
13
|
|
12
14
|
| fluent-plugin-rewrite-tag-filter | Fluentd | Ruby |
|
@@ -16,34 +18,41 @@ user-agent, request-uri, regex-backreference and so on with regular expression.
|
|
16
18
|
|
17
19
|
## Installation
|
18
20
|
|
19
|
-
Install with `gem
|
21
|
+
Install with `gem` or `td-agent-gem` command as:
|
20
22
|
|
21
23
|
```
|
22
24
|
# for system installed fluentd
|
23
25
|
$ gem install fluent-plugin-rewrite-tag-filter
|
24
26
|
|
25
|
-
# for td-agent (Legacy)
|
26
|
-
$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-rewrite-tag-filter -v 1.5.6
|
27
|
-
|
28
27
|
# for td-agent2 (with fluentd v0.12)
|
29
|
-
$ sudo td-agent-gem install fluent-plugin-rewrite-tag-filter -v 1.
|
28
|
+
$ sudo td-agent-gem install fluent-plugin-rewrite-tag-filter -v 1.6.0
|
29
|
+
|
30
|
+
# for td-agent3 (with fluentd v0.14)
|
31
|
+
$ sudo td-agent-gem install fluent-plugin-rewrite-tag-filter
|
30
32
|
```
|
31
33
|
|
34
|
+
For more details, see [Plugin Management](https://docs.fluentd.org/deployment/plugin-management)
|
35
|
+
|
32
36
|
## Configuration
|
33
37
|
|
34
38
|
* **rewriterule\<num\>** (string) (optional) \<attribute\> \<regex_pattern\> \<new_tag\>
|
35
|
-
*
|
39
|
+
* Obsoleted: Use \<rule\> section
|
36
40
|
* **capitalize_regex_backreference** (bool) (optional): Capitalize letter for every matched regex backreference. (ex: maps -> Maps) for more details, see usage.
|
37
41
|
* Default value: no
|
38
42
|
* **remove_tag_prefix** (string) (optional): Remove tag prefix for tag placeholder. (see the section of "Tag placeholder")
|
39
43
|
* **hostname_command** (string) (optional): Override hostname command for placeholder. (see the section of "Tag placeholder")
|
40
|
-
* Default value: `hostname
|
44
|
+
* Default value: `hostname`
|
45
|
+
* **emit_mode** (enum) (required): Specify emit_mode to `batch` or `record`. `batch` will emit events per rewritten tag, and decrease IO. `record` will emit events per record.
|
46
|
+
* Default value: `batch`
|
41
47
|
|
42
48
|
### \<rule\> section (optional) (multiple)
|
43
49
|
|
44
50
|
* **key** (string) (required): The field name to which the regular expression is applied
|
45
|
-
* **pattern** (regexp) (required): The regular expression
|
51
|
+
* **pattern** (regexp) (required): The regular expression.
|
52
|
+
`/regexp/` is preferred because `/regexp/` style can support character classes such as `/[a-z]/`.
|
53
|
+
The pattern without slashes will cause errors if you use patterns start with character classes.
|
46
54
|
* **tag** (string) (required): New tag
|
55
|
+
* **label** (string) (optional): New label. If specified, label can be changed per-rule.
|
47
56
|
* **invert** (bool) (optional): If true, rewrite tag when unmatch pattern
|
48
57
|
* Default value: `false`
|
49
58
|
|
@@ -70,39 +79,39 @@ It's a sample to exclude some static file log before split tag by domain.
|
|
70
79
|
capitalize_regex_backreference yes
|
71
80
|
<rule>
|
72
81
|
key path
|
73
|
-
pattern
|
82
|
+
pattern /\.(gif|jpe?g|png|pdf|zip)$/
|
74
83
|
tag clear
|
75
84
|
</rule>
|
76
85
|
<rule>
|
77
86
|
key status
|
78
|
-
pattern
|
87
|
+
pattern /^200$/
|
79
88
|
tag clear
|
80
89
|
invert true
|
81
90
|
</rule>
|
82
91
|
<rule>
|
83
92
|
key domain
|
84
|
-
pattern
|
93
|
+
pattern /^.+\.com$/
|
85
94
|
tag clear
|
86
95
|
invert true
|
87
96
|
</rule>
|
88
97
|
<rule>
|
89
98
|
key domain
|
90
|
-
pattern
|
99
|
+
pattern /^maps\.example\.com$/
|
91
100
|
tag site.ExampleMaps
|
92
101
|
</rule>
|
93
102
|
<rule>
|
94
103
|
key domain
|
95
|
-
pattern
|
104
|
+
pattern /^news\.example\.com$/
|
96
105
|
tag site.ExampleNews
|
97
106
|
</rule>
|
98
107
|
<rule>
|
99
108
|
key domain
|
100
|
-
pattern
|
109
|
+
pattern /^(mail)\.(example)\.com$/
|
101
110
|
tag site.$2$1
|
102
111
|
</rule>
|
103
112
|
<rule>
|
104
113
|
key domain
|
105
|
-
pattern
|
114
|
+
pattern /.+/
|
106
115
|
tag site.unmatched
|
107
116
|
</rule>
|
108
117
|
</match>
|
@@ -157,7 +166,11 @@ Dot notation:
|
|
157
166
|
```
|
158
167
|
<match kubernetes.**>
|
159
168
|
@type rewrite_tag_filter
|
160
|
-
|
169
|
+
<rule>
|
170
|
+
key $.kubernetes.namespace_name
|
171
|
+
pattern ^(.+)$
|
172
|
+
tag $1.${tag}
|
173
|
+
</rule>
|
161
174
|
</match>
|
162
175
|
```
|
163
176
|
|
@@ -166,7 +179,11 @@ Bracket notation:
|
|
166
179
|
```
|
167
180
|
<match kubernetes.**>
|
168
181
|
@type rewrite_tag_filter
|
169
|
-
|
182
|
+
<rule>
|
183
|
+
key $['kubernetes']['namespace_name']
|
184
|
+
pattern ^(.+)$
|
185
|
+
tag $1.${tag}
|
186
|
+
</rule>
|
170
187
|
</match>
|
171
188
|
```
|
172
189
|
|
@@ -184,7 +201,7 @@ When original tag is `kubernetes.var.log`, this will be converted to `default.ku
|
|
184
201
|
|
185
202
|
### Tag placeholder
|
186
203
|
|
187
|
-
It is supported these placeholder for new_tag (
|
204
|
+
It is supported these placeholder for new_tag (rewritten tag).
|
188
205
|
|
189
206
|
- `${tag}`
|
190
207
|
- `__TAG__`
|
@@ -198,12 +215,16 @@ For example with `td.apache.access` tag, it will get `td` by `${tag_parts[0]}` a
|
|
198
215
|
|
199
216
|
**Note** Currently, range expression ```${tag_parts[0..2]}``` is not supported.
|
200
217
|
|
201
|
-
#### Placeholder
|
218
|
+
#### Placeholder Options
|
202
219
|
|
203
220
|
* `remove_tag_prefix`
|
204
221
|
|
205
222
|
This option adds removing tag prefix for `${tag}` or `__TAG__` in placeholder.
|
206
223
|
|
224
|
+
* `remove_tag_regexp`
|
225
|
+
|
226
|
+
This option adds removing tag regexp for `${tag}` or `__TAG__` in placeholder.
|
227
|
+
|
207
228
|
* `hostname_command`
|
208
229
|
|
209
230
|
By default, execute command as `hostname` to get full hostname.
|
@@ -215,49 +236,100 @@ It comes short hostname with `hostname_command hostname -s` configuration specif
|
|
215
236
|
It's a sample to rewrite a tag with placeholder.
|
216
237
|
|
217
238
|
```
|
218
|
-
# It will get "
|
239
|
+
# It will get "rewritten.access.ExampleMail"
|
219
240
|
<match apache.access>
|
220
241
|
@type rewrite_tag_filter
|
221
242
|
remove_tag_prefix apache
|
222
243
|
<rule>
|
223
244
|
key domain
|
224
245
|
pattern ^(mail)\.(example)\.com$
|
225
|
-
tag
|
246
|
+
tag rewritten.${tag}.$2$1
|
247
|
+
</rule>
|
248
|
+
</match>
|
249
|
+
|
250
|
+
# It will get "rewritten.access.ExampleMail"
|
251
|
+
<match apache.access>
|
252
|
+
@type rewrite_tag_filter
|
253
|
+
remove_tag_regexp /^apache\./
|
254
|
+
<rule>
|
255
|
+
key domain
|
256
|
+
pattern ^(mail)\.(example)\.com$
|
257
|
+
tag rewritten.${tag}.$2$1
|
258
|
+
</rule>
|
259
|
+
</match>
|
260
|
+
|
261
|
+
# It will get "http.access.log"
|
262
|
+
<match input.{apache,nginx}.access.log>
|
263
|
+
@type rewrite_tag_filter
|
264
|
+
remove_tag_regexp /^input\.(apache|nginx)\./
|
265
|
+
<rule>
|
266
|
+
key domain
|
267
|
+
pattern ^.+$
|
268
|
+
tag http.${tag}
|
226
269
|
</rule>
|
227
270
|
</match>
|
228
271
|
|
229
|
-
# It will get "
|
272
|
+
# It will get "rewritten.ExampleMail.app30-124.foo.com" when hostname is "app30-124.foo.com"
|
230
273
|
<match apache.access>
|
231
274
|
@type rewrite_tag_filter
|
232
275
|
<rule>
|
233
276
|
key domain
|
234
277
|
pattern ^(mail)\.(example)\.com$
|
235
|
-
tag
|
278
|
+
tag rewritten.$2$1.${hostname}
|
236
279
|
</rule>
|
237
280
|
</match>
|
238
281
|
|
239
|
-
# It will get "
|
282
|
+
# It will get "rewritten.ExampleMail.app30-124" when hostname is "app30-124.foo.com"
|
240
283
|
<match apache.access>
|
241
284
|
@type rewrite_tag_filter
|
242
285
|
hostname_command hostname -s
|
243
286
|
<rule>
|
244
287
|
key domain
|
245
288
|
pattern ^(mail)\.(example)\.com$
|
246
|
-
tag
|
289
|
+
tag rewritten.$2$1.${hostname}
|
247
290
|
</rule>
|
248
291
|
</match>
|
249
292
|
|
250
|
-
# It will get "
|
293
|
+
# It will get "rewritten.game.pool"
|
251
294
|
<match app.game.pool.activity>
|
252
295
|
@type rewrite_tag_filter
|
253
296
|
<rule>
|
254
297
|
key domain
|
255
298
|
pattern ^.+$
|
256
|
-
tag
|
299
|
+
tag rewritten.${tag_parts[1]}.${tag_parts[2]}
|
257
300
|
</rule>
|
258
301
|
</match>
|
259
302
|
```
|
260
303
|
|
304
|
+
### Altering Labels
|
305
|
+
|
306
|
+
In addition to changing tags, you can also change event's route by setting
|
307
|
+
the label for the re-emitted event.
|
308
|
+
|
309
|
+
For example, given this configuration:
|
310
|
+
|
311
|
+
```
|
312
|
+
<match apache.access>
|
313
|
+
@type rewrite_tag_filter
|
314
|
+
<rule>
|
315
|
+
key domain
|
316
|
+
pattern ^www\.example\.com$
|
317
|
+
tag web.${tag}
|
318
|
+
</rule>
|
319
|
+
<rule>
|
320
|
+
key domain
|
321
|
+
pattern ^(.*)\.example\.com$
|
322
|
+
tag other.$1
|
323
|
+
label other
|
324
|
+
</rule>
|
325
|
+
</match>
|
326
|
+
```
|
327
|
+
|
328
|
+
message: `{"domain": "www.example.com"}` will get its tag changed to
|
329
|
+
`web.apache.access`, while message
|
330
|
+
`{"domain": "api.example.com"}` will get its tag changed to `other.api` and
|
331
|
+
be sent to label `other`
|
332
|
+
|
261
333
|
## Example
|
262
334
|
|
263
335
|
- Example1: how to analyze response_time, response_code and user_agent for each virtual domain websites.
|
data/example.conf
CHANGED
@@ -2,15 +2,17 @@
|
|
2
2
|
|
3
3
|
# appearing plugins:
|
4
4
|
# rewrite_tag_filter: http://rubygems.org/gems/fluent-plugin-rewrite-tag-filter
|
5
|
-
# forest: http://rubygems.org/gems/fluent-plugin-forest
|
6
5
|
# datacounter: http://rubygems.org/gems/fluent-plugin-datacounter
|
7
6
|
# growthforecast: http://rubygems.org/gems/fluent-plugin-growthforecast
|
8
7
|
|
9
8
|
<source>
|
10
|
-
type tail
|
9
|
+
@type tail
|
11
10
|
path /var/log/httpd/access_log
|
12
|
-
|
13
|
-
|
11
|
+
<parse>
|
12
|
+
@type regexp
|
13
|
+
expression /^(?<domain>[^ ]*) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/
|
14
|
+
time_format %d/%b/%Y:%H:%M:%S %z
|
15
|
+
</parse>
|
14
16
|
tag td.apache.access
|
15
17
|
pos_file /var/log/td-agent/apache_access.pos
|
16
18
|
</source>
|
@@ -18,139 +20,173 @@
|
|
18
20
|
|
19
21
|
# Extract specified virtual domain
|
20
22
|
<match td.apache.access>
|
21
|
-
type copy
|
23
|
+
@type copy
|
22
24
|
<store>
|
23
|
-
type rewrite_tag_filter
|
25
|
+
@type rewrite_tag_filter
|
24
26
|
capitalize_regex_backreference yes
|
25
|
-
|
27
|
+
<rule>
|
28
|
+
key domain
|
29
|
+
pattern /^(maps|news|mail)\.google\.com$/
|
30
|
+
tag site.Google$1
|
31
|
+
</rule>
|
26
32
|
</store>
|
27
33
|
<store>
|
28
|
-
type rewrite_tag_filter
|
34
|
+
@type rewrite_tag_filter
|
29
35
|
capitalize_regex_backreference yes
|
30
|
-
|
36
|
+
<rule>
|
37
|
+
key domain
|
38
|
+
pattern /^(maps)\.google\.com$/
|
39
|
+
tag sitepath.Google$1
|
40
|
+
</rule>
|
31
41
|
</store>
|
32
42
|
</match>
|
33
43
|
|
34
44
|
|
35
45
|
# Second level analyzing
|
36
46
|
<match sitepath.GoogleMaps>
|
47
|
+
@type copy
|
37
48
|
<store>
|
38
|
-
type rewrite_tag_filter
|
39
|
-
|
40
|
-
|
41
|
-
|
49
|
+
@type rewrite_tag_filter
|
50
|
+
<rule>
|
51
|
+
key path
|
52
|
+
pattern /^\/labs/
|
53
|
+
tag site.GoogleMaps.Labs
|
54
|
+
</rule>
|
55
|
+
<rule>
|
56
|
+
key path
|
57
|
+
pattern /^\/static/\d+/
|
58
|
+
tag site.GoogleMaps.Static
|
59
|
+
</rule>
|
60
|
+
<rule>
|
61
|
+
key path
|
62
|
+
pattern /^\/help\/maps\/(streetview|getmaps)/
|
63
|
+
tag site.GoogleMaps.Help
|
64
|
+
</rule>
|
42
65
|
</store>
|
43
66
|
<store>
|
44
|
-
type rewrite_tag_filter
|
45
|
-
|
46
|
-
|
67
|
+
@type rewrite_tag_filter
|
68
|
+
<rule>
|
69
|
+
key referer
|
70
|
+
pattern /headlines\.yahoo\.co\.jp/
|
71
|
+
tag site.GoogleMaps.referer_YahooHeadlines
|
72
|
+
</rule>
|
73
|
+
<rule>
|
74
|
+
key referer
|
75
|
+
pattern /news\.livedoor\.com/
|
76
|
+
tag site.GoogleMaps.referer_LivedoorNews
|
77
|
+
</rule>
|
47
78
|
</store>
|
48
79
|
<store>
|
49
|
-
type rewrite_tag_filter
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
80
|
+
@type rewrite_tag_filter
|
81
|
+
<rule>
|
82
|
+
key agent
|
83
|
+
pattern /Googlebot\//
|
84
|
+
tag site.GoogleMaps.agent_Googlebot
|
85
|
+
</rule>
|
86
|
+
<rule>
|
87
|
+
key agent
|
88
|
+
pattern /^.* iPhone .+Googlebot-Mobile\/.*$/
|
89
|
+
tag site.GoogleMaps.agent_GooglebotSmartphone
|
90
|
+
</rule>
|
91
|
+
<rule>
|
92
|
+
key agent
|
93
|
+
pattern /Googlebot-Mobile\//
|
94
|
+
tag site.GoogleMaps.agent_GooglebotMobile
|
95
|
+
</rule>
|
96
|
+
<rule>
|
97
|
+
key agent
|
98
|
+
pattern /bingbot/
|
99
|
+
tag site.GoogleMaps.agent_Bingbot
|
100
|
+
</rule>
|
101
|
+
<rule>
|
102
|
+
key agent
|
103
|
+
pattern /Baiduspider/
|
104
|
+
tag site.GoogleMaps.agent_Baiduspider
|
105
|
+
</rule>
|
55
106
|
</store>
|
56
107
|
</match>
|
57
108
|
|
58
109
|
|
59
110
|
# Summarize
|
60
111
|
<match site.**>
|
61
|
-
type copy
|
112
|
+
@type copy
|
62
113
|
<store>
|
63
|
-
type
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
pattern7 10sec_over ^\d{8,}$
|
78
|
-
</template>
|
114
|
+
@type datacounter
|
115
|
+
unit minute
|
116
|
+
count_key response_time
|
117
|
+
outcast_unmatched false
|
118
|
+
aggregate tag
|
119
|
+
tag_prefix gf.responsetime
|
120
|
+
output_per_tag yes
|
121
|
+
pattern1 0-100msec ^\d{1,5}$
|
122
|
+
pattern2 100-300msec ^[1-2]\d{5}$
|
123
|
+
pattern3 300-600msec ^[3-5]\d{5}$
|
124
|
+
pattern4 600msec-1sec ^[6-9]\d{5}$
|
125
|
+
pattern5 1-2sec ^1\d{6}$
|
126
|
+
pattern6 2-10sec ^[2-9]\d{6}$
|
127
|
+
pattern7 10sec_over ^\d{8,}$
|
79
128
|
</store>
|
80
129
|
<store>
|
81
|
-
type
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
pattern10 500 ^5\d\d$
|
99
|
-
</template>
|
130
|
+
@type datacounter
|
131
|
+
unit minute
|
132
|
+
outcast_unmatched false
|
133
|
+
aggregate tag
|
134
|
+
tag_prefix gf.responsecode
|
135
|
+
output_per_tag yes
|
136
|
+
count_key status
|
137
|
+
pattern1 200 ^200$
|
138
|
+
pattern2 2xx ^2\d\d$
|
139
|
+
pattern3 301 ^301$
|
140
|
+
pattern4 302 ^302$
|
141
|
+
pattern5 3xx ^3\d\d$
|
142
|
+
pattern6 403 ^403$
|
143
|
+
pattern7 404 ^404$
|
144
|
+
pattern8 410 ^410$
|
145
|
+
pattern9 4xx ^4\d\d$
|
146
|
+
pattern10 500 ^5\d\d$
|
100
147
|
</store>
|
101
148
|
<store>
|
102
|
-
type
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
110
|
-
|
111
|
-
|
112
|
-
|
113
|
-
|
114
|
-
pattern5 pc .+
|
115
|
-
</template>
|
149
|
+
@type datacounter
|
150
|
+
unit minute
|
151
|
+
count_key agent
|
152
|
+
outcast_unmatched false
|
153
|
+
aggregate tag
|
154
|
+
tag_prefix gf.useragent
|
155
|
+
output_per_tag yes
|
156
|
+
pattern1 api HttpRequest
|
157
|
+
pattern2 robot (spider|bot|crawler|\+http\:)
|
158
|
+
pattern3 smartphone (iPhone|iPod|Android|dream|CUPCAKE|blackberry|webOS|incognito|webmate|IEMobile)
|
159
|
+
pattern4 mobile (^KDDI|UP.Browser|DoCoMo|Vodafone|SoftBank|WILLCOM)
|
160
|
+
pattern5 pc .+
|
116
161
|
</store>
|
117
162
|
</match>
|
118
163
|
|
119
164
|
|
120
165
|
# Graph
|
121
166
|
<match gf.responsetime.**>
|
122
|
-
type
|
123
|
-
subtype growthforecast
|
167
|
+
@type growthforecast
|
124
168
|
remove_prefix gf.responsetime.site
|
125
|
-
|
126
|
-
|
127
|
-
|
128
|
-
|
129
|
-
name_keys 0-100msec_percentage,100-300msec_percentage,300-600msec_percentage,600msec-1sec_percentage,1-2sec_percentage,2-10sec_percentage,10sec_over_percentage
|
130
|
-
</template>
|
169
|
+
gfapi_url http://localhost:5125/api/
|
170
|
+
tag_for service
|
171
|
+
section response_time
|
172
|
+
name_keys 0-100msec_percentage,100-300msec_percentage,300-600msec_percentage,600msec-1sec_percentage,1-2sec_percentage,2-10sec_percentage,10sec_over_percentage
|
131
173
|
</match>
|
132
174
|
|
133
175
|
<match gf.responsecode.**>
|
134
|
-
type
|
135
|
-
subtype growthforecast
|
176
|
+
@type growthforecast
|
136
177
|
remove_prefix gf.responsecode.site
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
141
|
-
name_keys 301_count,302_count,3xx_count,403_count,404_count,410_count,4xx_count,500_count
|
142
|
-
</template>
|
178
|
+
gfapi_url http://localhost:5125/api/
|
179
|
+
tag_for service
|
180
|
+
section response_code
|
181
|
+
name_keys 301_count,302_count,3xx_count,403_count,404_count,410_count,4xx_count,500_count
|
143
182
|
</match>
|
144
183
|
|
145
184
|
<match gf.useragent.**>
|
146
|
-
type
|
147
|
-
subtype growthforecast
|
185
|
+
@type growthforecast
|
148
186
|
remove_prefix gf.useragent.site
|
149
|
-
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
name_keys pc_count,mobile_count,smartphone_count,robot_count,api_count
|
154
|
-
</template>
|
187
|
+
gfapi_url http://localhost:5125/api/
|
188
|
+
tag_for service
|
189
|
+
section useragent
|
190
|
+
name_keys pc_count,mobile_count,smartphone_count,robot_count,api_count
|
155
191
|
</match>
|
156
192
|
|
data/example2.conf
CHANGED
@@ -7,10 +7,13 @@
|
|
7
7
|
# growthforecast: http://rubygems.org/gems/fluent-plugin-growthforecast
|
8
8
|
|
9
9
|
<source>
|
10
|
-
type tail
|
10
|
+
@type tail
|
11
11
|
path /var/log/httpd/access_log
|
12
|
-
|
13
|
-
|
12
|
+
<parse>
|
13
|
+
@type regexp
|
14
|
+
expression /^(?<domain>[^ ]*) (?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<status>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<response_time>[^ ]*))?$/
|
15
|
+
time_format %d/%b/%Y:%H:%M:%S %z
|
16
|
+
</parse>
|
14
17
|
tag td.apache.access
|
15
18
|
pos_file /var/log/td-agent/apache_access.pos
|
16
19
|
</source>
|
@@ -18,60 +21,83 @@
|
|
18
21
|
|
19
22
|
# Extract specified virtual domain
|
20
23
|
<match td.apache.access>
|
21
|
-
type rewrite_tag_filter
|
22
|
-
|
24
|
+
@type rewrite_tag_filter
|
25
|
+
<rule>
|
26
|
+
key domain
|
27
|
+
pattern /^maps\.google\.com$/
|
28
|
+
tag filter.GoogleMap
|
29
|
+
</rule>
|
23
30
|
</match>
|
24
31
|
|
25
32
|
|
26
33
|
# Filtering
|
27
34
|
<match filter.GoogleMap>
|
28
|
-
type rewrite_tag_filter
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
+
@type rewrite_tag_filter
|
36
|
+
<rule>
|
37
|
+
key path
|
38
|
+
pattern /^\/(img|css|js|static|assets)\//
|
39
|
+
tag clear
|
40
|
+
</rule>
|
41
|
+
<rule>
|
42
|
+
key status
|
43
|
+
pattern /^(?!200)$/
|
44
|
+
tag clear
|
45
|
+
</rule>
|
46
|
+
<rule>
|
47
|
+
key method
|
48
|
+
pattern /^(?!GET)$/
|
49
|
+
tag clear
|
50
|
+
</rule>
|
51
|
+
<rule>
|
52
|
+
key agent
|
53
|
+
pattern /(spider|bot|crawler|\+http\:)/
|
54
|
+
tag clean
|
55
|
+
</rule>
|
56
|
+
<rule>
|
57
|
+
key path
|
58
|
+
pattern /^\/(admin|api|backend)/
|
59
|
+
tag site.GoogleMap.backend
|
60
|
+
</rule>
|
61
|
+
<rule>
|
62
|
+
key path
|
63
|
+
pattern /.+/
|
64
|
+
tag site.GoogleMap.front
|
65
|
+
</rule>
|
35
66
|
</match>
|
36
67
|
|
37
68
|
|
38
69
|
# Summarize
|
39
70
|
<match site.**>
|
40
|
-
type
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
pattern7 10sec_over ^\d{8,}$
|
55
|
-
</template>
|
71
|
+
@type datacounter
|
72
|
+
unit minute
|
73
|
+
count_key response_time
|
74
|
+
outcast_unmatched false
|
75
|
+
aggregate tag
|
76
|
+
tag_prefix gf.responsetime
|
77
|
+
output_per_tag yes
|
78
|
+
pattern1 0-100msec ^\d{1,5}$
|
79
|
+
pattern2 100-300msec ^[1-2]\d{5}$
|
80
|
+
pattern3 300-600msec ^[3-5]\d{5}$
|
81
|
+
pattern4 600msec-1sec ^[6-9]\d{5}$
|
82
|
+
pattern5 1-2sec ^1\d{6}$
|
83
|
+
pattern6 2-10sec ^[2-9]\d{6}$
|
84
|
+
pattern7 10sec_over ^\d{8,}$
|
56
85
|
</match>
|
57
86
|
|
58
87
|
|
59
88
|
# Graph
|
60
89
|
<match gf.responsetime.**>
|
61
|
-
type
|
62
|
-
subtype growthforecast
|
90
|
+
@type growthforecast
|
63
91
|
remove_prefix gf.responsetime.site
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
name_keys 0-100msec_percentage,100-300msec_percentage,300-600msec_percentage,600msec-1sec_percentage,1-2sec_percentage,2-10sec_percentage,10sec_over_percentage
|
69
|
-
</template>
|
92
|
+
gfapi_url http://localhost:5125/api/
|
93
|
+
tag_for service
|
94
|
+
section response_time
|
95
|
+
name_keys 0-100msec_percentage,100-300msec_percentage,300-600msec_percentage,600msec-1sec_percentage,1-2sec_percentage,2-10sec_percentage,10sec_over_percentage
|
70
96
|
</match>
|
71
97
|
|
72
98
|
|
73
99
|
# Clear tag
|
74
100
|
<match clear>
|
75
|
-
type null
|
101
|
+
@type null
|
76
102
|
</match>
|
77
103
|
|
@@ -1,9 +1,8 @@
|
|
1
|
-
# -*- encoding: utf-8 -*-
|
2
1
|
$:.push File.expand_path("../lib", __FILE__)
|
3
2
|
|
4
3
|
Gem::Specification.new do |s|
|
5
4
|
s.name = "fluent-plugin-rewrite-tag-filter"
|
6
|
-
s.version = "2.
|
5
|
+
s.version = "2.4.0"
|
7
6
|
s.license = "Apache-2.0"
|
8
7
|
s.authors = ["Kentaro Yoshida"]
|
9
8
|
s.email = ["y.ken.studio@gmail.com"]
|
@@ -16,6 +15,8 @@ Gem::Specification.new do |s|
|
|
16
15
|
s.require_paths = ["lib"]
|
17
16
|
|
18
17
|
s.add_development_dependency "test-unit", ">= 3.1.0"
|
18
|
+
s.add_development_dependency "test-unit-rr"
|
19
19
|
s.add_development_dependency "rake"
|
20
|
-
s.add_runtime_dependency "fluentd", "
|
20
|
+
s.add_runtime_dependency "fluentd", ">= 0.14.2", "< 2"
|
21
|
+
s.add_runtime_dependency "fluent-config-regexp-type"
|
21
22
|
end
|
@@ -1,4 +1,5 @@
|
|
1
1
|
require "fluent/plugin/output"
|
2
|
+
require "fluent/config/regexp_type"
|
2
3
|
|
3
4
|
class Fluent::Plugin::RewriteTagFilterOutput < Fluent::Plugin::Output
|
4
5
|
Fluent::Plugin.register_output('rewrite_tag_filter', self)
|
@@ -9,18 +10,22 @@ class Fluent::Plugin::RewriteTagFilterOutput < Fluent::Plugin::Output
|
|
9
10
|
config_param :capitalize_regex_backreference, :bool, :default => false
|
10
11
|
desc 'Remove tag prefix for tag placeholder.'
|
11
12
|
config_param :remove_tag_prefix, :string, :default => nil
|
13
|
+
desc "Remove tag regexp for tag placeholder."
|
14
|
+
config_param :remove_tag_regexp, :regexp, default: nil
|
12
15
|
desc 'Override hostname command for placeholder.'
|
13
16
|
config_param :hostname_command, :string, :default => 'hostname'
|
17
|
+
desc "The emit mode. If `batch`, this plugin will emit events per rewritten tag."
|
18
|
+
config_param :emit_mode, :enum, list: [:record, :batch], default: :batch
|
14
19
|
|
15
20
|
config_section :rule, param_name: :rules, multi: true do
|
16
21
|
desc "The field name to which the regular expression is applied"
|
17
22
|
config_param :key, :string
|
18
23
|
desc "The regular expression"
|
19
|
-
config_param :pattern
|
20
|
-
Regexp.compile(value)
|
21
|
-
end
|
24
|
+
config_param :pattern, :regexp
|
22
25
|
desc "New tag"
|
23
26
|
config_param :tag, :string
|
27
|
+
desc "New label. If specified, label can be changed per-rule."
|
28
|
+
config_param :label, :string, default: nil
|
24
29
|
desc "If true, rewrite tag when unmatch pattern"
|
25
30
|
config_param :invert, :bool, default: false
|
26
31
|
end
|
@@ -40,7 +45,7 @@ class Fluent::Plugin::RewriteTagFilterOutput < Fluent::Plugin::Output
|
|
40
45
|
end
|
41
46
|
|
42
47
|
invert = rule.invert ? MATCH_OPERATOR_EXCLUDE : ""
|
43
|
-
@rewriterules.push([record_accessor_create(rule.key), rule.pattern, invert, rule.tag])
|
48
|
+
@rewriterules.push([record_accessor_create(rule.key), rule.pattern, invert, rule.tag, rule.label])
|
44
49
|
rewriterule_names.push(rule.key + invert + rule.pattern.to_s)
|
45
50
|
log.info "adding rewrite_tag_filter rule: #{rule.key} #{@rewriterules.last}"
|
46
51
|
end
|
@@ -57,26 +62,60 @@ class Fluent::Plugin::RewriteTagFilterOutput < Fluent::Plugin::Output
|
|
57
62
|
raise Fluent::ConfigError, "duplicated rewriterules found #{@rewriterules.inspect}"
|
58
63
|
end
|
59
64
|
|
65
|
+
if @remove_tag_prefix && @remove_tag_regexp
|
66
|
+
raise Fluent::ConfigError, "remove_tag_prefix and remove_tag_regexp are exclusive"
|
67
|
+
end
|
68
|
+
|
60
69
|
unless @remove_tag_prefix.nil?
|
61
|
-
@
|
70
|
+
@remove_tag_regexp = /^#{Regexp.escape(@remove_tag_prefix)}\.?/
|
62
71
|
end
|
72
|
+
|
73
|
+
@batch_mode = @emit_mode == :batch
|
63
74
|
end
|
64
75
|
|
65
76
|
def multi_workers_ready?
|
66
77
|
true
|
67
78
|
end
|
68
79
|
|
80
|
+
def get_router(tgt_label)
|
81
|
+
label_router = if tgt_label.nil? || tgt_label.empty?
|
82
|
+
router
|
83
|
+
else
|
84
|
+
event_emitter_router(tgt_label)
|
85
|
+
end
|
86
|
+
#log.trace "Got router for #{tgt_label}: #{!label_router.nil?}"
|
87
|
+
return label_router
|
88
|
+
end
|
89
|
+
|
69
90
|
def process(tag, es)
|
91
|
+
placeholder = get_placeholder(tag)
|
92
|
+
new_event_streams = Hash.new {|hh, kk| hh[kk] = Hash.new {|h, k| h[k] = Fluent::MultiEventStream.new }}if @batch_mode
|
93
|
+
|
70
94
|
es.each do |time, record|
|
71
|
-
rewrited_tag = rewrite_tag(tag, record)
|
72
|
-
|
73
|
-
|
95
|
+
rewrited_tag, rewrited_label = rewrite_tag(tag, record, placeholder)
|
96
|
+
if (rewrited_tag.nil? || tag == rewrited_tag) && rewrited_label.nil?
|
97
|
+
log.trace("rewrite_tag_filter: tag has not been rewritten", record)
|
98
|
+
next
|
99
|
+
end
|
100
|
+
rewrited_tag = tag if rewrited_tag.nil?
|
101
|
+
if new_event_streams.nil?
|
102
|
+
get_router(rewrited_label).emit(rewrited_tag, time, record)
|
103
|
+
else
|
104
|
+
new_event_streams[rewrited_label][rewrited_tag].add(time, record)
|
105
|
+
end
|
106
|
+
end
|
107
|
+
if !new_event_streams.nil?
|
108
|
+
new_event_streams.each do |rewrited_label, label_event_streams |
|
109
|
+
labeled_router = get_router(rewrited_label)
|
110
|
+
label_event_streams.each do |rewrited_tag, new_es|
|
111
|
+
labeled_router.emit_stream(rewrited_tag, new_es)
|
112
|
+
end
|
113
|
+
end
|
74
114
|
end
|
75
115
|
end
|
76
116
|
|
77
|
-
def rewrite_tag(tag, record)
|
78
|
-
|
79
|
-
@rewriterules.each do |record_accessor, regexp, match_operator, rewritetag|
|
117
|
+
def rewrite_tag(tag, record, placeholder)
|
118
|
+
@rewriterules.each do |record_accessor, regexp, match_operator, rewritetag, rewritelabel|
|
80
119
|
rewritevalue = record_accessor.call(record).to_s
|
81
120
|
next if rewritevalue.empty? && match_operator != MATCH_OPERATOR_EXCLUDE
|
82
121
|
last_match = regexp_last_match(regexp, rewritevalue)
|
@@ -92,9 +131,9 @@ class Fluent::Plugin::RewriteTagFilterOutput < Fluent::Plugin::Output
|
|
92
131
|
log.warn "rewrite_tag_filter: unknown placeholder found. :placeholder=>#{$1} :tag=>#{tag} :rewritetag=>#{rewritetag}" unless placeholder.include?($1)
|
93
132
|
placeholder[$1]
|
94
133
|
end
|
95
|
-
return rewritetag
|
134
|
+
return rewritetag, rewritelabel
|
96
135
|
end
|
97
|
-
return nil
|
136
|
+
return nil, nil
|
98
137
|
end
|
99
138
|
|
100
139
|
def regexp_last_match(regexp, rewritevalue)
|
@@ -114,7 +153,7 @@ class Fluent::Plugin::RewriteTagFilterOutput < Fluent::Plugin::Output
|
|
114
153
|
end
|
115
154
|
|
116
155
|
def get_placeholder(tag)
|
117
|
-
tag = tag.sub(@
|
156
|
+
tag = tag.sub(@remove_tag_regexp, '') if @remove_tag_regexp
|
118
157
|
|
119
158
|
result = {
|
120
159
|
'__HOSTNAME__' => @hostname,
|
data/test/helper.rb
CHANGED
@@ -1,9 +1,13 @@
|
|
1
1
|
require 'bundler/setup'
|
2
2
|
require 'test/unit'
|
3
|
+
require 'test/unit/rr'
|
3
4
|
|
4
5
|
$LOAD_PATH.unshift(File.join(__dir__, '..', 'lib'))
|
5
6
|
$LOAD_PATH.unshift(__dir__)
|
6
7
|
require 'fluent/test'
|
7
8
|
require 'fluent/test/driver/output'
|
9
|
+
require 'fluent/test/helpers'
|
10
|
+
|
11
|
+
Test::Unit::TestCase.include(Fluent::Test::Helpers)
|
8
12
|
|
9
13
|
require 'fluent/plugin/out_rewrite_tag_filter'
|
@@ -17,6 +17,45 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
|
|
17
17
|
create_driver(conf)
|
18
18
|
end
|
19
19
|
end
|
20
|
+
|
21
|
+
test "pattern with character classes" do
|
22
|
+
conf = %[
|
23
|
+
<rule>
|
24
|
+
key $['email']['domain']
|
25
|
+
pattern /[sv]d[a-z]+\\d*$/
|
26
|
+
tag $2.$1
|
27
|
+
</rule>
|
28
|
+
]
|
29
|
+
d = create_driver(conf)
|
30
|
+
assert_equal(/[sv]d[a-z]+\d*$/, d.instance.rules.first.pattern)
|
31
|
+
end
|
32
|
+
|
33
|
+
test "pattern w/o slashes" do
|
34
|
+
conf = %[
|
35
|
+
<rule>
|
36
|
+
key $['email']['domain']
|
37
|
+
pattern .+
|
38
|
+
tag $2.$1
|
39
|
+
</rule>
|
40
|
+
]
|
41
|
+
d = create_driver(conf)
|
42
|
+
assert_equal(/.+/, d.instance.rules.first.pattern)
|
43
|
+
end
|
44
|
+
|
45
|
+
test "remove_tag_prefix and remove_tag_regexp are exclusive" do
|
46
|
+
conf = %[
|
47
|
+
remove_tag_prefix prefix
|
48
|
+
remove_tag_regexp /^prefix\./
|
49
|
+
<rule>
|
50
|
+
key message
|
51
|
+
pattern .+
|
52
|
+
tag ${tag}
|
53
|
+
</rule>
|
54
|
+
]
|
55
|
+
assert_raise(Fluent::ConfigError) do
|
56
|
+
create_driver(conf)
|
57
|
+
end
|
58
|
+
end
|
20
59
|
end
|
21
60
|
|
22
61
|
sub_test_case "section style" do
|
@@ -103,6 +142,30 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
|
|
103
142
|
assert_equal 'access', events[0][0] # tag
|
104
143
|
end
|
105
144
|
|
145
|
+
sub_test_case "remove_tag_regexp" do
|
146
|
+
test "plain" do
|
147
|
+
config = %[
|
148
|
+
remove_tag_regexp /^input\.(apache|nginx)\./
|
149
|
+
<rule>
|
150
|
+
key domain
|
151
|
+
pattern ^www\.google\.com$
|
152
|
+
tag rewritten.${tag}
|
153
|
+
</rule>
|
154
|
+
]
|
155
|
+
d = create_driver(config)
|
156
|
+
d.run do
|
157
|
+
d.feed('input.apache.access', event_time, {'domain' => 'www.google.com', 'path' => '/foo/bar?key=value', 'agent' => 'Googlebot', 'response_time' => 1000000})
|
158
|
+
d.feed('input.nginx.access', event_time, {'domain' => 'www.google.com', 'path' => '/foo/bar?key=value', 'agent' => 'Googlebot', 'response_time' => 1000000})
|
159
|
+
d.feed('input.tomcat.access', event_time, {'domain' => 'www.google.com', 'path' => '/foo/bar?key=value', 'agent' => 'Googlebot', 'response_time' => 1000000})
|
160
|
+
end
|
161
|
+
events = d.events
|
162
|
+
assert_equal 3, events.length
|
163
|
+
assert_equal 'rewritten.access', events[0][0]
|
164
|
+
assert_equal 'rewritten.access', events[1][0]
|
165
|
+
assert_equal 'rewritten.input.tomcat.access', events[2][0]
|
166
|
+
end
|
167
|
+
end
|
168
|
+
|
106
169
|
test "short hostname" do
|
107
170
|
config = %[
|
108
171
|
remove_tag_prefix input
|
@@ -260,5 +323,166 @@ class RewriteTagFilterOutputTest < Test::Unit::TestCase
|
|
260
323
|
events = d.events
|
261
324
|
assert_equal "com.example", events[0][0]
|
262
325
|
end
|
326
|
+
|
327
|
+
test "tag has not been rewritten and log_level trace" do
|
328
|
+
conf = %[
|
329
|
+
@log_level trace
|
330
|
+
<rule>
|
331
|
+
key $['email']['domain']
|
332
|
+
pattern ^(example)\.(com)$
|
333
|
+
tag $2.$1
|
334
|
+
</rule>
|
335
|
+
]
|
336
|
+
d = create_driver(conf)
|
337
|
+
d.run(default_tag: "input") do
|
338
|
+
d.feed({ "email" => { "localpart" => "john", "domain" => "example.com" }})
|
339
|
+
d.feed({ "email" => { "localpart" => "doe", "domain" => "example.jp" }})
|
340
|
+
end
|
341
|
+
events = d.events
|
342
|
+
assert_equal(1, events.size)
|
343
|
+
log = d.logs.grep(/\[trace\]/).first
|
344
|
+
assert_equal('rewrite_tag_filter: tag has not been rewritten email={"localpart"=>"doe", "domain"=>"example.jp"}',
|
345
|
+
log.slice(/\[trace\]: (.+)$/, 1))
|
346
|
+
assert_equal "com.example", events[0][0]
|
347
|
+
end
|
348
|
+
|
349
|
+
test "get_router" do
|
350
|
+
conf = %[
|
351
|
+
<rule>
|
352
|
+
key key
|
353
|
+
pattern /^(odd|even)$/
|
354
|
+
tag $1
|
355
|
+
label new_label
|
356
|
+
</rule>
|
357
|
+
<rule>
|
358
|
+
key key
|
359
|
+
pattern /^(.*)$/
|
360
|
+
tag $1
|
361
|
+
</rule>
|
362
|
+
]
|
363
|
+
time = event_time
|
364
|
+
d = create_driver(conf)
|
365
|
+
assert_equal(d.instance.router, d.instance.get_router(nil))
|
366
|
+
assert_equal(d.instance.router, d.instance.get_router(""))
|
367
|
+
new_label_router = d.instance.get_router("new_label")
|
368
|
+
refute_equal(d.instance.router, new_label_router)
|
369
|
+
end
|
370
|
+
|
371
|
+
test "relabel" do
|
372
|
+
conf = %[
|
373
|
+
emit_mode record
|
374
|
+
<rule>
|
375
|
+
key key
|
376
|
+
pattern /^(odd)$/
|
377
|
+
tag $1
|
378
|
+
label odd_label
|
379
|
+
</rule>
|
380
|
+
<rule>
|
381
|
+
key key
|
382
|
+
pattern /^(even)$/
|
383
|
+
tag ${tag}
|
384
|
+
label even_label
|
385
|
+
</rule>
|
386
|
+
<rule>
|
387
|
+
key key
|
388
|
+
pattern /^(.*)$/
|
389
|
+
tag $1
|
390
|
+
</rule>
|
391
|
+
]
|
392
|
+
time = event_time
|
393
|
+
d = create_driver(conf)
|
394
|
+
# Router only called for default label
|
395
|
+
mock.proxy(d.instance.router).emit(anything, anything, anything).times(2)
|
396
|
+
mock.proxy(d.instance).get_router("odd_label").times(2)
|
397
|
+
mock.proxy(d.instance).get_router("even_label").times(2)
|
398
|
+
mock.proxy(d.instance).get_router(nil).times(2)
|
399
|
+
mock.proxy(d.instance.router).emit_stream(anything, anything).times(0)
|
400
|
+
d.run(default_tag: "input") do
|
401
|
+
d.feed([[time, { "key" => "odd", "message" => "message-1" }],
|
402
|
+
[time, { "key" => "even", "message" => "message-2" }],
|
403
|
+
[time, { "key" => "zero", "message" => "message-3" }],
|
404
|
+
[time, { "key" => "odd", "message" => "message-4" }],
|
405
|
+
[time, { "key" => "even", "message" => "message-5" }],
|
406
|
+
[time, { "key" => "zero", "message" => "message-6" }]])
|
407
|
+
end
|
408
|
+
events = d.events
|
409
|
+
expected_events = [
|
410
|
+
["odd", time, { "key" => "odd", "message" => "message-1" }],
|
411
|
+
["input", time, { "key" => "even", "message" => "message-2" }],
|
412
|
+
["zero", time, { "key" => "zero", "message" => "message-3" }],
|
413
|
+
["odd", time, { "key" => "odd", "message" => "message-4" }],
|
414
|
+
["input", time, { "key" => "even", "message" => "message-5" }],
|
415
|
+
["zero", time, { "key" => "zero", "message" => "message-6" }],
|
416
|
+
]
|
417
|
+
assert_equal(events, expected_events)
|
418
|
+
end
|
419
|
+
|
420
|
+
sub_test_case "emit_mode" do
|
421
|
+
test "record" do
|
422
|
+
conf = %[
|
423
|
+
emit_mode record
|
424
|
+
<rule>
|
425
|
+
key key
|
426
|
+
pattern /^(odd|even)$/
|
427
|
+
tag $1
|
428
|
+
</rule>
|
429
|
+
]
|
430
|
+
time = event_time
|
431
|
+
d = create_driver(conf)
|
432
|
+
mock.proxy(d.instance.router).emit(anything, anything, anything).times(6)
|
433
|
+
mock.proxy(d.instance.router).emit_stream(anything, anything).times(0)
|
434
|
+
d.run(default_tag: "input") do
|
435
|
+
d.feed([[time, { "key" => "odd", "message" => "message-1" }],
|
436
|
+
[time, { "key" => "even", "message" => "message-2" }],
|
437
|
+
[time, { "key" => "odd", "message" => "message-3" }],
|
438
|
+
[time, { "key" => "even", "message" => "message-4" }],
|
439
|
+
[time, { "key" => "odd", "message" => "message-5" }],
|
440
|
+
[time, { "key" => "even", "message" => "message-6" }]])
|
441
|
+
end
|
442
|
+
events = d.events
|
443
|
+
expected_events = [
|
444
|
+
["odd", time, { "key" => "odd", "message" => "message-1" }],
|
445
|
+
["even", time, { "key" => "even", "message" => "message-2" }],
|
446
|
+
["odd", time, { "key" => "odd", "message" => "message-3" }],
|
447
|
+
["even", time, { "key" => "even", "message" => "message-4" }],
|
448
|
+
["odd", time, { "key" => "odd", "message" => "message-5" }],
|
449
|
+
["even", time, { "key" => "even", "message" => "message-6" }],
|
450
|
+
]
|
451
|
+
assert_equal(events, expected_events)
|
452
|
+
end
|
453
|
+
|
454
|
+
test "batch" do
|
455
|
+
conf = %[
|
456
|
+
emit_mode batch
|
457
|
+
<rule>
|
458
|
+
key key
|
459
|
+
pattern /^(odd|even)$/
|
460
|
+
tag $1
|
461
|
+
</rule>
|
462
|
+
]
|
463
|
+
time = event_time
|
464
|
+
d = create_driver(conf)
|
465
|
+
mock.proxy(d.instance.router).emit(anything, anything, anything).times(0)
|
466
|
+
mock.proxy(d.instance.router).emit_stream(anything, anything).times(2)
|
467
|
+
d.run(default_tag: "input") do
|
468
|
+
d.feed([[time, { "key" => "odd", "message" => "message-1" }],
|
469
|
+
[time, { "key" => "even", "message" => "message-2" }],
|
470
|
+
[time, { "key" => "odd", "message" => "message-3" }],
|
471
|
+
[time, { "key" => "even", "message" => "message-4" }],
|
472
|
+
[time, { "key" => "odd", "message" => "message-5" }],
|
473
|
+
[time, { "key" => "even", "message" => "message-6" }]])
|
474
|
+
end
|
475
|
+
events = d.events
|
476
|
+
expected_records = [
|
477
|
+
["odd", time, { "key" => "odd", "message" => "message-1" }],
|
478
|
+
["odd", time, { "key" => "odd", "message" => "message-3" }],
|
479
|
+
["odd", time, { "key" => "odd", "message" => "message-5" }],
|
480
|
+
["even", time, { "key" => "even", "message" => "message-2" }],
|
481
|
+
["even", time, { "key" => "even", "message" => "message-4" }],
|
482
|
+
["even", time, { "key" => "even", "message" => "message-6" }],
|
483
|
+
]
|
484
|
+
assert_equal(events, expected_records)
|
485
|
+
end
|
486
|
+
end
|
263
487
|
end
|
264
488
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-rewrite-tag-filter
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Kentaro Yoshida
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-12-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: test-unit
|
@@ -24,6 +24,20 @@ dependencies:
|
|
24
24
|
- - ">="
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: 3.1.0
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: test-unit-rr
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '0'
|
34
|
+
type: :development
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '0'
|
27
41
|
- !ruby/object:Gem::Dependency
|
28
42
|
name: rake
|
29
43
|
requirement: !ruby/object:Gem::Requirement
|
@@ -42,16 +56,36 @@ dependencies:
|
|
42
56
|
name: fluentd
|
43
57
|
requirement: !ruby/object:Gem::Requirement
|
44
58
|
requirements:
|
45
|
-
- - "
|
59
|
+
- - ">="
|
46
60
|
- !ruby/object:Gem::Version
|
47
61
|
version: 0.14.2
|
62
|
+
- - "<"
|
63
|
+
- !ruby/object:Gem::Version
|
64
|
+
version: '2'
|
48
65
|
type: :runtime
|
49
66
|
prerelease: false
|
50
67
|
version_requirements: !ruby/object:Gem::Requirement
|
51
68
|
requirements:
|
52
|
-
- - "
|
69
|
+
- - ">="
|
53
70
|
- !ruby/object:Gem::Version
|
54
71
|
version: 0.14.2
|
72
|
+
- - "<"
|
73
|
+
- !ruby/object:Gem::Version
|
74
|
+
version: '2'
|
75
|
+
- !ruby/object:Gem::Dependency
|
76
|
+
name: fluent-config-regexp-type
|
77
|
+
requirement: !ruby/object:Gem::Requirement
|
78
|
+
requirements:
|
79
|
+
- - ">="
|
80
|
+
- !ruby/object:Gem::Version
|
81
|
+
version: '0'
|
82
|
+
type: :runtime
|
83
|
+
prerelease: false
|
84
|
+
version_requirements: !ruby/object:Gem::Requirement
|
85
|
+
requirements:
|
86
|
+
- - ">="
|
87
|
+
- !ruby/object:Gem::Version
|
88
|
+
version: '0'
|
55
89
|
description:
|
56
90
|
email:
|
57
91
|
- y.ken.studio@gmail.com
|
@@ -90,8 +124,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
90
124
|
- !ruby/object:Gem::Version
|
91
125
|
version: '0'
|
92
126
|
requirements: []
|
93
|
-
|
94
|
-
rubygems_version: 2.6.13
|
127
|
+
rubygems_version: 3.1.4
|
95
128
|
signing_key:
|
96
129
|
specification_version: 4
|
97
130
|
summary: Fluentd Output filter plugin. It has designed to rewrite tag like mod_rewrite.
|