fluent-plugin-grok-parser 2.1.6 → 2.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1bc85cddf3ae82876f8fa683d15d4554a2c83a1f077a8fc46ece9e6d14f3ec3d
4
- data.tar.gz: 4b4321c7a01f13be984bcb853a8048dbe6e590b73980984e122be0407b0a37aa
3
+ metadata.gz: b7652c7a0665b1f6d46aa18bd2e1b373d45facfebe5d0343ab83e34fda23b74f
4
+ data.tar.gz: 9a124bd9fedd164691d1eedc434d3f29dbfcddc8fe0dba94cdc5babe05185194
5
5
  SHA512:
6
- metadata.gz: 1fe9e8af7dcfec87f6fbdb1d0e7f70aaeed3a1ae930810e58f0eeb19d39dbb21bceb8dbf8260ee58793711abd2dda9a7ac1247f7db300e0f69865fc60cbbb345
7
- data.tar.gz: d1cfba449d9fe5747b2746dd0d3a9e3ad8aff05560c85f6f5630238f46f63493dc3645f24d7d2d484dff468723b7c79ba407e92d74fe8e00b867639003645f32
6
+ metadata.gz: 60b36ca6f266b804b95f2e325c16a3e2ff003d75f5b49826b5bb12fc4160b32dccf7b540b9741b5a9a58a8996afbb7dcdad3d66679369ba254a27a8e4e73a6bc
7
+ data.tar.gz: c45fa99fcb9cd319b6d9d7956d7db94a0c2f40ed8e0919d7807cec51139c7fe845fdbf0c852778735ff45ed2f74bc5b78c2d59c81eef6951f4718fa38c6afc38
@@ -2,8 +2,8 @@ sudo: false
2
2
  language: ruby
3
3
 
4
4
  rvm:
5
- - 2.1
6
- - 2.2
7
- - 2.3.3
8
- - 2.4.0
5
+ - 2.2.10
6
+ - 2.3.7
7
+ - 2.4.4
8
+ - 2.5.1
9
9
 
data/README.md CHANGED
@@ -6,8 +6,8 @@ This is a Fluentd plugin to enable Logstash's Grok-like parsing logic.
6
6
 
7
7
  | fluent-plugin-grok-parser | fluentd | ruby |
8
8
  |---------------------------|------------|--------|
9
- | >= 1.0.0 | >= v0.14.0 | >= 2.1 |
10
- | < 1.0.0 | >= v0.12.0 | >= 1.9 |
9
+ | >= 2.0.0 | >= v0.14.0 | >= 2.1 |
10
+ | < 2.0.0 | >= v0.12.0 | >= 1.9 |
11
11
 
12
12
 
13
13
  ## What's Grok?
@@ -33,18 +33,6 @@ extracts the first IP address that matches in the log.
33
33
  </source>
34
34
  ```
35
35
 
36
- For Fluentd v0.12, use following style:
37
-
38
- ```aconf
39
- <source>
40
- @type tail
41
- path /path/to/log
42
- tag grokked_log
43
- format grok
44
- grok_pattern %{IP:ip_address}
45
- </source>
46
- ```
47
-
48
36
  **If you want to try multiple grok patterns and use the first matched one**, you can use the following syntax:
49
37
 
50
38
  ```aconf
@@ -68,27 +56,6 @@ For Fluentd v0.12, use following style:
68
56
  </source>
69
57
  ```
70
58
 
71
- For Fluentd v0.12, use following style:
72
-
73
- ```aconf
74
- <source>
75
- @type tail
76
- path /path/to/log
77
- tag grokked_log
78
- format grok
79
- <grok>
80
- pattern %{COMBINEDAPACHELOG}
81
- time_format "%d/%b/%Y:%H:%M:%S %z"
82
- </grok>
83
- <grok>
84
- pattern %{IP:ip_address}
85
- </grok>
86
- <grok>
87
- pattern %{GREEDYDATA:message}
88
- </grok>
89
- </source>
90
- ```
91
-
92
59
  ### Multiline support
93
60
 
94
61
  You can parse multiple line text.
@@ -106,19 +73,6 @@ You can parse multiple line text.
106
73
  </source>
107
74
  ```
108
75
 
109
- For Fluentd v0.12, use following style:
110
-
111
- ```aconf
112
- <source>
113
- @type tail
114
- path /path/to/log
115
- format multiline_grok
116
- grok_pattern %{IP:ip_address}%{GREEDYDATA:message}
117
- multiline_start_regexp /^[^\s]/
118
- tag grokked_log
119
- </source>
120
- ```
121
-
122
76
  You can use multiple grok patterns to parse your data.
123
77
 
124
78
  ```aconf
@@ -135,41 +89,25 @@ You can use multiple grok patterns to parse your data.
135
89
  </source>
136
90
  ```
137
91
 
138
- For Fluentd v0.12, use following style:
139
-
140
- ```aconf
141
- <source>
142
- @type tail
143
- path /path/to/log
144
- format multiline_grok
145
- <grok>
146
- pattern Started %{WORD:verb} "%{URIPATH:pathinfo}" for %{IP:ip} at %{TIMESTAMP_ISO8601:timestamp}\nProcessing by %{WORD:controller}#%{WORD:action} as %{WORD:format}%{DATA:message}Completed %{NUMBER:response} %{WORD} in %{NUMBER:elapsed} (%{DATA:elapsed_details})
147
- </grok>
148
- tag grokked_log
149
- </source>
150
- ```
151
-
152
92
  Fluentd accumulates data in the buffer forever to parse complete data when no pattern matches.
153
93
 
154
94
  You can use this parser without `multiline_start_regexp` when you know your data structure perfectly.
155
95
 
156
96
  ## Configurations
157
97
 
158
- **time_format**
159
-
160
- The format of the time field.
161
-
162
- **grok_pattern**
98
+ * See also: [TimeParameters Plugin Overview](https://docs.fluentd.org/v1.0/articles/timeparameters-plugin-overview)
99
+ * See also: [Parser Plugin Overview](https://docs.fluentd.org/v1.0/articles/parser-plugin-overview)
163
100
 
164
- The pattern of grok. You cannot specify multiple grok pattern with this.
101
+ * **time_format** (string) (optional): The format of the time field.
102
+ * **grok_pattern** (string) (optional): The pattern of grok. You cannot specify multiple grok pattern with this.
103
+ * **custom_pattern_path** (string) (optional): Path to the file that includes custom grok patterns
104
+ * **grok_failure_key** (string) (optional): The key has grok failure reason.
105
+ * **grok_name_key** (string) (optional): The key name to store grok section's name
106
+ * **multi_line_start_regexp** (string) (optional): The regexp to match beginning of multiline. This is only for "multiline_grok".
165
107
 
166
- **custom_pattern_path**
108
+ ## Examples
167
109
 
168
- Path to the file that includes custom grok patterns
169
-
170
- **grok_failure_key**
171
-
172
- The key has grok failure reason. Default is `nil`.
110
+ ### Using grok\_failure\_key
173
111
 
174
112
  ```aconf
175
113
  <source>
@@ -209,21 +147,42 @@ This generates following events:
209
147
  2016-11-28 13:07:09.010400923 +0900 dummy.log: {"message1":"/","prog":"bar","path":"/"}
210
148
  ```
211
149
 
212
-
213
- **grok/pattern**
214
-
215
- Section for grok patterns. You can use multiple grok patterns with
216
- multiple `<grok>` sections.
150
+ ### Using grok\_name\_key
217
151
 
218
152
  ```aconf
219
- <grok>
220
- pattern %{IP:ipaddress}
221
- </grok>
153
+ <source>
154
+ @type tail
155
+ path /path/to/log
156
+ tag grokked_log
157
+ grok_name_key grok_name
158
+ grok_failure_key grokfailure
159
+ <parse>
160
+ @type grok
161
+ <grok>
162
+ name apache_log
163
+ pattern %{COMBINEDAPACHELOG}
164
+ time_format "%d/%b/%Y:%H:%M:%S %z"
165
+ </grok>
166
+ <grok>
167
+ name ip_address
168
+ pattern %{IP:ip_address}
169
+ </grok>
170
+ <grok>
171
+ name rest_message
172
+ pattern %{GREEDYDATA:message}
173
+ </grok>
174
+ </parse>
175
+ </source>
222
176
  ```
223
177
 
224
- **multiline_start_regexp**
178
+ This will add keys like following:
179
+
180
+ * Add `grok_name: "apache_log"` if the record matches `COMBINEDAPACHELOG`
181
+ * Add `grok_name: "ip_address"` if the record matches `IP`
182
+ * Add `grok_name: "rest_message"` if the record matches `GREEDYDATA`
225
183
 
226
- The regexp to match beginning of multiline. This is only for "multiline_grok".
184
+ Add `grokfailure` key to the record if the record does not match any grok pattern.
185
+ See also test code for more details.
227
186
 
228
187
  ## How to write Grok patterns
229
188
 
@@ -310,7 +269,7 @@ Here is a sample config using the Grok parser with `in_tail` and the `types` par
310
269
 
311
270
  ## Notice
312
271
 
313
- If you want to use this plugin with Fluentd v0.12.x or earlier, you can use this plugin version v1.0.0.
272
+ If you want to use this plugin with Fluentd v0.12.x or earlier, you can use this plugin version v1.x.
314
273
 
315
274
  See also: [Plugin Management | Fluentd](http://docs.fluentd.org/articles/plugin-management#plugin-version-management)
316
275
 
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |spec|
6
6
  spec.name = "fluent-plugin-grok-parser"
7
- spec.version = "2.1.6"
7
+ spec.version = "2.2.0"
8
8
  spec.authors = ["kiyoto", "Kenji Okimoto"]
9
9
  spec.email = ["kiyoto@treasure-data.com", "okimoto@clear-code.com"]
10
10
  spec.summary = %q{Fluentd plugin to support Logstash-inspired Grok format for parsing logs}
@@ -23,14 +23,15 @@ module Fluent
23
23
 
24
24
  def initialize(plugin, conf)
25
25
  @pattern_map = {}
26
- @parsers = []
26
+ @parsers = {}
27
27
  @multiline_mode = false
28
28
  @conf = conf
29
- if plugin.respond_to?(:firstline?)
29
+ @plugin = plugin
30
+ if @plugin.respond_to?(:firstline?)
30
31
  @multiline_mode = true
31
32
  end
32
- if @conf["multiline_start_regexp"]
33
- @multiline_start_regexp = Regexp.compile(@conf["multiline_start_regexp"][1..-2])
33
+ if @plugin.respond_to?(:multiline_start_regexp) && @plugin.multiline_start_regexp
34
+ @multiline_start_regexp = Regexp.compile(@plugin.multiline_start_regexp[1..-2])
34
35
  end
35
36
  end
36
37
 
@@ -43,15 +44,16 @@ module Fluent
43
44
  end
44
45
 
45
46
  def setup
46
- if @conf["grok_pattern"]
47
- @parsers << expand_pattern_expression(@conf["grok_pattern"], @conf)
47
+ if @plugin.grok_pattern
48
+ @parsers[:grok_pattern] = expand_pattern_expression(@plugin.grok_pattern, @conf)
48
49
  else
49
- grok_confs = @conf.elements.select {|e| e.name == "grok"}
50
- grok_confs.each do |grok_conf|
51
- @parsers << expand_pattern_expression(grok_conf["pattern"], grok_conf)
50
+ @plugin.grok_confs.each.with_index do |grok_conf, index|
51
+ @parsers[grok_conf.name || index] = expand_pattern_expression(grok_conf.pattern, grok_conf)
52
52
  end
53
53
  end
54
- @parsers.compact!
54
+ @parsers.reject! do |key, parser|
55
+ parser.nil?
56
+ end
55
57
  if @parsers.empty?
56
58
  raise Fluent::ConfigError, 'no grok patterns. Check configuration, e.g. typo, configuration syntax, etc'
57
59
  end
@@ -61,11 +63,12 @@ module Fluent
61
63
 
62
64
  def expand_pattern_expression(grok_pattern, conf)
63
65
  regexp, types = expand_pattern(grok_pattern)
64
- $log.info "Expanded the pattern #{conf['grok_pattern']} into #{regexp}"
66
+ $log.info "Expanded the pattern #{grok_pattern} into #{regexp}"
67
+ _conf = conf.to_h
65
68
  unless types.empty?
66
- conf["types"] = types.map{|subname,type| "#{subname}:#{type}" }.join(",")
69
+ _conf["types"] = types.map{|subname,type| "#{subname}:#{type}" }.join(",")
67
70
  end
68
- _conf = conf.merge("expression" => regexp, "multiline" => @multiline_mode)
71
+ _conf = _conf.merge("expression" => regexp, "multiline" => @multiline_mode)
69
72
  config = Fluent::Config::Element.new("parse", nil, _conf, [])
70
73
  parser = Fluent::Plugin::RegexpParser.new
71
74
  parser.configure(config)
@@ -14,6 +14,15 @@ module Fluent
14
14
  config_param :custom_pattern_path, :string, default: nil
15
15
  desc "The key has grok failure reason"
16
16
  config_param :grok_failure_key, :string, default: nil
17
+ desc "The key name to store grok section's name"
18
+ config_param :grok_name_key, :string, default: nil
19
+
20
+ config_section :grok, param_name: "grok_confs", multi: true do
21
+ desc "The name of this grok section"
22
+ config_param :name, :string, default: nil
23
+ desc "The pattern of grok"
24
+ config_param :pattern, :string
25
+ end
17
26
 
18
27
  def initialize
19
28
  super
@@ -44,9 +53,10 @@ module Fluent
44
53
  end
45
54
 
46
55
  def parse(text)
47
- @grok.parsers.each do |parser|
56
+ @grok.parsers.each do |name_or_index, parser|
48
57
  parser.parse(text) do |time, record|
49
58
  if time and record
59
+ record[@grok_name_key] = name_or_index if @grok_name_key
50
60
  yield time, record
51
61
  return
52
62
  end
@@ -17,9 +17,10 @@ module Fluent
17
17
  end
18
18
 
19
19
  def parse(text)
20
- @grok.parsers.each do |parser|
20
+ @grok.parsers.each do |name_or_index, parser|
21
21
  parser.parse(text) do |time, record|
22
22
  if time and record
23
+ record[@grok_name_key] = name_or_index if @grok_name_key
23
24
  yield time, record
24
25
  return
25
26
  end
@@ -11,49 +11,49 @@ def str2time(str_time, format = nil)
11
11
  end
12
12
 
13
13
  class GrokParserTest < ::Test::Unit::TestCase
14
- def setup
14
+ setup do
15
15
  Fluent::Test.setup
16
16
  end
17
17
 
18
- class Timestamp < self
19
- def test_timestamp_iso8601
18
+ sub_test_case "timestamp" do
19
+ test "timestamp iso8601" do
20
20
  internal_test_grok_pattern("%{TIMESTAMP_ISO8601:time}", "Some stuff at 2014-01-01T00:00:00+0900",
21
21
  event_time("2014-01-01T00:00:00+0900"), {})
22
22
  end
23
23
 
24
- def test_datestamp_rfc822_with_zone
24
+ test "datestamp rfc822 with zone" do
25
25
  internal_test_grok_pattern("%{DATESTAMP_RFC822:time}", "Some stuff at Mon Aug 15 2005 15:52:01 UTC",
26
26
  event_time("Mon Aug 15 2005 15:52:01 UTC"), {})
27
27
  end
28
28
 
29
- def test_datestamp_rfc822_with_numeric_zone
29
+ test "datestamp rfc822 with numeric zone" do
30
30
  internal_test_grok_pattern("%{DATESTAMP_RFC2822:time}", "Some stuff at Mon, 15 Aug 2005 15:52:01 +0000",
31
31
  event_time("Mon, 15 Aug 2005 15:52:01 +0000"), {})
32
32
  end
33
33
 
34
- def test_syslogtimestamp
34
+ test "syslogtimestamp" do
35
35
  internal_test_grok_pattern("%{SYSLOGTIMESTAMP:time}", "Some stuff at Aug 01 00:00:00",
36
36
  event_time("Aug 01 00:00:00"), {})
37
37
  end
38
38
  end
39
39
 
40
- def test_date
40
+ test "date" do
41
41
  internal_test_grok_pattern("\\[(?<date>%{DATE} %{TIME} (?:AM|PM))\\]", "[2/16/2018 10:19:34 AM]",
42
42
  nil, { "date" => "2/16/2018 10:19:34 AM" })
43
43
  end
44
44
 
45
- def test_call_for_grok_pattern_not_found
45
+ test "grok pattern not found" do
46
46
  assert_raise Fluent::Grok::GrokPatternNotFoundError do
47
47
  internal_test_grok_pattern("%{THIS_PATTERN_DOESNT_EXIST}", "Some stuff at somewhere", nil, {})
48
48
  end
49
49
  end
50
50
 
51
- def test_call_for_multiple_fields
51
+ test "multiple fields" do
52
52
  internal_test_grok_pattern("%{MAC:mac_address} %{IP:ip_address}", "this.wont.match DEAD.BEEF.1234 127.0.0.1", nil,
53
53
  {"mac_address" => "DEAD.BEEF.1234", "ip_address" => "127.0.0.1"})
54
54
  end
55
55
 
56
- def test_call_for_complex_pattern
56
+ test "complex pattern" do
57
57
  internal_test_grok_pattern("%{COMBINEDAPACHELOG}", '127.0.0.1 192.168.0.1 - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0"',
58
58
  str2time("28/Feb/2013:12:00:00 +0900", "%d/%b/%Y:%H:%M:%S %z"),
59
59
  {
@@ -73,59 +73,59 @@ class GrokParserTest < ::Test::Unit::TestCase
73
73
  )
74
74
  end
75
75
 
76
- def test_call_for_custom_pattern
76
+ test "custom pattern" do
77
77
  internal_test_grok_pattern("%{MY_AWESOME_PATTERN:message}", "this is awesome",
78
78
  nil, {"message" => "this is awesome"},
79
79
  "custom_pattern_path" => fixtures("my_pattern").to_s)
80
80
  end
81
81
 
82
- class OptionalType < self
83
- def test_simple
82
+ sub_test_case "OptionalType" do
83
+ test "simple" do
84
84
  internal_test_grok_pattern("%{INT:user_id:integer} paid %{NUMBER:paid_amount:float}",
85
85
  "12345 paid 6789.10", nil,
86
86
  {"user_id" => 12345, "paid_amount" => 6789.1 })
87
87
  end
88
88
 
89
- def test_array
89
+ test "array" do
90
90
  internal_test_grok_pattern("%{GREEDYDATA:message:array}",
91
91
  "a,b,c,d", nil,
92
92
  {"message" => %w(a b c d)})
93
93
  end
94
94
 
95
- def test_array_with_delimiter
95
+ test "array with delimiter" do
96
96
  internal_test_grok_pattern("%{GREEDYDATA:message:array:|}",
97
97
  "a|b|c|d", nil,
98
98
  {"message" => %w(a b c d)})
99
99
  end
100
100
 
101
- def test_timestamp_iso8601
101
+ test "timestamp iso8601" do
102
102
  internal_test_grok_pattern("%{TIMESTAMP_ISO8601:stamp:time}", "Some stuff at 2014-01-01T00:00:00+0900",
103
103
  nil, {"stamp" => event_time("2014-01-01T00:00:00+0900")})
104
104
  end
105
105
 
106
- def test_datestamp_rfc822_with_zone
106
+ test "datestamp rfc822 with zone" do
107
107
  internal_test_grok_pattern("%{DATESTAMP_RFC822:stamp:time}", "Some stuff at Mon Aug 15 2005 15:52:01 UTC",
108
108
  nil, {"stamp" => event_time("Mon Aug 15 2005 15:52:01 UTC")})
109
109
  end
110
110
 
111
- def test_datestamp_rfc822_with_numeric_zone
111
+ test "datestamp rfc822 with numeric zone" do
112
112
  internal_test_grok_pattern("%{DATESTAMP_RFC2822:stamp:time}", "Some stuff at Mon, 15 Aug 2005 15:52:01 +0000",
113
113
  nil, {"stamp" => event_time("Mon, 15 Aug 2005 15:52:01 +0000")})
114
114
  end
115
115
 
116
- def test_syslogtimestamp
116
+ test "syslogtimestamp" do
117
117
  internal_test_grok_pattern("%{SYSLOGTIMESTAMP:stamp:time}", "Some stuff at Aug 01 00:00:00",
118
118
  nil, {"stamp" => event_time("Aug 01 00:00:00")})
119
119
  end
120
120
 
121
- def test_timestamp_with_format
121
+ test "timestamp with format" do
122
122
  internal_test_grok_pattern("%{TIMESTAMP_ISO8601:stamp:time:%Y-%m-%d %H%M}", "Some stuff at 2014-01-01 1000",
123
123
  nil, {"stamp" => event_time("2014-01-01 10:00")})
124
124
  end
125
125
  end
126
126
 
127
- class NoGrokPatternMatched < self
128
- def test_with_grok_failure_key
127
+ sub_test_case "NoGrokPatternMatched" do
128
+ test "with grok_failure_key" do
129
129
  config = %[
130
130
  grok_failure_key grok_failure
131
131
  <grok>
@@ -142,7 +142,7 @@ class GrokParserTest < ::Test::Unit::TestCase
142
142
  end
143
143
  end
144
144
 
145
- def test_without_grok_failure_key
145
+ test "without grok_failure_key" do
146
146
  config = %[
147
147
  <grok>
148
148
  pattern %{PATH:path}
@@ -158,13 +158,13 @@ class GrokParserTest < ::Test::Unit::TestCase
158
158
  end
159
159
  end
160
160
 
161
- def test_no_grok_patterns
161
+ test "no grok patterns" do
162
162
  assert_raise Fluent::ConfigError do
163
163
  create_driver('')
164
164
  end
165
165
  end
166
166
 
167
- def test_invalid_config_value_type
167
+ test "invalid config value type" do
168
168
  assert_raise Fluent::ConfigError do
169
169
  create_driver(%[
170
170
  <grok>
@@ -174,7 +174,7 @@ class GrokParserTest < ::Test::Unit::TestCase
174
174
  end
175
175
  end
176
176
 
177
- def test_invalid_config_value_type_and_normal_grok_pattern
177
+ test "invalid config value type and normal grok pattern" do
178
178
  d = create_driver(%[
179
179
  <grok>
180
180
  pattern %{PATH:path:foo}
@@ -191,6 +191,116 @@ class GrokParserTest < ::Test::Unit::TestCase
191
191
  assert_equal("unknown value conversion for key:'path', type:'foo'", error_message)
192
192
  end
193
193
 
194
+ sub_test_case "grok_name_key" do
195
+ test "one grok section with name" do
196
+ d = create_driver(%[
197
+ grok_name_key grok_name
198
+ <grok>
199
+ name path
200
+ pattern %{PATH:path}
201
+ </grok>
202
+ ])
203
+ expected = {
204
+ "path" => "/",
205
+ "grok_name" => "path"
206
+ }
207
+ d.instance.parse("/") do |time, record|
208
+ assert_equal(expected, record)
209
+ end
210
+ end
211
+
212
+ test "one grok section without name" do
213
+ d = create_driver(%[
214
+ grok_name_key grok_name
215
+ <grok>
216
+ pattern %{PATH:path}
217
+ </grok>
218
+ ])
219
+ expected = {
220
+ "path" => "/",
221
+ "grok_name" => 0
222
+ }
223
+ d.instance.parse("/") do |time, record|
224
+ assert_equal(expected, record)
225
+ end
226
+ end
227
+
228
+ test "multiple grok sections with name" do
229
+ d = create_driver(%[
230
+ grok_name_key grok_name
231
+ <grok>
232
+ name path
233
+ pattern %{PATH:path}
234
+ </grok>
235
+ <grok>
236
+ name ip
237
+ pattern %{IP:ip_address}
238
+ </grok>
239
+ ])
240
+ expected = [
241
+ { "path" => "/", "grok_name" => "path" },
242
+ { "ip_address" => "127.0.0.1", "grok_name" => "ip" },
243
+ ]
244
+ records = []
245
+ d.instance.parse("/") do |time, record|
246
+ records << record
247
+ end
248
+ d.instance.parse("127.0.0.1") do |time, record|
249
+ records << record
250
+ end
251
+ assert_equal(expected, records)
252
+ end
253
+
254
+ test "multiple grok sections without name" do
255
+ d = create_driver(%[
256
+ grok_name_key grok_name
257
+ <grok>
258
+ pattern %{PATH:path}
259
+ </grok>
260
+ <grok>
261
+ pattern %{IP:ip_address}
262
+ </grok>
263
+ ])
264
+ expected = [
265
+ { "path" => "/", "grok_name" => 0 },
266
+ { "ip_address" => "127.0.0.1", "grok_name" => 1 },
267
+ ]
268
+ records = []
269
+ d.instance.parse("/") do |time, record|
270
+ records << record
271
+ end
272
+ d.instance.parse("127.0.0.1") do |time, record|
273
+ records << record
274
+ end
275
+ assert_equal(expected, records)
276
+ end
277
+
278
+ test "multiple grok sections with both name and index" do
279
+ d = create_driver(%[
280
+ grok_name_key grok_name
281
+ <grok>
282
+ name path
283
+ pattern %{PATH:path}
284
+ </grok>
285
+ <grok>
286
+ pattern %{IP:ip_address}
287
+ </grok>
288
+ ])
289
+ expected = [
290
+ { "path" => "/", "grok_name" => "path" },
291
+ { "ip_address" => "127.0.0.1", "grok_name" => 1 },
292
+ ]
293
+ records = []
294
+ d.instance.parse("/") do |time, record|
295
+ records << record
296
+ end
297
+ d.instance.parse("127.0.0.1") do |time, record|
298
+ records << record
299
+ end
300
+ assert_equal(expected, records)
301
+ end
302
+ end
303
+
194
304
  private
195
305
 
196
306
  def create_driver(conf)
@@ -16,7 +16,7 @@ class TcpInputWithGrokTest < Test::Unit::TestCase
16
16
  end
17
17
  end
18
18
 
19
- def setup
19
+ setup do
20
20
  Fluent::Test.setup
21
21
  end
22
22
 
@@ -54,15 +54,14 @@ class TcpInputWithGrokTest < Test::Unit::TestCase
54
54
  configs[:ipv6] = ["::1", IPv6_CONFIG] if ipv6_enabled?
55
55
  configs
56
56
  end
57
- def test_configure(data)
58
- k, config = data
57
+ test "configure" do |(ip, config)|
59
58
  d = create_driver(config)
60
59
  assert_equal PORT, d.instance.port
61
- assert_equal k, d.instance.bind
60
+ assert_equal ip, d.instance.bind
62
61
  assert_equal "\n", d.instance.delimiter
63
62
  end
64
63
 
65
- def test_grok_pattern
64
+ test "grok_pattern" do
66
65
  tests = [
67
66
  {"msg" => "tcptest1\n", "expected" => "tcptest1"},
68
67
  {"msg" => "tcptest2\n", "expected" => "tcptest2"},
@@ -77,7 +76,7 @@ class TcpInputWithGrokTest < Test::Unit::TestCase
77
76
  internal_test_grok(config, tests)
78
77
  end
79
78
 
80
- def test_grok_pattern_block_config
79
+ test "grok_pattern_block_config" do
81
80
  tests = [
82
81
  {"msg" => "tcptest1\n", "expected" => "tcptest1"},
83
82
  {"msg" => "tcptest2\n", "expected" => "tcptest2"},
@@ -94,7 +93,7 @@ class TcpInputWithGrokTest < Test::Unit::TestCase
94
93
  internal_test_grok(block_config, tests)
95
94
  end
96
95
 
97
- def test_grok_multi_patterns
96
+ test "grok_multi_patterns" do
98
97
  tests = [
99
98
  {"msg" => "Current time is 2014-01-01T00:00:00+0900\n", "expected" => "2014-01-01T00:00:00+0900"},
100
99
  {"msg" => "The first word matches\n", "expected" => "The"}
@@ -50,7 +50,7 @@ TEXT
50
50
  end
51
51
  end
52
52
 
53
- def test_empty_range_text_in_text
53
+ test "empty_range_text_in_text" do
54
54
  text = " [b-a]"
55
55
  conf = %[
56
56
  grok_pattern %{HOSTNAME:hostname} %{GREEDYDATA:message}
@@ -61,8 +61,8 @@ TEXT
61
61
  assert(d.instance.firstline?(text))
62
62
  end
63
63
 
64
- class NoGrokPatternMatched < self
65
- def test_with_grok_failure_key
64
+ sub_test_case "NoGrokPatternMatched" do
65
+ test "with grok_failure_key" do
66
66
  config = %[
67
67
  grok_failure_key grok_failure
68
68
  <grok>
@@ -79,7 +79,7 @@ TEXT
79
79
  end
80
80
  end
81
81
 
82
- def test_without_grok_failure_key
82
+ test "without grok_failure" do
83
83
  config = %[
84
84
  <grok>
85
85
  pattern %{PATH:path}
@@ -95,7 +95,7 @@ TEXT
95
95
  end
96
96
  end
97
97
 
98
- def test_no_grok_patterns
98
+ test "no_grok_patterns" do
99
99
  assert_raise Fluent::ConfigError do
100
100
  create_driver('')
101
101
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-grok-parser
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.1.6
4
+ version: 2.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - kiyoto
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2018-02-20 00:00:00.000000000 Z
12
+ date: 2018-08-15 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: bundler
@@ -133,7 +133,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
133
133
  version: '0'
134
134
  requirements: []
135
135
  rubyforge_project:
136
- rubygems_version: 2.7.3
136
+ rubygems_version: 2.7.6
137
137
  signing_key:
138
138
  specification_version: 4
139
139
  summary: Fluentd plugin to support Logstash-inspired Grok format for parsing logs