logstash-input-syslog 3.2.4 → 3.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a308c5702d6887cf5bf7842fdd4b3cd37d8b3591bb9abf84914473b5711ffffd
4
- data.tar.gz: bda506472180eed7313a59b0927b589658ac3344484cfef7b245bbb463a92611
3
+ metadata.gz: afc32918fa577b058932c4b231b47a8b1f1fe9b568e5126aacf8e9d0224563bb
4
+ data.tar.gz: 5db1690089b6f82efcad455170defb3f1987d9420197817af225ca2480511ffc
5
5
  SHA512:
6
- metadata.gz: 86baef4a8e94743bb2523eede569072947d26cddede101162c72b1ee17ed74be9aef62232f4a4d2f0a7b2b4adbe3134051abf6ba91c5b17d2ea7125d255da0be
7
- data.tar.gz: 5073d8e96af959e9d71cda4d77e11a2224249733c95386cf818277ed35e6a19b56c43bf33f5efee7039bcab64db771204d5da5b83c9d887d7d589273f0a7b642
6
+ metadata.gz: 1e25afad30245aa9bb745861f8fef0d474eb8c0de28c1964e71e939aa02aabe55785fc03312d14771c9608378e045168691e284485facea58dfa48d9a660a581
7
+ data.tar.gz: d04b48e1192204d932cdb011dfd79ae310370dacb9fb5120b0c2421cbeb2a7d82119eb8719fc31810646b487699d32b453350654c00ca4aa3b7f3db3abf4e333
@@ -1,3 +1,6 @@
1
+ ## 3.3.0
2
+ - Make the grok pattern a configurable option
3
+
1
4
  ## 3.2.4
2
5
  - Fix issue where stopping a pipeline (e.g., while reloading configuration) with active inbound syslog connections could cause Logstash to crash
3
6
 
data/LICENSE CHANGED
@@ -1,4 +1,4 @@
1
- Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
1
+ Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
2
2
 
3
3
  Licensed under the Apache License, Version 2.0 (the "License");
4
4
  you may not use this file except in compliance with the License.
data/NOTICE.TXT CHANGED
@@ -1,5 +1,5 @@
1
1
  Elasticsearch
2
- Copyright 2012-2015 Elasticsearch
2
+ Copyright 2012-2018 Elasticsearch
3
3
 
4
4
  This product includes software developed by The Apache Software
5
5
  Foundation (http://www.apache.org/).
@@ -27,10 +27,11 @@ It is also a good choice if you want to receive logs from
27
27
  appliances and network devices where you cannot run your own
28
28
  log collector.
29
29
 
30
- Of course, 'syslog' is a very muddy term. This input only supports `RFC3164`
31
- syslog with some small modifications. The date format is allowed to be
32
- `RFC3164` style or `ISO8601`. Otherwise the rest of `RFC3164` must be obeyed.
33
- If you do not use `RFC3164`, do not use this input.
30
+ Of course, 'syslog' is a very muddy term. By default, this input only
31
+ supports `RFC3164` syslog with some small modifications. However, some
32
+ non-standard syslog formats can be read and parsed if a functional
33
+ `grok_pattern` is provided. The date format is still only allowed to be
34
+ `RFC3164` style or `ISO8601`.
34
35
 
35
36
  For more information see the http://www.ietf.org/rfc/rfc3164.txt[RFC3164 page].
36
37
 
@@ -46,6 +47,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
46
47
  |=======================================================================
47
48
  |Setting |Input type|Required
48
49
  | <<plugins-{type}s-{plugin}-facility_labels>> |<<array,array>>|No
50
+ | <<plugins-{type}s-{plugin}-grok_pattern>> |<<string,string>>|No
49
51
  | <<plugins-{type}s-{plugin}-host>> |<<string,string>>|No
50
52
  | <<plugins-{type}s-{plugin}-locale>> |<<string,string>>|No
51
53
  | <<plugins-{type}s-{plugin}-port>> |<<number,number>>|No
@@ -61,15 +63,33 @@ input plugins.
61
63
  &nbsp;
62
64
 
63
65
  [id="plugins-{type}s-{plugin}-facility_labels"]
64
- ===== `facility_labels`
66
+ ===== `facility_labels`
65
67
 
66
68
  * Value type is <<array,array>>
67
69
  * Default value is `["kernel", "user-level", "mail", "system", "security/authorization", "syslogd", "line printer", "network news", "UUCP", "clock", "security/authorization", "FTP", "NTP", "log audit", "log alert", "clock", "local0", "local1", "local2", "local3", "local4", "local5", "local6", "local7"]`
68
70
 
69
71
  Labels for facility levels. These are defined in RFC3164.
70
72
 
73
+ [id="plugins-{type}s-{plugin}-grok_pattern"]
74
+ ===== `grok_pattern`
75
+
76
+ * Value type is <<string,string>>
77
+ * Default value is `"<%{POSINT:priority}>%{SYSLOGLINE}"`
78
+
79
+ The default value should read and properly parse syslog lines which are
80
+ fully compliant with http://www.ietf.org/rfc/rfc3164.txt[RFC3164].
81
+
82
+ You can override this value to parse non-standard lines with a valid grok
83
+ pattern which will parse the received lines. If the line is unable to
84
+ be parsed, the `_grokparsefailure_sysloginput` tag will be added.
85
+
86
+ The grok pattern must provide a `timestamp` field. If the `timestamp`
87
+ field is omitted, or is unable to be parsed as `RFC3164` style or
88
+ `ISO8601`, a `_dateparsefailure` tag will be added.
89
+
90
+
71
91
  [id="plugins-{type}s-{plugin}-host"]
72
- ===== `host`
92
+ ===== `host`
73
93
 
74
94
  * Value type is <<string,string>>
75
95
  * Default value is `"0.0.0.0"`
@@ -77,7 +97,7 @@ Labels for facility levels. These are defined in RFC3164.
77
97
  The address to listen on.
78
98
 
79
99
  [id="plugins-{type}s-{plugin}-locale"]
80
- ===== `locale`
100
+ ===== `locale`
81
101
 
82
102
  * Value type is <<string,string>>
83
103
  * There is no default value for this setting.
@@ -91,7 +111,7 @@ weekday names (pattern with EEE).
91
111
 
92
112
 
93
113
  [id="plugins-{type}s-{plugin}-port"]
94
- ===== `port`
114
+ ===== `port`
95
115
 
96
116
  * Value type is <<number,number>>
97
117
  * Default value is `514`
@@ -100,7 +120,7 @@ The port to listen on. Remember that ports less than 1024 (privileged
100
120
  ports) may require root to use.
101
121
 
102
122
  [id="plugins-{type}s-{plugin}-proxy_protocol"]
103
- ===== `proxy_protocol`
123
+ ===== `proxy_protocol`
104
124
 
105
125
  * Value type is <<boolean,boolean>>
106
126
  * Default value is `false`
@@ -109,7 +129,7 @@ Proxy protocol support, only v1 is supported at this time
109
129
  http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
110
130
 
111
131
  [id="plugins-{type}s-{plugin}-severity_labels"]
112
- ===== `severity_labels`
132
+ ===== `severity_labels`
113
133
 
114
134
  * Value type is <<array,array>>
115
135
  * Default value is `["Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"]`
@@ -117,7 +137,7 @@ http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
117
137
  Labels for severity levels. These are defined in RFC3164.
118
138
 
119
139
  [id="plugins-{type}s-{plugin}-timezone"]
120
- ===== `timezone`
140
+ ===== `timezone`
121
141
 
122
142
  * Value type is <<string,string>>
123
143
  * There is no default value for this setting.
@@ -131,7 +151,7 @@ Canonical ID is good as it takes care of daylight saving time for you
131
151
  For example, `America/Los_Angeles` or `Europe/France` are valid IDs.
132
152
 
133
153
  [id="plugins-{type}s-{plugin}-use_labels"]
134
- ===== `use_labels`
154
+ ===== `use_labels`
135
155
 
136
156
  * Value type is <<boolean,boolean>>
137
157
  * Default value is `true`
@@ -141,4 +161,4 @@ Use label parsing for severity and facility levels.
141
161
 
142
162
 
143
163
  [id="plugins-{type}s-{plugin}-common-options"]
144
- include::{include_path}/{type}.asciidoc[]
164
+ include::{include_path}/{type}.asciidoc[]
@@ -36,6 +36,10 @@ class LogStash::Inputs::Syslog < LogStash::Inputs::Base
36
36
  # ports) may require root to use.
37
37
  config :port, :validate => :number, :default => 514
38
38
 
39
+ # Set custom grok pattern to parse the syslog, in case the format differs
40
+ # from the defined standard. This is common in security and other appliances
41
+ config :grok_pattern, :validate => :string, :default => "<%{POSINT:priority}>%{SYSLOGLINE}"
42
+
39
43
  # Proxy protocol support, only v1 is supported at this time
40
44
  # http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
41
45
  config :proxy_protocol, :validate => :boolean, :default => false
@@ -79,12 +83,12 @@ class LogStash::Inputs::Syslog < LogStash::Inputs::Base
79
83
  require "thread_safe"
80
84
  @grok_filter = LogStash::Filters::Grok.new(
81
85
  "overwrite" => "message",
82
- "match" => { "message" => "<%{POSINT:priority}>%{SYSLOGLINE}" },
86
+ "match" => { "message" => @grok_pattern },
83
87
  "tag_on_failure" => ["_grokparsefailure_sysloginput"],
84
88
  )
85
89
 
86
90
  @date_filter = LogStash::Filters::Date.new(
87
- "match" => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"],
91
+ "match" => [ "timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss", "ISO8601"],
88
92
  "locale" => @locale,
89
93
  "timezone" => @timezone,
90
94
  )
@@ -1,7 +1,7 @@
1
1
  Gem::Specification.new do |s|
2
2
 
3
3
  s.name = 'logstash-input-syslog'
4
- s.version = '3.2.4'
4
+ s.version = '3.3.0'
5
5
  s.licenses = ['Apache License (2.0)']
6
6
  s.summary = "Reads syslog messages as events"
7
7
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -210,4 +210,42 @@ describe LogStash::Inputs::Syslog do
210
210
  it_behaves_like 'an interruptible input plugin' do
211
211
  let(:config) { { "port" => 5511 } }
212
212
  end
213
+
214
+ it "should properly handle a custom grok_pattern" do
215
+ port = 5511
216
+ event_count = 1
217
+ custom_grok = "<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp} atypical %{GREEDYDATA:message}"
218
+ message_field = "This part constitutes the message field"
219
+ timestamp = "Oct 26 15:19:25"
220
+ custom_line = "<164>#{timestamp} atypical #{message_field}"
221
+
222
+ conf = <<-CONFIG
223
+ input {
224
+ syslog {
225
+ type => "blah"
226
+ port => #{port}
227
+ grok_pattern => "#{custom_grok}"
228
+ }
229
+ }
230
+ CONFIG
231
+
232
+ events = input(conf) do |pipeline, queue|
233
+ socket = Stud.try(5.times) { TCPSocket.new("127.0.0.1", port) }
234
+ event_count.times do |i|
235
+ socket.puts(custom_line)
236
+ end
237
+ socket.close
238
+
239
+ event_count.times.collect { queue.pop }
240
+ end
241
+
242
+ insist { events.length } == event_count
243
+ events.each do |event|
244
+ insist { event.get("priority") } == 164
245
+ insist { event.get("severity") } == 4
246
+ insist { event.get("facility") } == 20
247
+ insist { event.get("message") } == "#{message_field}\n"
248
+ insist { event.get("timestamp") } == timestamp
249
+ end
250
+ end
213
251
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-syslog
3
3
  version: !ruby/object:Gem::Version
4
- version: 3.2.4
4
+ version: 3.3.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2017-12-06 00:00:00.000000000 Z
11
+ date: 2018-02-06 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement