logstash-input-beats 6.0.13-java → 6.1.3-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: ef7779105775e36517acd284df5cbb37c939bd8b0efd65e4a3ffb666800e4c6a
4
- data.tar.gz: f9d9e7d95f1bcae77c92bff84eb8c80b74abc8065e4ebcec602104aac08ce029
3
+ metadata.gz: e3d70597b1b447c581f1a7c6d9757aeff377cb57f9e0bf30ae62a1a82ae2e3d7
4
+ data.tar.gz: f2fbd686d950b35fb9d337330e226029b4dce5eaaf56e3ac31722cc6dbd849ac
5
5
  SHA512:
6
- metadata.gz: 5d0894f99926e28eead2857c4f3290b6bc2e59595b305c7feb5c4fb331d54267030eb3791bbff0c6838103528161a64f02d41f4ce9345dd500e58b3a3723219b
7
- data.tar.gz: ce4d2455d86c6f1d40d92d9e476ca942bd92ca156fdc764ecf8941716295c0038ec799004da6c7f7e7d626e2aabd5e96d5555a80c0de5386cf885d4cec930240
6
+ metadata.gz: 892b7166ecd256432677e4bda1dbc2ebd4c9555fc2aecf26e73ed0d2747b2e736b6e1421582b3ec786ba6bef4c6a705c4c0a6d0b54423980c0f9f7ddbf4ddaf9
7
+ data.tar.gz: e8101ebad94b1924d68a2000b8c2b9c9996ef2634fcb1e8d87d445e64941b5278b6af8c24d7f93e3c271638028f36f258c717d9401595c213a3f99ec46f138ed
data/CHANGELOG.md CHANGED
@@ -1,3 +1,21 @@
1
+ ## 6.1.3
2
+ - Fix: safe-guard byte buf allocation [#420](https://github.com/logstash-plugins/logstash-input-beats/pull/420)
3
+ - Updated Jackson dependencies
4
+
5
+ ## 6.1.2
6
+ - [DOC] Added naming attribute to control plugin name that appears in docs, and set up framework to make attributes viable in code sample
7
+
8
+ ## 6.1.1
9
+ - [DOC] Enhanced ECS compatibility information for ease of use and readability
10
+ [#413](https://github.com/logstash-plugins/logstash-input-beats/pull/413)
11
+
12
+ ## 6.1.0
13
+ - ECS compatibility enablement. Adds `ecs_compatibility` setting to declare the level of ECS compatibility (`disabled` or `v1`) at plugin level. When `disabled`, the plugin behaves like before, while `v1` does a rename of
14
+ `host` and `@metadata.ip_address` event fields. [#404](https://github.com/logstash-plugins/logstash-input-beats/pull/404)
15
+
16
+ ## 6.0.14
17
+ - Feat: log + unwrap generic SSL context exceptions [#405](https://github.com/logstash-plugins/logstash-input-beats/pull/405)
18
+
1
19
  ## 6.0.13
2
20
  - [DOC] Update links to use shared attributes
3
21
 
data/VERSION CHANGED
@@ -1 +1 @@
1
- 6.0.13
1
+ 6.1.3
data/docs/index.asciidoc CHANGED
@@ -1,6 +1,7 @@
1
1
  :plugin: beats
2
2
  :type: input
3
3
  :default_codec: plain
4
+ :plugin-uc: Beats
4
5
 
5
6
  ///////////////////////////////////////////
6
7
  START - GENERATED VARIABLES, DO NOT EDIT!
@@ -15,7 +16,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
15
16
 
16
17
  [id="plugins-{type}s-{plugin}"]
17
18
 
18
- === Beats input plugin
19
+ === {plugin-uc} input plugin
19
20
 
20
21
  include::{include_path}/plugin_header.asciidoc[]
21
22
 
@@ -25,9 +26,9 @@ This input plugin enables Logstash to receive events from the
25
26
  https://www.elastic.co/products/beats[Elastic Beats] framework.
26
27
 
27
28
  The following example shows how to configure Logstash to listen on port
28
- 5044 for incoming Beats connections and to index into Elasticsearch.
29
+ 5044 for incoming {plugin-uc} connections and to index into Elasticsearch.
29
30
 
30
- [source,logstash]
31
+ ["source","sh",subs="attributes"]
31
32
  -----
32
33
 
33
34
  input {
@@ -45,16 +46,16 @@ output {
45
46
  -----
46
47
  <1> `%{[@metadata][beat]}` sets the first part of the index name to the value
47
48
  of the `beat` metadata field and `%{[@metadata][version]}` sets the second part to
48
- the Beat's version. For example:
49
+ the {plugin-uc}'s version. For example:
49
50
  metricbeat-7.4.0.
50
51
 
51
52
  Events indexed into Elasticsearch with the Logstash configuration shown here
52
- will be similar to events directly indexed by Beats into Elasticsearch.
53
+ will be similar to events directly indexed by {plugin-uc} into Elasticsearch.
53
54
 
54
55
  NOTE: If ILM is not being used, set `index` to
55
56
  `%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}` instead so
56
57
  Logstash creates an index per day, based on the `@timestamp` value of the events
57
- coming from Beats.
58
+ coming from {plugin-uc}.
58
59
 
59
60
  IMPORTANT: If you are shipping events that span multiple lines, you need to use
60
61
  the {filebeat-ref}/multiline-examples.html[configuration options available in
@@ -64,7 +65,7 @@ plugin] to handle multiline events. Doing so will result in the failure to start
64
65
  Logstash.
65
66
 
66
67
  [id="plugins-{type}s-{plugin}-versioned-indexes"]
67
- ==== Versioned Beats Indices
68
+ ==== Versioned indices
68
69
 
69
70
  To minimize the impact of future schema changes on your existing indices and
70
71
  mappings in Elasticsearch, configure the Elasticsearch output to write to
@@ -86,8 +87,32 @@ Logstash `@timestamp` field.
86
87
  This configuration results in daily index names like
87
88
  +filebeat-{logstash_version}-{localdate}+.
88
89
 
90
+
91
+ [id="plugins-{type}s-{plugin}-ecs_metadata"]
92
+ ==== Event Metadata and the Elastic Common Schema (ECS)
93
+
94
+ When decoding {plugin-uc} events, this plugin adds two fields related to the event:
95
+ the deprecated `host` which contains the `hostname` provided by {plugin-uc} and the
96
+ `ip_address` containing the remote address of the client's connection. When
97
+ <<plugins-{type}s-{plugin}-ecs_compatibility,ECS compatibility mode>> is enabled
98
+ these are now moved in ECS compatible namespace. Here's how
99
+ <<plugins-{type}s-{plugin}-ecs_compatibility,ECS compatibility mode>> affects
100
+ output.
101
+
102
+ [cols="<l,<l,e,<e"]
103
+ |=======================================================================
104
+ |ECS disabled |ECS v1 |Availability |Description
105
+
106
+ |[host] |[@metadata][input][beats][host][name] |Always |Name or address of the beat host
107
+ |[@metadata][ip_address] |[@metadata][input][beats][host][ip] |Always |IP address of the Beats client
108
+ |[@metadata][tls_peer][status] | [@metadata][tls_peer][status] | When SSL related fields are populated | Contains "verified"/"unverified" labels in `disabled`, `true`/`false` in `v1`
109
+ |[@metadata][tls_peer][protocol] | [@metadata][input][beats][tls][version_protocol] | When SSL status is "verified" | Contains the TLS version used (e.g. `TLSv1.2`)
110
+ |[@metadata][tls_peer][subject] | [@metadata][input][beats][tls][client][subject] | When SSL status is "verified" | Contains the identity name of the remote end (e.g. `CN=artifacts-no-kpi.elastic.co`)
111
+ |[@metadata][tls_peer][cipher_suite] | [@metadata][input][beats][tls][cipher] | When SSL status is "verified" | Contains the name of cipher suite used (e.g. `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`)
112
+ |=======================================================================
113
+
89
114
  [id="plugins-{type}s-{plugin}-options"]
90
- ==== Beats Input Configuration Options
115
+ ==== {plugin-uc} Input Configuration Options
91
116
 
92
117
  This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
93
118
 
@@ -97,6 +122,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
97
122
  | <<plugins-{type}s-{plugin}-add_hostname>> |<<boolean,boolean>>|No
98
123
  | <<plugins-{type}s-{plugin}-cipher_suites>> |<<array,array>>|No
99
124
  | <<plugins-{type}s-{plugin}-client_inactivity_timeout>> |<<number,number>>|No
125
+ | <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
100
126
  | <<plugins-{type}s-{plugin}-host>> |<<string,string>>|No
101
127
  | <<plugins-{type}s-{plugin}-include_codec_tag>> |<<boolean,boolean>>|No
102
128
  | <<plugins-{type}s-{plugin}-port>> |<<number,number>>|Yes
@@ -144,6 +170,19 @@ The list of ciphers suite to use, listed by priorities.
144
170
 
145
171
  Close Idle clients after X seconds of inactivity.
146
172
 
173
+ [id="plugins-{type}s-{plugin}-ecs_compatibility"]
174
+ ===== `ecs_compatibility`
175
+
176
+ * Value type is <<string,string>>
177
+ * Supported values are:
178
+ ** `disabled`: unstructured connection metadata added at root level
179
+ ** `v1`: structured connection metadata added under ECS compliant namespaces
180
+ * Default value depends on which version of Logstash is running:
181
+ ** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
182
+ ** Otherwise, the default value is `disabled`.
183
+
184
+ Refer to <<plugins-{type}s-{plugin}-ecs_metadata,ECS mapping>> for detailed information.
185
+
147
186
  [id="plugins-{type}s-{plugin}-host"]
148
187
  ===== `host`
149
188
 
@@ -5,7 +5,7 @@ require_jar('io.netty', 'netty-all', '4.1.49.Final')
5
5
  require_jar('org.javassist', 'javassist', '3.24.0-GA')
6
6
  require_jar('com.fasterxml.jackson.core', 'jackson-core', '2.9.10')
7
7
  require_jar('com.fasterxml.jackson.core', 'jackson-annotations', '2.9.10')
8
- require_jar('com.fasterxml.jackson.core', 'jackson-databind', '2.9.10.4')
8
+ require_jar('com.fasterxml.jackson.core', 'jackson-databind', '2.9.10.8')
9
9
  require_jar('com.fasterxml.jackson.module', 'jackson-module-afterburner', '2.9.10')
10
10
  require_jar('org.apache.logging.log4j', 'log4j-api', '2.11.1')
11
- require_jar('org.logstash.beats', 'logstash-input-beats', '6.0.13')
11
+ require_jar('org.logstash.beats', 'logstash-input-beats', '6.1.3')
@@ -5,6 +5,7 @@ require "logstash/timestamp"
5
5
  require "logstash/codecs/multiline"
6
6
  require "logstash/util"
7
7
  require "logstash-input-beats_jars"
8
+ require "logstash/plugin_mixins/ecs_compatibility_support"
8
9
  require_relative "beats/patch"
9
10
 
10
11
  # This input plugin enables Logstash to receive events from the
@@ -49,6 +50,9 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
49
50
  require "logstash/inputs/beats/message_listener"
50
51
  require "logstash/inputs/beats/tls"
51
52
 
53
+ # adds ecs_compatibility config which could be :disabled or :v1
54
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled,:v1)
55
+
52
56
  config_name "beats"
53
57
 
54
58
  default :codec, "plain"
@@ -121,6 +125,8 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
121
125
  # Beats handler executor thread
122
126
  config :executor_threads, :validate => :number, :default => LogStash::Config::CpuCoreStrategy.maximum
123
127
 
128
+ attr_reader :field_hostname, :field_hostip
129
+
124
130
  def register
125
131
  # For Logstash 2.4 we need to make sure that the logger is correctly set for the
126
132
  # java classes before actually loading them.
@@ -156,6 +162,13 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
156
162
  configuration_error "Multiline codec with beats input is not supported. Please refer to the beats documentation for how to best manage multiline data. See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html"
157
163
  end
158
164
 
165
+ # define ecs name mapping
166
+ @field_hostname = ecs_select[disabled: "host", v1: "[@metadata][input][beats][host][name]"]
167
+ @field_hostip = ecs_select[disabled: "[@metadata][ip_address]", v1: "[@metadata][input][beats][host][ip]"]
168
+ @field_tls_protocol_version = ecs_select[disabled: "[@metadata][tls_peer][protocol]", v1: "[@metadata][input][beats][tls][version_protocol]"]
169
+ @field_tls_peer_subject = ecs_select[disabled: "[@metadata][tls_peer][subject]", v1: "[@metadata][input][beats][tls][client][subject]"]
170
+ @field_tls_cipher = ecs_select[disabled: "[@metadata][tls_peer][cipher_suite]", v1: "[@metadata][input][beats][tls][cipher]"]
171
+
159
172
  @logger.info("Starting input listener", :address => "#{@host}:#{@port}")
160
173
 
161
174
  @server = create_server
@@ -220,7 +233,7 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
220
233
  rescue java.lang.IllegalArgumentException => e
221
234
  @logger.error("SSL configuration invalid", error_details(e))
222
235
  raise LogStash::ConfigurationError, e
223
- rescue java.security.GeneralSecurityException => e
236
+ rescue java.lang.Exception => e # java.security.GeneralSecurityException
224
237
  @logger.error("SSL configuration failed", error_details(e, true))
225
238
  raise e
226
239
  end
@@ -254,6 +267,11 @@ class LogStash::Inputs::Beats < LogStash::Inputs::Base
254
267
  def error_details(e, trace = false)
255
268
  error_details = { :exception => e.class, :message => e.message }
256
269
  error_details[:backtrace] = e.backtrace if trace || @logger.debug?
270
+ cause = e.cause
271
+ if cause && e != cause
272
+ error_details[:cause] = { :exception => cause.class, :message => cause.message }
273
+ error_details[:cause][:backtrace] = cause.backtrace if trace || @logger.debug?
274
+ end
257
275
  error_details
258
276
  end
259
277
 
@@ -9,7 +9,16 @@ module LogStash module Inputs class Beats
9
9
  ts = coerce_ts(hash.delete("@timestamp"))
10
10
 
11
11
  event.set("@timestamp", ts) unless ts.nil?
12
- hash.each { |k, v| event.set(k, v) }
12
+ hash.each do |k, v|
13
+ #could be a nested map, so we need to merge and not overwrite
14
+ existing_value = event.get(k)
15
+ if existing_value.is_a?(Hash)
16
+ existing_value = existing_value.merge(v)
17
+ else
18
+ existing_value = v
19
+ end
20
+ event.set(k, existing_value)
21
+ end
13
22
  super(event)
14
23
  event.tag("beats_input_codec_#{codec_name}_applied") if include_codec_tag?
15
24
  event
@@ -15,8 +15,8 @@ module LogStash module Inputs class Beats
15
15
  return unless @input.add_hostname
16
16
  host = event.get("[beat][hostname]")
17
17
 
18
- if host && event.get("host").nil?
19
- event.set("host", host)
18
+ if host && event.get(@input.field_hostname).nil?
19
+ event.set(@input.field_hostname, host)
20
20
  end
21
21
  end
22
22
 
@@ -31,7 +31,9 @@ module LogStash module Inputs class Beats
31
31
  hash = message.getData
32
32
  ip_address = ip_address(ctx)
33
33
 
34
- hash['@metadata']['ip_address'] = ip_address unless ip_address.nil? || hash['@metadata'].nil?
34
+ unless ip_address.nil? || hash['@metadata'].nil?
35
+ set_nested(hash, @input.field_hostip, ip_address)
36
+ end
35
37
  target_field = extract_target_field(hash)
36
38
 
37
39
  extract_tls_peer(hash, ctx)
@@ -140,11 +142,12 @@ module LogStash module Inputs class Beats
140
142
  end
141
143
 
142
144
  if tls_verified
145
+ set_nested(hash, @field_tls_protocol_version, tls_session.getProtocol())
146
+ set_nested(hash, @field_tls_peer_subject, tls_session.getPeerPrincipal().getName())
147
+ set_nested(hash, @field_tls_cipher, tls_session.getCipherSuite())
148
+
143
149
  hash['@metadata']['tls_peer'] = {
144
- :status => "verified",
145
- :protocol => tls_session.getProtocol(),
146
- :subject => tls_session.getPeerPrincipal().getName(),
147
- :cipher_suite => tls_session.getCipherSuite()
150
+ :status => "verified"
148
151
  }
149
152
  else
150
153
  hash['@metadata']['tls_peer'] = {
@@ -154,6 +157,28 @@ module LogStash module Inputs class Beats
154
157
  end
155
158
  end
156
159
 
160
+ # set the value for field_name into the hash, nesting into sub-hashes and creating hashes where necessary
161
+ public #only to make it testable
162
+ def set_nested(hash, field_name, value)
163
+ field_ref = Java::OrgLogstash::FieldReference.from(field_name)
164
+ # create @metadata sub-hash if needed
165
+ if field_ref.type == Java::OrgLogstash::FieldReference::META_CHILD
166
+ unless hash.key?("@metadata")
167
+ hash["@metadata"] = {}
168
+ end
169
+ nesting_hash = hash["@metadata"]
170
+ else
171
+ nesting_hash = hash
172
+ end
173
+
174
+ field_ref.path.each do |token|
175
+ nesting_hash[token] = {} unless nesting_hash.key?(token)
176
+ nesting_hash = nesting_hash[token]
177
+ end
178
+ nesting_hash[field_ref.key] = value
179
+ end
180
+
181
+ private
157
182
  def extract_target_field(hash)
158
183
  if from_filebeat?(hash)
159
184
  hash.delete(FILEBEAT_LOG_LINE_FIELD).to_s
@@ -27,6 +27,7 @@ Gem::Specification.new do |s|
27
27
  s.add_runtime_dependency "thread_safe", "~> 0.3.5"
28
28
  s.add_runtime_dependency "logstash-codec-multiline", ">= 2.0.5"
29
29
  s.add_runtime_dependency 'jar-dependencies', '~> 0.3', '>= 0.3.4'
30
+ s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.1'
30
31
 
31
32
  s.add_development_dependency "flores", "~>0.0.6"
32
33
  s.add_development_dependency "rspec"
@@ -29,7 +29,8 @@ describe LogStash::Inputs::Beats::DecodedEventTransform do
29
29
 
30
30
  subject { described_class.new(input).transform(event, map) }
31
31
 
32
- include_examples "Common Event Transformation"
32
+ include_examples "Common Event Transformation", :disabled, "host"
33
+ include_examples "Common Event Transformation", :v1, "[@metadata][input][beats][host][name]"
33
34
 
34
35
  it "tags the event" do
35
36
  expect(subject.get("tags")).to include("beats_input_codec_plain_applied")
@@ -43,7 +44,7 @@ describe LogStash::Inputs::Beats::DecodedEventTransform do
43
44
  context "map contains a timestamp" do
44
45
  context "when its valid" do
45
46
  let(:timestamp) { Time.now }
46
- let(:map) { super.merge({"@timestamp" => timestamp }) }
47
+ let(:map) { super().merge({"@timestamp" => timestamp }) }
47
48
 
48
49
  it "uses as the event timestamp" do
49
50
  expect(subject.get("@timestamp")).to eq(LogStash::Timestamp.coerce(timestamp))
@@ -51,7 +52,7 @@ describe LogStash::Inputs::Beats::DecodedEventTransform do
51
52
  end
52
53
 
53
54
  context "when its not valid" do
54
- let(:map) { super.merge({"@timestamp" => "invalid" }) }
55
+ let(:map) { super().merge({"@timestamp" => "invalid" }) }
55
56
 
56
57
  it "fallback the current time" do
57
58
  expect(subject.get("@timestamp")).to be_kind_of(LogStash::Timestamp)
@@ -7,5 +7,6 @@ require "spec_helper"
7
7
  describe LogStash::Inputs::Beats::EventTransformCommon do
8
8
  subject { described_class.new(input).transform(event) }
9
9
 
10
- include_examples "Common Event Transformation"
10
+ include_examples "Common Event Transformation", :disabled, "host"
11
+ include_examples "Common Event Transformation", :v1, "[@metadata][input][beats][host][name]"
11
12
  end
@@ -52,10 +52,73 @@ class DummyCodec < LogStash::Codecs::Base
52
52
  end
53
53
  end
54
54
 
55
+ shared_examples "when the message is from any libbeat" do |ecs_compatibility, host_field_name|
56
+ let(:input) do
57
+ input = LogStash::Inputs::Beats.new({ "port" => 5555, "codec" => codec, "ecs_compatibility" => "#{ecs_compatibility}" })
58
+ input.register
59
+ input
60
+ end
61
+
62
+ #Requires data modeled as Java, not Ruby since the actual code pulls from Java backed (Netty) object
63
+ let(:data) do
64
+ d = HashMap.new
65
+ d.put('@metadata', HashMap.new)
66
+ d.put('metric', 1)
67
+ d.put('name', "super-stats")
68
+ d
69
+ end
70
+
71
+ let(:message) { MockMessage.new("abc", data)}
72
+
73
+ it "extract the event" do
74
+ subject.onNewMessage(ctx, message)
75
+ event = queue.pop
76
+ expect(event.get("message")).to be_nil
77
+ expect(event.get("metric")).to eq(1)
78
+ expect(event.get("name")).to eq("super-stats")
79
+ expect(event.get(host_field_name)).to eq(ip_address)
80
+ end
81
+
82
+ context 'when the remote address is nil' do
83
+ let(:ctx) { OngoingMethodMock.new("remoteAddress", nil)}
84
+
85
+ it 'extracts the event' do
86
+ subject.onNewMessage(ctx, message)
87
+ event = queue.pop
88
+ expect(event.get("message")).to be_nil
89
+ expect(event.get("metric")).to eq(1)
90
+ expect(event.get("name")).to eq("super-stats")
91
+ expect(event.get(host_field_name)).to eq(nil)
92
+ end
93
+ end
94
+
95
+ context 'when getting the remote address raises' do
96
+ let(:raising_ctx) { double("context")}
97
+
98
+ before do
99
+ allow(raising_ctx).to receive(:channel).and_raise("nope")
100
+ subject.onNewConnection(raising_ctx)
101
+ end
102
+
103
+ it 'extracts the event' do
104
+ subject.onNewMessage(raising_ctx, message)
105
+ event = queue.pop
106
+ expect(event.get("message")).to be_nil
107
+ expect(event.get("metric")).to eq(1)
108
+ expect(event.get("name")).to eq("super-stats")
109
+ expect(event.get(host_field_name)).to eq(nil)
110
+ end
111
+ end
112
+ end
113
+
55
114
  describe LogStash::Inputs::Beats::MessageListener do
56
115
  let(:queue) { Queue.new }
57
116
  let(:codec) { DummyCodec.new }
58
- let(:input) { LogStash::Inputs::Beats.new({ "port" => 5555, "codec" => codec }) }
117
+ let(:input) do
118
+ input = LogStash::Inputs::Beats.new({ "port" => 5555, "codec" => codec })
119
+ input.register
120
+ input
121
+ end
59
122
 
60
123
  let(:ip_address) { "10.0.0.1" }
61
124
  let(:remote_address) { OngoingMethodMock.new("getHostAddress", ip_address) }
@@ -146,59 +209,8 @@ describe LogStash::Inputs::Beats::MessageListener do
146
209
  end
147
210
  end
148
211
 
149
- context "when the message is from any libbeat" do
150
- #Requires data modeled as Java, not Ruby since the actual code pulls from Java backed (Netty) object
151
- let(:data) do
152
- d = HashMap.new
153
- d.put('@metadata', HashMap.new)
154
- d.put('metric', 1)
155
- d.put('name', "super-stats")
156
- d
157
- end
158
-
159
- let(:message) { MockMessage.new("abc", data)}
160
-
161
- it "extract the event" do
162
- subject.onNewMessage(ctx, message)
163
- event = queue.pop
164
- expect(event.get("message")).to be_nil
165
- expect(event.get("metric")).to eq(1)
166
- expect(event.get("name")).to eq("super-stats")
167
- expect(event.get("[@metadata][ip_address]")).to eq(ip_address)
168
- end
169
-
170
- context 'when the remote address is nil' do
171
- let(:ctx) { OngoingMethodMock.new("remoteAddress", nil)}
172
-
173
- it 'extracts the event' do
174
- subject.onNewMessage(ctx, message)
175
- event = queue.pop
176
- expect(event.get("message")).to be_nil
177
- expect(event.get("metric")).to eq(1)
178
- expect(event.get("name")).to eq("super-stats")
179
- expect(event.get("[@metadata][ip_address]")).to eq(nil)
180
- end
181
- end
182
-
183
- context 'when getting the remote address raises' do
184
- let(:raising_ctx) { double("context")}
185
-
186
- before do
187
- allow(raising_ctx).to receive(:channel).and_raise("nope")
188
- subject.onNewConnection(raising_ctx)
189
- end
190
-
191
- it 'extracts the event' do
192
- subject.onNewMessage(raising_ctx, message)
193
- event = queue.pop
194
- expect(event.get("message")).to be_nil
195
- expect(event.get("metric")).to eq(1)
196
- expect(event.get("name")).to eq("super-stats")
197
- expect(event.get("[@metadata][ip_address]")).to eq(nil)
198
- end
199
- end
200
-
201
- end
212
+ it_behaves_like "when the message is from any libbeat", :disabled, "[@metadata][ip_address]"
213
+ it_behaves_like "when the message is from any libbeat", :v1, "[@metadata][input][beats][host][ip]"
202
214
  end
203
215
 
204
216
  context "onException" do
@@ -222,4 +234,20 @@ describe LogStash::Inputs::Beats::MessageListener do
222
234
  expect(queue).not_to be_empty
223
235
  end
224
236
  end
237
+
238
+ context "set_nested" do
239
+ let(:hash) {{}}
240
+
241
+ it "creates correctly the nested maps" do
242
+ subject.set_nested(hash, "[root][inner][leaf]", 5)
243
+ expect(hash["root"]["inner"]["leaf"]).to eq(5)
244
+ end
245
+
246
+ it "doesn't overwrite existing the nested maps" do
247
+ hash = {"root" => {"foo" => {"bar" => "Hello"}}}
248
+ subject.set_nested(hash, "[root][inner][leaf]", 5)
249
+ expect(hash["root"]["inner"]["leaf"]).to eq(5)
250
+ expect(hash["root"]["foo"]["bar"]).to eq("Hello")
251
+ end
252
+ end
225
253
  end
@@ -18,7 +18,8 @@ describe LogStash::Inputs::Beats::RawEventTransform do
18
18
 
19
19
  subject { described_class.new(input).transform(event) }
20
20
 
21
- include_examples "Common Event Transformation"
21
+ include_examples "Common Event Transformation", :disabled, "host"
22
+ include_examples "Common Event Transformation", :v1, "[@metadata][input][beats][host][name]"
22
23
 
23
24
  it "tags the event" do
24
25
  expect(subject.get("tags")).to include("beats_input_raw_event")
@@ -25,7 +25,7 @@ describe LogStash::Inputs::Beats do
25
25
 
26
26
  context "#register" do
27
27
  context "host related configuration" do
28
- let(:config) { super.merge("host" => host, "port" => port, "client_inactivity_timeout" => client_inactivity_timeout, "executor_threads" => threads) }
28
+ let(:config) { super().merge("host" => host, "port" => port, "client_inactivity_timeout" => client_inactivity_timeout, "executor_threads" => threads) }
29
29
  let(:host) { "192.168.1.20" }
30
30
  let(:port) { 9000 }
31
31
  let(:client_inactivity_timeout) { 400 }
@@ -76,7 +76,7 @@ describe LogStash::Inputs::Beats do
76
76
  end
77
77
 
78
78
  context "with invalid ciphers" do
79
- let(:config) { super.merge("ssl" => true, "cipher_suites" => "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA38") }
79
+ let(:config) { super().merge("ssl" => true, "cipher_suites" => "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA38") }
80
80
 
81
81
  it "should raise a configuration error" do
82
82
  plugin = LogStash::Inputs::Beats.new(config)
@@ -90,7 +90,7 @@ describe LogStash::Inputs::Beats do
90
90
 
91
91
  context "verify_mode" do
92
92
  context "verify_mode configured to PEER" do
93
- let(:config) { super.merge("ssl" => true, "ssl_verify_mode" => "peer") }
93
+ let(:config) { super().merge("ssl" => true, "ssl_verify_mode" => "peer") }
94
94
 
95
95
  it "raise a ConfigurationError when certificate_authorities is not set" do
96
96
  plugin = LogStash::Inputs::Beats.new(config)
@@ -105,7 +105,7 @@ describe LogStash::Inputs::Beats do
105
105
  end
106
106
 
107
107
  context "verify_mode configured to FORCE_PEER" do
108
- let(:config) { super.merge("ssl" => true, "ssl_verify_mode" => "force_peer") }
108
+ let(:config) { super().merge("ssl" => true, "ssl_verify_mode" => "force_peer") }
109
109
 
110
110
  it "raise a ConfigurationError when certificate_authorities is not set" do
111
111
  plugin = LogStash::Inputs::Beats.new(config)
@@ -154,7 +154,7 @@ describe LogStash::Inputs::Beats do
154
154
  let(:codec) { LogStash::Codecs::Multiline.new("pattern" => '^2015',
155
155
  "what" => "previous",
156
156
  "negate" => true) }
157
- let(:config) { super.merge({ "codec" => codec }) }
157
+ let(:config) { super().merge({ "codec" => codec }) }
158
158
 
159
159
  it "raise a ConfigurationError when multiline codec is set" do
160
160
  plugin = LogStash::Inputs::Beats.new(config)
@@ -86,7 +86,7 @@ describe "Filebeat", :integration => true do
86
86
  end
87
87
 
88
88
  context "without pipelining" do
89
- let(:filebeat_config) { config = super; config["output"]["logstash"]["pipelining"] = 0; config }
89
+ let(:filebeat_config) { config = super(); config["output"]["logstash"]["pipelining"] = 0; config }
90
90
  include_examples "send events"
91
91
 
92
92
  context "with large batches" do
@@ -99,7 +99,7 @@ describe "Filebeat", :integration => true do
99
99
  context "TLS" do
100
100
  context "Server verification" do
101
101
  let(:filebeat_config) do
102
- super.merge({
102
+ super().merge({
103
103
  "output" => {
104
104
  "logstash" => {
105
105
  "hosts" => ["#{host}:#{port}"],
@@ -111,7 +111,7 @@ describe "Filebeat", :integration => true do
111
111
  end
112
112
 
113
113
  let(:input_config) do
114
- super.merge({
114
+ super().merge({
115
115
  "ssl" => true,
116
116
  "ssl_certificate" => certificate_file,
117
117
  "ssl_key" => certificate_key_file
@@ -129,7 +129,7 @@ describe "Filebeat", :integration => true do
129
129
 
130
130
  context "when specifying a cipher" do
131
131
  let(:filebeat_config) do
132
- super.merge({
132
+ super().merge({
133
133
  "output" => {
134
134
  "logstash" => {
135
135
  "hosts" => ["#{host}:#{port}"],
@@ -145,7 +145,7 @@ describe "Filebeat", :integration => true do
145
145
  end
146
146
 
147
147
  let(:input_config) {
148
- super.merge({
148
+ super().merge({
149
149
  "cipher_suites" => [logstash_cipher],
150
150
  "tls_min_version" => "1.2"
151
151
  })
@@ -194,7 +194,7 @@ describe "Filebeat", :integration => true do
194
194
  LogStash::Inputs::Beats.new(input_config)
195
195
  }
196
196
  let(:input_config) {
197
- super.merge({
197
+ super().merge({
198
198
  "ssl_key_passphrase" => passphrase,
199
199
  "ssl_key" => certificate_key_file_pkcs8
200
200
  })}
@@ -229,7 +229,7 @@ describe "Filebeat", :integration => true do
229
229
 
230
230
  context "Client verification / Mutual validation" do
231
231
  let(:filebeat_config) do
232
- super.merge({
232
+ super().merge({
233
233
  "output" => {
234
234
  "logstash" => {
235
235
  "hosts" => ["#{host}:#{port}"],
@@ -245,7 +245,7 @@ describe "Filebeat", :integration => true do
245
245
  end
246
246
 
247
247
  let(:input_config) do
248
- super.merge({
248
+ super().merge({
249
249
  "ssl" => true,
250
250
  "ssl_certificate_authorities" => certificate_authorities,
251
251
  "ssl_certificate" => server_certificate_file,
@@ -327,7 +327,7 @@ describe "Filebeat", :integration => true do
327
327
 
328
328
  context "client from secondary CA" do
329
329
  let(:filebeat_config) do
330
- super.merge({
330
+ super().merge({
331
331
  "output" => {
332
332
  "logstash" => {
333
333
  "hosts" => ["#{host}:#{port}"],
@@ -74,7 +74,7 @@ describe "Logstash-Forwarder", :integration => true do
74
74
  context "TLS" do
75
75
  context "Server Verification" do
76
76
  let(:input_config) do
77
- super.merge({
77
+ super().merge({
78
78
  "ssl" => true,
79
79
  "ssl_certificate" => certificate_file,
80
80
  "ssl_key" => certificate_key_file,
@@ -1,5 +1,5 @@
1
1
  # encoding: utf-8
2
- shared_examples "Common Event Transformation" do
2
+ shared_examples "Common Event Transformation" do |ecs_compatibility, host_field_name|
3
3
  let(:tag) { "140-rpm-beats" }
4
4
  let(:config) do
5
5
  {
@@ -21,17 +21,23 @@ shared_examples "Common Event Transformation" do
21
21
  }
22
22
  end
23
23
 
24
+ def key_as_nested_maps(key, value)
25
+ evt = LogStash::Event.new
26
+ evt.set(key, value)
27
+ evt.to_hash_with_metadata
28
+ end
29
+
24
30
  it "adds configured tags to the event" do
25
31
  expect(subject.get("tags")).to include(tag)
26
32
  end
27
33
 
28
34
  context 'when add_hostname is true' do
29
- let(:config) { super.merge({'add_hostname' => true})}
35
+ let(:config) { super().merge({'add_hostname' => true, 'ecs_compatibility' => ecs_compatibility})}
30
36
 
31
37
  context 'when a host is provided in beat.host.name' do
32
38
  let(:already_exist) { "already_exist" }
33
39
  let(:producer_host) { "newhost01" }
34
- let(:event_map) { super.merge({ "beat" => { "host" => {"name" => producer_host }}}) }
40
+ let(:event_map) { super().merge({ "beat" => { "host" => {"name" => producer_host }}}) }
35
41
 
36
42
  context "when no `host` key already exists on the event" do
37
43
  it "does not set the host value" do
@@ -41,7 +47,7 @@ shared_examples "Common Event Transformation" do
41
47
 
42
48
  context "when `host` key exists on the event" do
43
49
  let(:already_exist) { "already_exist" }
44
- let(:event_map) { super.merge({ "host" => already_exist }) }
50
+ let(:event_map) { super().merge({ "host" => already_exist }) }
45
51
 
46
52
  it "doesn't override it" do
47
53
  expect(subject.get("host")).to eq(already_exist)
@@ -51,99 +57,99 @@ shared_examples "Common Event Transformation" do
51
57
 
52
58
  context "when a host is set in `beat.hostname`" do
53
59
  let(:producer_host) { "newhost01" }
54
- let(:event_map) { super.merge({ "beat" => { "hostname" => producer_host }}) }
60
+ let(:event_map) { super().merge({ "beat" => { "hostname" => producer_host }}) }
55
61
 
56
- context "when no `host` key already exists on the event" do
57
- it "copies the value in `beat.hostname` to `host`" do
58
- expect(subject.get("host")).to eq(producer_host)
62
+ context "when no `#{host_field_name}` key already exists on the event" do
63
+ it "copies the value in `beat.hostname` to `#{host_field_name}`" do
64
+ expect(subject.get(host_field_name)).to eq(producer_host)
59
65
  end
60
66
  end
61
67
 
62
- context "when `host` key exists on the event" do
68
+ context "when `#{host_field_name}` key exists on the event" do
63
69
  let(:already_exist) { "already_exist" }
64
- let(:event_map) { super.merge({ "host" => already_exist }) }
70
+ let(:event_map) { super().merge(key_as_nested_maps(host_field_name, already_exist)) }
65
71
 
66
72
  it "doesn't override it" do
67
- expect(subject.get("host")).to eq(already_exist)
73
+ expect(subject.get(host_field_name)).to eq(already_exist)
68
74
  end
69
75
  end
70
76
  end
71
77
 
72
78
  context "when no host is provided in beat" do
73
- context "when no `host` key already exists on the event" do
79
+ context "when no `#{host_field_name}` key already exists on the event" do
74
80
  it "does not set the host" do
75
- expect(subject.get("host")).to be_nil
81
+ expect(subject.get(host_field_name)).to be_nil
76
82
  end
77
83
  end
78
84
 
79
- context "when `host` key already exists on the event" do
85
+ context "when `#{host_field_name}` key already exists on the event" do
80
86
  let(:already_exist) { "already_exist" }
81
- let(:event_map) { super.merge({ "host" => already_exist }) }
87
+ let(:event_map) { super().merge(key_as_nested_maps(host_field_name, already_exist)) }
82
88
 
83
89
  it "doesn't override it" do
84
- expect(subject.get("host")).to eq(already_exist)
90
+ expect(subject.get(host_field_name)).to eq(already_exist)
85
91
  end
86
92
  end
87
93
  end
88
94
  end
89
95
 
90
96
  context 'when add hostname is false' do
91
- let(:config) { super.merge({'add_hostname' => false})}
97
+ let(:config) { super().merge({'add_hostname' => false})}
92
98
 
93
99
  context 'when a host is provided in beat.host.name' do
94
100
  let(:already_exist) { "already_exist" }
95
101
  let(:producer_host) { "newhost01" }
96
- let(:event_map) { super.merge({ "beat" => { "host" => {"name" => producer_host }}}) }
102
+ let(:event_map) { super().merge({ "beat" => { "host" => {"name" => producer_host }}}) }
97
103
 
98
- context "when no `host` key already exists on the event" do
104
+ context "when no `#{host_field_name}` key already exists on the event" do
99
105
  it "does not set the host" do
100
- expect(subject.get("host")).to be_nil
106
+ expect(subject.get(host_field_name)).to be_nil
101
107
  end
102
108
  end
103
109
 
104
- context "when `host` key already exists on the event" do
110
+ context "when `#{host_field_name}` key already exists on the event" do
105
111
  let(:already_exist) { "already_exist" }
106
- let(:event_map) { super.merge({ "host" => already_exist }) }
112
+ let(:event_map) { super().merge(key_as_nested_maps(host_field_name, already_exist)) }
107
113
 
108
114
  it "doesn't override it" do
109
- expect(subject.get("host")).to eq(already_exist)
115
+ expect(subject.get(host_field_name)).to eq(already_exist)
110
116
  end
111
117
  end
112
118
  end
113
119
 
114
120
  context "when a host is provided in `beat.hostname`" do
115
121
  let(:producer_host) { "newhost01" }
116
- let(:event_map) { super.merge({ "beat" => { "hostname" => producer_host }}) }
122
+ let(:event_map) { super().merge({ "beat" => { "hostname" => producer_host }}) }
117
123
 
118
- context "when no `host` key already exists on the event" do
124
+ context "when no `#{host_field_name}` key already exists on the event" do
119
125
  it "does not set the host" do
120
- expect(subject.get("host")).to be_nil
126
+ expect(subject.get(host_field_name)).to be_nil
121
127
  end
122
128
  end
123
129
 
124
130
  context "when `host` key already exists on the event" do
125
131
  let(:already_exist) { "already_exist" }
126
- let(:event_map) { super.merge({ "host" => already_exist }) }
132
+ let(:event_map) { super().merge(key_as_nested_maps(host_field_name, already_exist)) }
127
133
 
128
134
  it "doesn't override it" do
129
- expect(subject.get("host")).to eq(already_exist)
135
+ expect(subject.get(host_field_name)).to eq(already_exist)
130
136
  end
131
137
  end
132
138
  end
133
139
 
134
140
  context "when no host is provided in beat" do
135
- context "when no `host` key already exists on the event" do
141
+ context "when no `#{host_field_name}` key already exists on the event" do
136
142
  it "does not set the host" do
137
- expect(subject.get("host")).to be_nil
143
+ expect(subject.get(host_field_name)).to be_nil
138
144
  end
139
145
  end
140
146
 
141
- context "when `host` key already exists on the event" do
147
+ context "when `#{host_field_name}` key already exists on the event" do
142
148
  let(:already_exist) { "already_exist" }
143
- let(:event_map) { super.merge({ "host" => already_exist }) }
149
+ let(:event_map) { super().merge(key_as_nested_maps(host_field_name, already_exist)) }
144
150
 
145
151
  it "doesn't override it" do
146
- expect(subject.get("host")).to eq(already_exist)
152
+ expect(subject.get(host_field_name)).to eq(already_exist)
147
153
  end
148
154
  end
149
155
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-input-beats
3
3
  version: !ruby/object:Gem::Version
4
- version: 6.0.13
4
+ version: 6.1.3
5
5
  platform: java
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-01-15 00:00:00.000000000 Z
11
+ date: 2021-04-21 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -106,6 +106,20 @@ dependencies:
106
106
  - - ">="
107
107
  - !ruby/object:Gem::Version
108
108
  version: 0.3.4
109
+ - !ruby/object:Gem::Dependency
110
+ requirement: !ruby/object:Gem::Requirement
111
+ requirements:
112
+ - - "~>"
113
+ - !ruby/object:Gem::Version
114
+ version: '1.1'
115
+ name: logstash-mixin-ecs_compatibility_support
116
+ prerelease: false
117
+ type: :runtime
118
+ version_requirements: !ruby/object:Gem::Requirement
119
+ requirements:
120
+ - - "~>"
121
+ - !ruby/object:Gem::Version
122
+ version: '1.1'
109
123
  - !ruby/object:Gem::Dependency
110
124
  requirement: !ruby/object:Gem::Requirement
111
125
  requirements:
@@ -266,12 +280,12 @@ files:
266
280
  - spec/support/shared_examples.rb
267
281
  - vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-annotations/2.9.10/jackson-annotations-2.9.10.jar
268
282
  - vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-core/2.9.10/jackson-core-2.9.10.jar
269
- - vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.10.4/jackson-databind-2.9.10.4.jar
283
+ - vendor/jar-dependencies/com/fasterxml/jackson/core/jackson-databind/2.9.10.8/jackson-databind-2.9.10.8.jar
270
284
  - vendor/jar-dependencies/com/fasterxml/jackson/module/jackson-module-afterburner/2.9.10/jackson-module-afterburner-2.9.10.jar
271
285
  - vendor/jar-dependencies/io/netty/netty-all/4.1.49.Final/netty-all-4.1.49.Final.jar
272
286
  - vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1.jar
273
287
  - vendor/jar-dependencies/org/javassist/javassist/3.24.0-GA/javassist-3.24.0-GA.jar
274
- - vendor/jar-dependencies/org/logstash/beats/logstash-input-beats/6.0.13/logstash-input-beats-6.0.13.jar
288
+ - vendor/jar-dependencies/org/logstash/beats/logstash-input-beats/6.1.3/logstash-input-beats-6.1.3.jar
275
289
  homepage: http://www.elastic.co/guide/en/logstash/current/index.html
276
290
  licenses:
277
291
  - Apache License (2.0)