logstash-output-elasticsearch 11.9.3-java → 11.11.0-java

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 6d02f6b7c9fca46bf86c64ad4dc6ec39f692899d30a069c9813a174be6b7436f
4
- data.tar.gz: b68bdafacbbfa03c1eb0968ca111141353a1911a328144c62e9810de9902995c
3
+ metadata.gz: 6c35d86223e3353d75bc09c1e86ac45b164133ed877b64496e5734627f8bfe9b
4
+ data.tar.gz: 57bbcc13083b42c010c13fd51466ac9872ab09522ac4fb8e2833f429f1d18fbf
5
5
  SHA512:
6
- metadata.gz: 2fef1eeb91cb331f340b36bbd76778f16dc1baedb3d550683b2ea2a8c454b1f87c65847b44ba9ae1323d9a413fe7610c510415d275a56a84c7ea743b9dbbfb23
7
- data.tar.gz: 897c6b6d949c1fa5674faa745d4646c035058cc7ac84f58e69f0f1fe375f3374af8f07cce54ee534c6c2d2804ca6e87f8472d5b96cff0b8b1a38fc1009135fdb
6
+ metadata.gz: adfd2a4f7d288019fff8908bfd06a1e34ad5d07ca0bd1ca31d73d5394eddf2d8b0dfdbd0f8da983f5e66022d8d81c7852070be78914442cdc1b36fa7b67dc4ac
7
+ data.tar.gz: a99011dbe58769d6b9069c8ac5a4afc5b5a3682a6226d49b1d7b3d1ca687cbbabbfcd6816c68a27027b2b315a8501efdbba43013e980191b8cb5ac3ba78a56cb
data/CHANGELOG.md CHANGED
@@ -1,8 +1,14 @@
1
+ ## 11.11.0
2
+ - When using an `api_key` along with either `cloud_id` or https `hosts`, you no longer need to also specify `ssl => true` [#1065](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1065)
3
+
4
+ ## 11.10.0
5
+ - Feature: expose `dlq_routed` document metric to track the documents routed into DLQ [#1090](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1090)
6
+
1
7
  ## 11.9.3
2
8
  - DOC: clarify that `http_compression` option only affects _requests_; compressed _responses_ have always been read independent of this setting [#1030 ](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1030)
3
9
 
4
10
  ## 11.9.2
5
- - Fix broken link to Logstash Reference [#1085](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1085)
11
+ - Fix broken link to Logstash Reference [#1085](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1085)
6
12
 
7
13
  ## 11.9.1
8
14
  - Fixes a possible infinite-retry-loop that could occur when this plugin is configured with an `action` whose value contains a [sprintf-style placeholder][] that fails to be resolved for an individual event. Events in this state will be routed to the pipeline's [dead letter queue][DLQ] if it is available, or will be logged-and-dropped so that the remaining events in the batch can be processed [#1080](https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/1080)
data/docs/index.asciidoc CHANGED
@@ -406,8 +406,8 @@ For more details on actions, check out the {ref}/docs-bulk.html[Elasticsearch bu
406
406
  * Value type is <<password,password>>
407
407
  * There is no default value for this setting.
408
408
 
409
- Authenticate using Elasticsearch API key. Note that this option also requires
410
- enabling the `ssl` option.
409
+ Authenticate using Elasticsearch API key.
410
+ Note that this option also requires SSL/TLS, which can be enabled by supplying a <<plugins-{type}s-{plugin}-cloud_id>>, a list of HTTPS <<plugins-{type}s-{plugin}-hosts>>, or by setting <<plugins-{type}s-{plugin}-ssl,`ssl => true`>>.
411
411
 
412
412
  Format is `id:api_key` where `id` and `api_key` are as returned by the
413
413
  Elasticsearch {ref}/security-api-create-api-key.html[Create API key API].
@@ -1040,11 +1040,9 @@ do not use full URL here, only paths, e.g. "/sniff/_nodes/http"
1040
1040
  * Value type is <<boolean,boolean>>
1041
1041
  * There is no default value for this setting.
1042
1042
 
1043
- Enable SSL/TLS secured communication to Elasticsearch cluster. Leaving this
1044
- unspecified will use whatever scheme is specified in the URLs listed in 'hosts'.
1045
- If no explicit protocol is specified plain HTTP will be used. If SSL is
1046
- explicitly disabled here the plugin will refuse to start if an HTTPS URL is
1047
- given in 'hosts'
1043
+ Enable SSL/TLS secured communication to Elasticsearch cluster.
1044
+ Leaving this unspecified will use whatever scheme is specified in the URLs listed in <<plugins-{type}s-{plugin}-hosts>> or extracted from the <<plugins-{type}s-{plugin}-cloud_id>>.
1045
+ If no explicit protocol is specified plain HTTP will be used.
1048
1046
 
1049
1047
  [id="plugins-{type}s-{plugin}-ssl_certificate_verification"]
1050
1048
  ===== `ssl_certificate_verification`
@@ -23,10 +23,14 @@ module LogStash; module PluginMixins; module ElasticSearch
23
23
  # because they must be executed prior to building the client and logstash
24
24
  # monitoring and management rely on directly calling build_client
25
25
  # see https://github.com/logstash-plugins/logstash-output-elasticsearch/pull/934#pullrequestreview-396203307
26
- validate_authentication
27
26
  fill_hosts_from_cloud_id
27
+ validate_authentication
28
+
28
29
  setup_hosts
29
30
 
31
+
32
+ params['ssl'] = effectively_ssl? unless params.include?('ssl')
33
+
30
34
  # inject the TrustStrategy from CATrustedFingerprintSupport
31
35
  if trust_strategy_for_ca_trusted_fingerprint
32
36
  params["ssl_trust_strategy"] = trust_strategy_for_ca_trusted_fingerprint
@@ -49,7 +53,7 @@ module LogStash; module PluginMixins; module ElasticSearch
49
53
  raise LogStash::ConfigurationError, 'Multiple authentication options are specified, please only use one of user/password, cloud_auth or api_key'
50
54
  end
51
55
 
52
- if @api_key && @api_key.value && @ssl != true
56
+ if @api_key && @api_key.value && !effectively_ssl?
53
57
  raise(LogStash::ConfigurationError, "Using api_key authentication requires SSL/TLS secured communication using the `ssl => true` option")
54
58
  end
55
59
 
@@ -69,6 +73,15 @@ module LogStash; module PluginMixins; module ElasticSearch
69
73
  end
70
74
  end
71
75
 
76
+ def effectively_ssl?
77
+ return @ssl unless @ssl.nil?
78
+
79
+ hosts = Array(@hosts)
80
+ return false if hosts.nil? || hosts.empty?
81
+
82
+ hosts.all? { |host| host && host.scheme == "https" }
83
+ end
84
+
72
85
  def hosts_default?(hosts)
73
86
  # NOTE: would be nice if pipeline allowed us a clean way to detect a config default :
74
87
  hosts.is_a?(Array) && hosts.size == 1 && hosts.first.equal?(LogStash::PluginMixins::ElasticSearch::APIConfigs::DEFAULT_HOST)
@@ -208,12 +221,12 @@ module LogStash; module PluginMixins; module ElasticSearch
208
221
 
209
222
  def handle_dlq_response(message, action, status, response)
210
223
  _, action_params = action.event, [action[0], action[1], action[2]]
211
-
224
+
212
225
  # TODO: Change this to send a map with { :status => status, :action => action } in the future
213
226
  detailed_message = "#{message} status: #{status}, action: #{action_params}, response: #{response}"
214
-
227
+
215
228
  log_level = dig_value(response, 'index', 'error', 'type') == 'invalid_index_name_exception' ? :error : :warn
216
-
229
+
217
230
  handle_dlq_status(action.event, log_level, detailed_message)
218
231
  end
219
232
 
@@ -273,7 +286,7 @@ module LogStash; module PluginMixins; module ElasticSearch
273
286
  next
274
287
  elsif @dlq_codes.include?(status)
275
288
  handle_dlq_response("Could not index event to Elasticsearch.", action, status, response)
276
- @document_level_metrics.increment(:non_retryable_failures)
289
+ @document_level_metrics.increment(:dlq_routed)
277
290
  next
278
291
  else
279
292
  # only log what the user whitelisted
@@ -1,7 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-output-elasticsearch'
3
- s.version = '11.9.3'
4
-
3
+ s.version = '11.11.0'
5
4
  s.licenses = ['apache-2.0']
6
5
  s.summary = "Stores logs in Elasticsearch"
7
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -58,7 +58,7 @@ describe "metrics", :integration => true do
58
58
  end
59
59
 
60
60
  it "increases number of successful and non retryable documents" do
61
- expect(document_level_metrics).to receive(:increment).with(:non_retryable_failures).once
61
+ expect(document_level_metrics).to receive(:increment).with(:dlq_routed).once
62
62
  expect(document_level_metrics).to receive(:increment).with(:successes).once
63
63
  subject.multi_receive(bulk)
64
64
  end
@@ -17,12 +17,17 @@ describe LogStash::Outputs::ElasticSearch do
17
17
  allow_any_instance_of(LogStash::Outputs::ElasticSearch::HttpClient::Pool).to receive(:start)
18
18
  end
19
19
 
20
+ let(:spy_http_client_builder!) do
21
+ allow(described_class::HttpClientBuilder).to receive(:build).with(any_args).and_call_original
22
+ end
23
+
20
24
  let(:after_successful_connection_thread_mock) do
21
25
  double('after_successful_connection_thread', value: true)
22
26
  end
23
27
 
24
28
  before(:each) do
25
29
  if do_register
30
+ spy_http_client_builder!
26
31
  stub_http_client_pool!
27
32
 
28
33
  allow(subject).to receive(:finish_register) # stub-out thread completion (to avoid error log entries)
@@ -1003,29 +1008,88 @@ describe LogStash::Outputs::ElasticSearch do
1003
1008
  let(:api_key) { "some_id:some_api_key" }
1004
1009
  let(:base64_api_key) { "ApiKey c29tZV9pZDpzb21lX2FwaV9rZXk=" }
1005
1010
 
1006
- context "when set without ssl" do
1011
+ shared_examples 'secure api-key authenticated client' do
1012
+ let(:do_register) { true }
1013
+
1014
+ it 'adds the appropriate Authorization header to the manticore client' do
1015
+ expect(manticore_options[:headers]).to eq({ "Authorization" => base64_api_key })
1016
+ end
1017
+ it 'is provides ssl=>true to the http client builder' do; aggregate_failures do
1018
+ expect(described_class::HttpClientBuilder).to have_received(:build).with(anything, anything, hash_including('ssl'=>true))
1019
+ end; end
1020
+ end
1021
+
1022
+ context "when set without ssl => true" do
1007
1023
  let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
1008
1024
  let(:options) { { "api_key" => api_key } }
1009
1025
 
1010
1026
  it "should raise a configuration error" do
1011
1027
  expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
1012
1028
  end
1029
+
1030
+ context 'with cloud_id' do
1031
+ let(:sample_cloud_id) { 'sample:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJGFjMzFlYmI5MDI0MTc3MzE1NzA0M2MzNGZkMjZmZDQ2OjkyNDMkYTRjMDYyMzBlNDhjOGZjZTdiZTg4YTA3NGEzYmIzZTA6OTI0NA==' }
1032
+ let(:options) { super().merge('cloud_id' => sample_cloud_id) }
1033
+
1034
+ it_behaves_like 'secure api-key authenticated client'
1035
+ end
1013
1036
  end
1014
1037
 
1015
- context "when set without ssl but with a https host" do
1038
+ context "when set without ssl specified but with an https host" do
1016
1039
  let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
1017
1040
  let(:options) { { "hosts" => ["https://some.host.com"], "api_key" => api_key } }
1018
1041
 
1042
+ it_behaves_like 'secure api-key authenticated client'
1043
+ end
1044
+
1045
+ context "when set without ssl specified but with an http host`" do
1046
+ let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
1047
+ let(:options) { { "hosts" => ["http://some.host.com"], "api_key" => api_key } }
1048
+
1049
+ it "should raise a configuration error" do
1050
+ expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
1051
+ end
1052
+ end
1053
+
1054
+ context "when set with `ssl => false`" do
1055
+ let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
1056
+ let(:options) { { "ssl" => "false", "api_key" => api_key } }
1057
+
1019
1058
  it "should raise a configuration error" do
1020
1059
  expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
1021
1060
  end
1022
1061
  end
1023
1062
 
1024
1063
  context "when set" do
1025
- let(:options) { { "ssl" => true, "api_key" => ::LogStash::Util::Password.new(api_key) } }
1064
+ let(:options) { { "api_key" => ::LogStash::Util::Password.new(api_key) } }
1026
1065
 
1027
- it "should use the custom headers in the adapter options" do
1028
- expect(manticore_options[:headers]).to eq({ "Authorization" => base64_api_key })
1066
+ context "with ssl => true" do
1067
+ let(:options) { super().merge("ssl" => true) }
1068
+ it_behaves_like 'secure api-key authenticated client'
1069
+ end
1070
+
1071
+ context "with ssl => false" do
1072
+ let(:options) { super().merge("ssl" => "false") }
1073
+
1074
+ let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
1075
+ it "should raise a configuration error" do
1076
+ expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
1077
+ end
1078
+ end
1079
+
1080
+ context "without ssl specified" do
1081
+ context "with an https host" do
1082
+ let(:options) { super().merge("hosts" => ["https://some.host.com"]) }
1083
+ it_behaves_like 'secure api-key authenticated client'
1084
+ end
1085
+ context "with an http host`" do
1086
+ let(:do_register) { false } # this is what we want to test, so we disable the before(:each) call
1087
+ let(:options) { { "hosts" => ["http://some.host.com"], "api_key" => api_key } }
1088
+
1089
+ it "should raise a configuration error" do
1090
+ expect { subject.register }.to raise_error LogStash::ConfigurationError, /requires SSL\/TLS/
1091
+ end
1092
+ end
1029
1093
  end
1030
1094
  end
1031
1095
 
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-elasticsearch
3
3
  version: !ruby/object:Gem::Version
4
- version: 11.9.3
4
+ version: 11.11.0
5
5
  platform: java
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-10-04 00:00:00.000000000 Z
11
+ date: 2022-10-12 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement