logstash-output-amazon_es 6.4.1-java → 7.1.0-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile +4 -0
- data/NOTICE.TXT +10 -10
- data/README.md +75 -39
- data/lib/logstash/outputs/amazon_es/common.rb +15 -7
- data/lib/logstash/outputs/amazon_es/common_configs.rb +5 -1
- data/lib/logstash/outputs/amazon_es/elasticsearch-template-es7x.json +29 -31
- data/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb +17 -15
- data/lib/logstash/outputs/amazon_es/http_client/pool.rb +29 -18
- data/lib/logstash/outputs/amazon_es/http_client.rb +18 -23
- data/lib/logstash/outputs/amazon_es/http_client_builder.rb +11 -3
- data/lib/logstash/outputs/amazon_es/template_manager.rb +5 -1
- data/lib/logstash/outputs/amazon_es.rb +21 -5
- data/logstash-output-amazon_es.gemspec +7 -5
- data/spec/es_spec_helper.rb +4 -5
- data/spec/unit/http_client_builder_spec.rb +6 -2
- data/spec/unit/outputs/elasticsearch/http_client/manticore_adapter_spec.rb +7 -3
- data/spec/unit/outputs/elasticsearch/http_client/pool_spec.rb +6 -2
- data/spec/unit/outputs/elasticsearch/http_client_spec.rb +11 -5
- data/spec/unit/outputs/elasticsearch/template_manager_spec.rb +5 -1
- data/spec/unit/outputs/elasticsearch_spec.rb +6 -2
- data/spec/unit/outputs/error_whitelist_spec.rb +5 -1
- metadata +12 -20
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 827216d23ab630ee15e9ab6fd7324ef2dd506ff3ca7296ada51afa09e78b08e3
|
4
|
+
data.tar.gz: 67471b7f10a922559cd3247a370f80b5989c30947bd3981ea2233c1b0f310a18
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7ce4a46376dce1d4f0cd684086b28d94c30fd1fb841e59562713b2c674b3ed01df89e6257a78612df7cad1b88569b771e2e513581981366994563fecd7d23edc
|
7
|
+
data.tar.gz: b7c513e943a05a6e78cf602d66cf6139d42a36881acd23b7d7fbebb1319d61c9ceff7753b2c1303f3c29b0afcf621f80d741e013bf8e79a35cdc2561bbe76694
|
data/Gemfile
CHANGED
data/NOTICE.TXT
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
Logstash-output-amazon_es Plugin
|
2
|
-
Copyright
|
2
|
+
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
3
3
|
|
4
|
-
Licensed under the Apache License, Version 2.0 (the "License");
|
5
|
-
you may not use this file except in compliance with the License.
|
6
|
-
You may obtain a copy of the License at
|
7
4
|
|
8
|
-
|
5
|
+
This product includes software from the Apache-2.0 licensed Elasticsearch project.
|
9
6
|
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
7
|
+
It requires the following notice:
|
8
|
+
|
9
|
+
Elasticsearch
|
10
|
+
Copyright 2012-2015 Elasticsearch
|
11
|
+
Copyright 2020 Elastic and contributors
|
12
|
+
|
13
|
+
This product includes software developed by The Apache Software
|
14
|
+
Foundation (http://www.apache.org/).
|
data/README.md
CHANGED
@@ -1,6 +1,9 @@
|
|
1
1
|
# Logstash Plugin
|
2
2
|
|
3
|
-
This is a plugin for [Logstash](https://github.com/elastic/logstash)
|
3
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash) which outputs
|
4
|
+
to [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/)
|
5
|
+
(successor to Amazon Elasticsearch Service) using
|
6
|
+
[SigV4 signing](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
|
4
7
|
|
5
8
|
## License
|
6
9
|
|
@@ -8,16 +11,43 @@ This library is licensed under Apache License 2.0.
|
|
8
11
|
|
9
12
|
## Compatibility
|
10
13
|
|
11
|
-
The following table shows the versions of logstash and logstash-output-amazon_es
|
14
|
+
The following table shows the versions of logstash and logstash-output-amazon_es plugin was built with.
|
12
15
|
|
13
|
-
|
|
14
|
-
|
15
|
-
| 6.0.0
|
16
|
-
| 6.4.
|
16
|
+
| logstash-output-amazon_es | Logstash |
|
17
|
+
|---------------------------|----------|
|
18
|
+
| 6.0.0 | < 6.0.0 |
|
19
|
+
| 6.4.2 | >= 6.0.0 |
|
20
|
+
| 7.0.1 | >= 7.0.0 |
|
17
21
|
|
18
|
-
## Configuration for Amazon Elasticsearch Output Plugin
|
19
22
|
|
20
|
-
|
23
|
+
Also, logstash-output-amazon_es plugin versions 6.4.0 and newer are tested to be compatible with Elasticsearch 6.5 and greater.
|
24
|
+
|
25
|
+
| logstash-output-amazon_es | Elasticsearch |
|
26
|
+
| ------------- |----------|
|
27
|
+
| 6.4.0+ | 6.5+ |
|
28
|
+
|
29
|
+
|
30
|
+
## Installation
|
31
|
+
|
32
|
+
To install the latest version, use the normal Logstash plugin script.
|
33
|
+
|
34
|
+
```sh
|
35
|
+
bin/logstash-plugin install logstash-output-amazon_es
|
36
|
+
```
|
37
|
+
|
38
|
+
If you want to use old version of logstash-output-amazon_es, you can use the `--version`
|
39
|
+
flag to specify the version. For example:
|
40
|
+
|
41
|
+
```sh
|
42
|
+
bin/logstash-plugin install --version 6.4.2 logstash-output-amazon_es
|
43
|
+
```
|
44
|
+
|
45
|
+
|
46
|
+
|
47
|
+
|
48
|
+
## Configuration for Amazon Elasticsearch Service Output Plugin
|
49
|
+
|
50
|
+
To run the Logstash Output Amazon Elasticsearch Service plugin, simply add a configuration following the below documentation.
|
21
51
|
|
22
52
|
An example configuration:
|
23
53
|
|
@@ -46,30 +76,48 @@ output {
|
|
46
76
|
* aws_access_key_id, :validate => :string - optional AWS access key
|
47
77
|
* aws_secret_access_key, :validate => :string - optional AWS secret key
|
48
78
|
|
49
|
-
|
79
|
+
The credential resolution logic can be described as follows:
|
50
80
|
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
81
|
+
- User passed `aws_access_key_id` and `aws_secret_access_key` in `amazon_es` configuration
|
82
|
+
- Environment variables - `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (RECOMMENDED since they are recognized by all the AWS SDKs and CLI except for .NET), or `AWS_ACCESS_KEY` and `AWS_SECRET_KEY` (only recognized by Java SDK)
|
83
|
+
- Credential profiles file at the default location (`~/.aws/credentials`) shared by all AWS SDKs and the AWS CLI
|
84
|
+
- Instance profile credentials delivered through the Amazon EC2 metadata service
|
55
85
|
|
56
86
|
- template (path) - You can set the path to your own template here, if you so desire. If not set, the included template will be used.
|
57
87
|
- template_name (string, default => "logstash") - defines how the template is named inside Elasticsearch
|
58
88
|
- port (string, default 443) - Amazon Elasticsearch Service listens on port 443 for HTTPS (default) and port 80 for HTTP. Tweak this value for a custom proxy.
|
59
89
|
- protocol (string, default https) - The protocol used to connect to the Amazon Elasticsearch Service
|
90
|
+
- max_bulk_bytes - The max size for a bulk request in bytes. Default is 20MB. It is recommended not to change this value unless needed. For guidance on changing this value, please consult the table for network limits for your instance type: https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html#network-limits
|
60
91
|
|
61
92
|
After 6.4.0, users can't set batch size in this output plugin config. However, users can still set batch size in logstash.yml file.
|
93
|
+
|
94
|
+
### Advanced Optional Parameters
|
95
|
+
|
96
|
+
Starting logstash-output-amazon_es v7.1.0, we have introduced the following optional parameters to resolve specific use cases:
|
97
|
+
|
98
|
+
- service_name (string, default => "es") - Users can define any service name to which the plugin will send a SigV4 signed request
|
99
|
+
- skip_healthcheck (boolean, default => false) - Boolean to skip healthcheck API and set the major ES version to 7
|
100
|
+
- skip_template_installation (boolean, default => false) - Boolean to allow users to skip installing templates in usecases that don't require them
|
101
|
+
|
62
102
|
## Developing
|
63
103
|
|
64
|
-
### 1.
|
104
|
+
### 1. Prerequisites
|
105
|
+
To get started, you can install JRuby with the Bundler gem using [RVM](https://rvm.io/rvm/install)
|
106
|
+
|
107
|
+
rvm install jruby
|
108
|
+
|
109
|
+
### 2. Plugin Development and Testing
|
65
110
|
|
66
111
|
#### Code
|
67
112
|
|
68
|
-
1.
|
113
|
+
1. Verify JRuby is already installed
|
114
|
+
|
115
|
+
```sh
|
116
|
+
jruby -v
|
117
|
+
```
|
69
118
|
|
70
|
-
2. Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. [Example plugins](https://github.com/logstash-plugins?query=example) exist.
|
71
119
|
|
72
|
-
|
120
|
+
2. Install dependencies:
|
73
121
|
|
74
122
|
```sh
|
75
123
|
bundle install
|
@@ -89,14 +137,14 @@ After 6.4.0, users can't set batch size in this output plugin config. However, u
|
|
89
137
|
bundle exec rspec
|
90
138
|
```
|
91
139
|
|
92
|
-
###
|
140
|
+
### 3. Running your unpublished plugin in Logstash
|
93
141
|
|
94
|
-
####
|
142
|
+
#### 3.1 Run in a local Logstash clone
|
95
143
|
|
96
144
|
1. Edit Logstash `Gemfile` and add the local plugin path, for example:
|
97
145
|
|
98
146
|
```ruby
|
99
|
-
gem "logstash-
|
147
|
+
gem "logstash-output-amazon_es", :path => "/your/local/logstash-output-amazon_es"
|
100
148
|
```
|
101
149
|
|
102
150
|
2. Install the plugin:
|
@@ -112,12 +160,12 @@ After 6.4.0, users can't set batch size in this output plugin config. However, u
|
|
112
160
|
3. Run Logstash with your plugin:
|
113
161
|
|
114
162
|
```sh
|
115
|
-
bin/logstash -e '
|
163
|
+
bin/logstash -e 'output {amazon_es {}}'
|
116
164
|
```
|
117
165
|
|
118
166
|
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply re-run Logstash.
|
119
167
|
|
120
|
-
####
|
168
|
+
#### 3.2 Run in an installed Logstash
|
121
169
|
|
122
170
|
Before build your `Gemfile`, please make sure use JRuby. Here is how you can know your local Ruby version:
|
123
171
|
|
@@ -128,37 +176,25 @@ rvm list
|
|
128
176
|
Please make sure you current using JRuby. Here is how you can change to JRuby
|
129
177
|
|
130
178
|
```sh
|
131
|
-
rvm jruby
|
179
|
+
rvm jruby-9.2.5.0
|
132
180
|
```
|
133
181
|
|
134
|
-
You can use the same **
|
182
|
+
You can use the same **3.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory. You can also build the gem and install it using:
|
135
183
|
|
136
184
|
1. Build your plugin gem:
|
137
185
|
|
138
186
|
```sh
|
139
|
-
gem build logstash-
|
187
|
+
gem build logstash-output-amazon_es.gemspec
|
140
188
|
```
|
141
189
|
|
142
|
-
2. Install the plugin from the Logstash home:
|
190
|
+
2. Install the plugin from the Logstash home. Please be sure to check the version number against the actual Gem file. Run:
|
143
191
|
|
144
192
|
```sh
|
145
|
-
|
146
|
-
bin/logstash-plugin install --no-verify
|
147
|
-
|
148
|
-
# Prior to Logstash 2.3
|
149
|
-
bin/plugin install --no-verify
|
193
|
+
bin/logstash-plugin install /your/local/logstash-output-amazon_es/logstash-output-amazon_es-7.0.1-java.gem
|
150
194
|
```
|
151
195
|
|
152
196
|
3. Start Logstash and test the plugin.
|
153
197
|
|
154
|
-
## Old version support
|
155
|
-
|
156
|
-
If you want to use old version of logstash-output-amazon_es, you can install with this:
|
157
|
-
```sh
|
158
|
-
bin/logstash-plugin install logstash-output-amazon_es -v 2.0.0
|
159
|
-
```
|
160
|
-
|
161
|
-
|
162
198
|
|
163
199
|
## Contributing
|
164
200
|
|
@@ -1,8 +1,12 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/outputs/amazon_es/template_manager"
|
2
6
|
|
3
|
-
module LogStash; module Outputs; class
|
7
|
+
module LogStash; module Outputs; class AmazonElasticSearch;
|
4
8
|
module Common
|
5
|
-
attr_reader :client, :hosts
|
9
|
+
attr_reader :client, :hosts, :skip_template_installation
|
6
10
|
|
7
11
|
# These codes apply to documents, not at the request level
|
8
12
|
DOC_DLQ_CODES = [400, 404]
|
@@ -115,8 +119,12 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
115
119
|
end
|
116
120
|
|
117
121
|
def install_template
|
118
|
-
|
119
|
-
|
122
|
+
if skip_template_installation == false
|
123
|
+
TemplateManager.install_template(self)
|
124
|
+
@template_installed.make_true
|
125
|
+
elsif skip_template_installation == true
|
126
|
+
@template_installed.make_true
|
127
|
+
end
|
120
128
|
end
|
121
129
|
|
122
130
|
def check_action_validity
|
@@ -274,7 +282,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
274
282
|
es_actions = actions.map {|action_type, params, event| [action_type, params, event.to_hash]}
|
275
283
|
response = @client.bulk(es_actions)
|
276
284
|
response
|
277
|
-
rescue ::LogStash::Outputs::
|
285
|
+
rescue ::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::HostUnreachableError => e
|
278
286
|
# If we can't even connect to the server let's just print out the URL (:hosts is actually a URL)
|
279
287
|
# and let the user sort it out from there
|
280
288
|
@logger.error(
|
@@ -290,7 +298,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
290
298
|
sleep_interval = sleep_for_interval(sleep_interval)
|
291
299
|
@bulk_request_metrics.increment(:failures)
|
292
300
|
retry unless @stopping.true?
|
293
|
-
rescue ::LogStash::Outputs::
|
301
|
+
rescue ::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::NoConnectionAvailableError => e
|
294
302
|
@logger.error(
|
295
303
|
"Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down?",
|
296
304
|
:error_message => e.message,
|
@@ -301,7 +309,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
301
309
|
sleep_interval = next_sleep_interval(sleep_interval)
|
302
310
|
@bulk_request_metrics.increment(:failures)
|
303
311
|
retry unless @stopping.true?
|
304
|
-
rescue ::LogStash::Outputs::
|
312
|
+
rescue ::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError => e
|
305
313
|
@bulk_request_metrics.increment(:failures)
|
306
314
|
log_hash = {:code => e.response_code, :url => e.url.sanitized.to_s}
|
307
315
|
log_hash[:body] = e.response_body if @logger.debug? # Generally this is too verbose
|
@@ -1,6 +1,10 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require 'forwardable' # Needed for logstash core SafeURI. We need to patch this in core: https://github.com/elastic/logstash/pull/5978
|
2
6
|
|
3
|
-
module LogStash; module Outputs; class
|
7
|
+
module LogStash; module Outputs; class AmazonElasticSearch
|
4
8
|
module CommonConfigs
|
5
9
|
def self.included(mod)
|
6
10
|
# The index to write events to. This can be dynamic using the `%{foo}` syntax.
|
@@ -1,44 +1,42 @@
|
|
1
1
|
{
|
2
2
|
"template" : "logstash-*",
|
3
|
-
"version" :
|
3
|
+
"version" : 60002,
|
4
4
|
"settings" : {
|
5
5
|
"index.refresh_interval" : "5s",
|
6
6
|
"number_of_shards": 1
|
7
7
|
},
|
8
8
|
"mappings" : {
|
9
|
-
"
|
10
|
-
"
|
11
|
-
"
|
12
|
-
|
13
|
-
|
14
|
-
"
|
15
|
-
|
16
|
-
"norms" : false
|
17
|
-
}
|
9
|
+
"dynamic_templates" : [ {
|
10
|
+
"message_field" : {
|
11
|
+
"path_match" : "message",
|
12
|
+
"match_mapping_type" : "string",
|
13
|
+
"mapping" : {
|
14
|
+
"type" : "text",
|
15
|
+
"norms" : false
|
18
16
|
}
|
19
|
-
}
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
}
|
17
|
+
}
|
18
|
+
}, {
|
19
|
+
"string_fields" : {
|
20
|
+
"match" : "*",
|
21
|
+
"match_mapping_type" : "string",
|
22
|
+
"mapping" : {
|
23
|
+
"type" : "text", "norms" : false,
|
24
|
+
"fields" : {
|
25
|
+
"keyword" : { "type": "keyword", "ignore_above": 256 }
|
28
26
|
}
|
29
27
|
}
|
30
|
-
}
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
}
|
28
|
+
}
|
29
|
+
} ],
|
30
|
+
"properties" : {
|
31
|
+
"@timestamp": { "type": "date"},
|
32
|
+
"@version": { "type": "keyword"},
|
33
|
+
"geoip" : {
|
34
|
+
"dynamic": true,
|
35
|
+
"properties" : {
|
36
|
+
"ip": { "type": "ip" },
|
37
|
+
"location" : { "type" : "geo_point" },
|
38
|
+
"latitude" : { "type" : "half_float" },
|
39
|
+
"longitude" : { "type" : "half_float" }
|
42
40
|
}
|
43
41
|
}
|
44
42
|
}
|
@@ -1,9 +1,13 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require 'manticore'
|
2
6
|
require 'cgi'
|
3
7
|
require 'aws-sdk-core'
|
4
8
|
require 'uri'
|
5
9
|
|
6
|
-
module LogStash; module Outputs; class
|
10
|
+
module LogStash; module Outputs; class AmazonElasticSearch; class HttpClient;
|
7
11
|
DEFAULT_HEADERS = { "content-type" => "application/json" }
|
8
12
|
|
9
13
|
CredentialConfig = Struct.new(
|
@@ -33,12 +37,16 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
33
37
|
@port = options[:port] || 9200
|
34
38
|
@protocol = options[:protocol] || 'http'
|
35
39
|
@region = options[:region] || 'us-east-1'
|
36
|
-
@
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
40
|
+
@service_name = options[:service_name] || 'es'
|
41
|
+
aws_access_key_id = options[:aws_access_key_id] || nil
|
42
|
+
aws_secret_access_key = options[:aws_secret_access_key] || nil
|
43
|
+
session_token = options[:session_token] || nil
|
44
|
+
profile = options[:profile] || 'default'
|
45
|
+
instance_cred_retries = options[:instance_profile_credentials_retries] || 0
|
46
|
+
instance_cred_timeout = options[:instance_profile_credentials_timeout] || 1
|
47
|
+
|
48
|
+
credential_config = CredentialConfig.new(aws_access_key_id, aws_secret_access_key, session_token, profile, instance_cred_retries, instance_cred_timeout, @region)
|
49
|
+
@credentials = Aws::CredentialProviderChain.new(credential_config).resolve
|
42
50
|
|
43
51
|
if options[:proxy]
|
44
52
|
options[:proxy] = manticore_proxy_hash(options[:proxy])
|
@@ -102,13 +110,7 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
102
110
|
key = Seahorse::Client::Http::Request.new(options={:endpoint=>url, :http_method => method.to_s.upcase,
|
103
111
|
:headers => params[:headers],:body => params[:body]})
|
104
112
|
|
105
|
-
|
106
|
-
|
107
|
-
credential_config = CredentialConfig.new(@aws_access_key_id, @aws_secret_access_key, @session_token, @profile, @instance_cred_retries, @instance_cred_timeout, @region)
|
108
|
-
|
109
|
-
|
110
|
-
credentials = Aws::CredentialProviderChain.new(credential_config).resolve
|
111
|
-
aws_signer = Aws::Signers::V4.new(credentials, 'es', @region )
|
113
|
+
aws_signer = Aws::Signers::V4.new(@credentials, @service_name, @region )
|
112
114
|
|
113
115
|
|
114
116
|
signed_key = aws_signer.sign(key)
|
@@ -126,7 +128,7 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
126
128
|
# template installation. We might need a better story around this later
|
127
129
|
# but for our current purposes this is correct
|
128
130
|
if resp.code < 200 || resp.code > 299 && resp.code != 404
|
129
|
-
raise ::LogStash::Outputs::
|
131
|
+
raise ::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError.new(resp.code, request_uri, body, resp.body)
|
130
132
|
end
|
131
133
|
|
132
134
|
resp
|
@@ -1,4 +1,8 @@
|
|
1
|
-
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
5
|
+
module LogStash; module Outputs; class AmazonElasticSearch; class HttpClient;
|
2
6
|
class Pool
|
3
7
|
class NoConnectionAvailableError < Error; end
|
4
8
|
class BadResponseCodeError < Error
|
@@ -28,7 +32,7 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
28
32
|
end
|
29
33
|
end
|
30
34
|
|
31
|
-
attr_reader :logger, :adapter, :sniffing, :sniffer_delay, :resurrect_delay, :healthcheck_path, :sniffing_path, :bulk_path
|
35
|
+
attr_reader :logger, :adapter, :sniffing, :sniffer_delay, :resurrect_delay, :healthcheck_path, :sniffing_path, :bulk_path, :skip_healthcheck
|
32
36
|
|
33
37
|
ROOT_URI_PATH = '/'.freeze
|
34
38
|
|
@@ -40,6 +44,7 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
40
44
|
:resurrect_delay => 5,
|
41
45
|
:sniffing => false,
|
42
46
|
:sniffer_delay => 10,
|
47
|
+
:skip_healthcheck => false,
|
43
48
|
}.freeze
|
44
49
|
|
45
50
|
def initialize(logger, adapter, initial_urls=[], options={})
|
@@ -57,6 +62,7 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
57
62
|
@resurrect_delay = merged[:resurrect_delay]
|
58
63
|
@sniffing = merged[:sniffing]
|
59
64
|
@sniffer_delay = merged[:sniffer_delay]
|
65
|
+
@skip_healthcheck = merged[:skip_healthcheck]
|
60
66
|
end
|
61
67
|
|
62
68
|
# Used for all concurrent operations in this class
|
@@ -238,25 +244,30 @@ module LogStash; module Outputs; class ElasticSearch; class HttpClient;
|
|
238
244
|
|
239
245
|
def healthcheck!
|
240
246
|
# Try to keep locking granularity low such that we don't affect IO...
|
241
|
-
@state_mutex.synchronize { @url_info.select {|url,meta| meta[:state] != :alive } }.each do |url,meta|
|
247
|
+
@state_mutex.synchronize { @url_info.select { |url, meta| meta[:state] != :alive } }.each do |url, meta|
|
242
248
|
begin
|
243
|
-
|
249
|
+
if skip_healthcheck == false
|
250
|
+
logger.info("Running health check to see if an Elasticsearch connection is working",
|
244
251
|
:healthcheck_url => url, :path => @healthcheck_path)
|
245
|
-
|
246
|
-
|
247
|
-
|
248
|
-
|
249
|
-
|
250
|
-
|
251
|
-
|
252
|
-
|
253
|
-
|
254
|
-
|
255
|
-
|
256
|
-
|
257
|
-
|
258
|
-
|
252
|
+
response = perform_request_to_url(url, :head, @healthcheck_path)
|
253
|
+
# If no exception was raised it must have succeeded!
|
254
|
+
logger.warn("Restored connection to ES instance", :url => url.sanitized.to_s)
|
255
|
+
# We reconnected to this node, check its ES version
|
256
|
+
es_version = get_es_version(url)
|
257
|
+
@state_mutex.synchronize do
|
258
|
+
meta[:version] = es_version
|
259
|
+
major = major_version(es_version)
|
260
|
+
if !@maximum_seen_major_version
|
261
|
+
@logger.info("ES Output version determined", :es_version => major)
|
262
|
+
set_new_major_version(major)
|
263
|
+
elsif major > @maximum_seen_major_version
|
264
|
+
@logger.warn("Detected a node with a higher major version than previously observed. This could be the result of an amazon_es cluster upgrade.", :previous_major => @maximum_seen_major_version, :new_major => major, :node_url => url)
|
265
|
+
set_new_major_version(major)
|
266
|
+
end
|
267
|
+
meta[:state] = :alive
|
259
268
|
end
|
269
|
+
elsif skip_healthcheck == true
|
270
|
+
set_new_major_version(7)
|
260
271
|
meta[:state] = :alive
|
261
272
|
end
|
262
273
|
rescue HostUnreachableError, BadResponseCodeError => e
|
@@ -1,3 +1,7 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/outputs/amazon_es"
|
2
6
|
require "cabin"
|
3
7
|
require "base64"
|
@@ -7,24 +11,11 @@ require 'cgi'
|
|
7
11
|
require 'zlib'
|
8
12
|
require 'stringio'
|
9
13
|
|
10
|
-
module LogStash; module Outputs; class
|
11
|
-
# This is a constant instead of a config option because
|
12
|
-
# there really isn't a good reason to configure it.
|
13
|
-
#
|
14
|
-
# The criteria used are:
|
15
|
-
# 1. We need a number that's less than 100MiB because ES
|
16
|
-
# won't accept bulks larger than that.
|
17
|
-
# 2. It must be large enough to amortize the connection constant
|
18
|
-
# across multiple requests.
|
19
|
-
# 3. It must be small enough that even if multiple threads hit this size
|
20
|
-
# we won't use a lot of heap.
|
21
|
-
#
|
22
|
-
# We wound up agreeing that a number greater than 10 MiB and less than 100MiB
|
23
|
-
# made sense. We picked one on the lowish side to not use too much heap.
|
24
|
-
TARGET_BULK_BYTES = 20 * 1024 * 1024 # 20MiB
|
25
|
-
|
14
|
+
module LogStash; module Outputs; class AmazonElasticSearch;
|
26
15
|
class HttpClient
|
27
|
-
attr_reader :client, :options, :logger, :pool, :action_count, :recv_count
|
16
|
+
attr_reader :client, :options, :logger, :pool, :action_count, :recv_count, :max_bulk_bytes
|
17
|
+
|
18
|
+
|
28
19
|
# This is here in case we use DEFAULT_OPTIONS in the future
|
29
20
|
# DEFAULT_OPTIONS = {
|
30
21
|
# :setting => value
|
@@ -38,7 +29,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
38
29
|
# * `:user` - String. The user to use for authentication.
|
39
30
|
# * `:password` - String. The password to use for authentication.
|
40
31
|
# * `:timeout` - Float. A duration value, in seconds, after which a socket
|
41
|
-
# operation or request will be aborted if not yet
|
32
|
+
# operation or request will be aborted if not yet successful
|
42
33
|
# * `:client_settings` - a hash; see below for keys.
|
43
34
|
#
|
44
35
|
# The `client_settings` key is a has that can contain other settings:
|
@@ -54,6 +45,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
54
45
|
@metric = options[:metric]
|
55
46
|
@bulk_request_metrics = @metric.namespace(:bulk_requests)
|
56
47
|
@bulk_response_metrics = @bulk_request_metrics.namespace(:responses)
|
48
|
+
@max_bulk_bytes = options[:max_bulk_bytes]
|
57
49
|
|
58
50
|
# Again, in case we use DEFAULT_OPTIONS in the future, uncomment this.
|
59
51
|
# @options = DEFAULT_OPTIONS.merge(options)
|
@@ -121,7 +113,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
121
113
|
action.map {|line| LogStash::Json.dump(line)}.join("\n") :
|
122
114
|
LogStash::Json.dump(action)
|
123
115
|
as_json << "\n"
|
124
|
-
if (body_stream.size + as_json.bytesize) >
|
116
|
+
if (body_stream.size + as_json.bytesize) > @max_bulk_bytes
|
125
117
|
bulk_responses << bulk_send(body_stream) unless body_stream.size == 0
|
126
118
|
end
|
127
119
|
stream_writer.write(as_json)
|
@@ -152,7 +144,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
152
144
|
|
153
145
|
if response.code != 200
|
154
146
|
url = ::LogStash::Util::SafeURI.new(response.final_url)
|
155
|
-
raise ::LogStash::Outputs::
|
147
|
+
raise ::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError.new(
|
156
148
|
response.code, url, body_stream.to_s, response.body
|
157
149
|
)
|
158
150
|
end
|
@@ -284,6 +276,8 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
284
276
|
|
285
277
|
adapter_options[:region] = options[:region]
|
286
278
|
|
279
|
+
adapter_options[:service_name] = options[:service_name]
|
280
|
+
|
287
281
|
adapter_options[:port] = options[:port]
|
288
282
|
|
289
283
|
adapter_options[:protocol] = options[:protocol]
|
@@ -292,7 +286,7 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
292
286
|
|
293
287
|
adapter_options[:aws_secret_access_key] = options[:aws_secret_access_key]
|
294
288
|
|
295
|
-
adapter_class = ::LogStash::Outputs::
|
289
|
+
adapter_class = ::LogStash::Outputs::AmazonElasticSearch::HttpClient::ManticoreAdapter
|
296
290
|
adapter = adapter_class.new(@logger, adapter_options)
|
297
291
|
end
|
298
292
|
|
@@ -306,11 +300,12 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
306
300
|
:healthcheck_path => options[:healthcheck_path],
|
307
301
|
:resurrect_delay => options[:resurrect_delay],
|
308
302
|
:url_normalizer => self.method(:host_to_url),
|
309
|
-
:metric => options[:metric]
|
303
|
+
:metric => options[:metric],
|
304
|
+
:skip_healthcheck => options[:skip_healthcheck],
|
310
305
|
}
|
311
306
|
pool_options[:scheme] = self.scheme if self.scheme
|
312
307
|
|
313
|
-
pool_class = ::LogStash::Outputs::
|
308
|
+
pool_class = ::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool
|
314
309
|
full_urls = @options[:hosts].map {|h| host_to_url(h) }
|
315
310
|
pool = pool_class.new(@logger, adapter, full_urls, pool_options)
|
316
311
|
pool.start
|
@@ -1,6 +1,10 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require 'cgi'
|
2
6
|
|
3
|
-
module LogStash; module Outputs; class
|
7
|
+
module LogStash; module Outputs; class AmazonElasticSearch;
|
4
8
|
module HttpClientBuilder
|
5
9
|
def self.build(logger, hosts, params)
|
6
10
|
client_settings = {
|
@@ -100,12 +104,16 @@ module LogStash; module Outputs; class ElasticSearch;
|
|
100
104
|
:protocol => params["protocol"],
|
101
105
|
:port => params["port"],
|
102
106
|
:region => params["region"],
|
107
|
+
:service_name => params["service_name"],
|
103
108
|
:aws_access_key_id => params["aws_access_key_id"],
|
104
|
-
:aws_secret_access_key => params["aws_secret_access_key"]
|
109
|
+
:aws_secret_access_key => params["aws_secret_access_key"],
|
110
|
+
:max_bulk_bytes => params["max_bulk_bytes"],
|
111
|
+
:skip_healthcheck => params["skip_healthcheck"])
|
112
|
+
)
|
105
113
|
end
|
106
114
|
|
107
115
|
def self.create_http_client(options)
|
108
|
-
LogStash::Outputs::
|
116
|
+
LogStash::Outputs::AmazonElasticSearch::HttpClient.new(options)
|
109
117
|
end
|
110
118
|
|
111
119
|
def self.setup_ssl(logger, params)
|
@@ -1,4 +1,8 @@
|
|
1
|
-
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
5
|
+
module LogStash; module Outputs; class AmazonElasticSearch
|
2
6
|
class TemplateManager
|
3
7
|
# To be mixed into the amazon_es plugin base
|
4
8
|
def self.install_template(plugin)
|
@@ -1,3 +1,7 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
# encoding: utf-8
|
2
6
|
require "logstash/namespace"
|
3
7
|
require "logstash/environment"
|
@@ -83,7 +87,7 @@ require "forwardable"
|
|
83
87
|
# For requests compression, regardless of the Elasticsearch version, users have to enable `http_compression`
|
84
88
|
# setting in their Logstash config file.
|
85
89
|
#
|
86
|
-
class LogStash::Outputs::
|
90
|
+
class LogStash::Outputs::AmazonElasticSearch < LogStash::Outputs::Base
|
87
91
|
declare_threadsafe!
|
88
92
|
|
89
93
|
require "logstash/outputs/amazon_es/http_client"
|
@@ -92,10 +96,10 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
|
|
92
96
|
require "logstash/outputs/amazon_es/common"
|
93
97
|
|
94
98
|
# Protocol agnostic (i.e. non-http, non-java specific) configs go here
|
95
|
-
include(LogStash::Outputs::
|
99
|
+
include(LogStash::Outputs::AmazonElasticSearch::CommonConfigs)
|
96
100
|
|
97
101
|
# Protocol agnostic methods
|
98
|
-
include(LogStash::Outputs::
|
102
|
+
include(LogStash::Outputs::AmazonElasticSearch::Common)
|
99
103
|
|
100
104
|
config_name "amazon_es"
|
101
105
|
|
@@ -121,11 +125,14 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
|
|
121
125
|
# You can set the remote port as part of the host, or explicitly here as well
|
122
126
|
config :port, :validate => :number, :default => 443
|
123
127
|
|
124
|
-
# Sets the protocol thats used to connect to
|
128
|
+
# Sets the protocol thats used to connect to elasticsearch
|
125
129
|
config :protocol, :validate => :string, :default => "https"
|
126
130
|
|
127
131
|
#Signing specific details
|
128
132
|
config :region, :validate => :string, :default => "us-east-1"
|
133
|
+
|
134
|
+
#Service name, default is `es`
|
135
|
+
config :service_name, :validate => :string, :default => "es"
|
129
136
|
# Credential resolution logic works as follows:
|
130
137
|
#
|
131
138
|
# - User passed aws_access_key_id and aws_secret_access_key in aes configuration
|
@@ -248,9 +255,18 @@ class LogStash::Outputs::ElasticSearch < LogStash::Outputs::Base
|
|
248
255
|
# Custom Headers to send on each request to amazon_es nodes
|
249
256
|
config :custom_headers, :validate => :hash, :default => {}
|
250
257
|
|
258
|
+
#Max bulk size in bytes
|
259
|
+
config :max_bulk_bytes, :validate => :number, :default => 20 * 1024 * 1024
|
260
|
+
|
261
|
+
#Option for user to skip Healthcheck API for a host when set to True
|
262
|
+
config :skip_healthcheck, :validate => :boolean, :default => false
|
263
|
+
|
264
|
+
#Allow user to skip installing template when set to True
|
265
|
+
config :skip_template_installation, :validate => :boolean, :default => false
|
266
|
+
|
251
267
|
def build_client
|
252
268
|
params["metric"] = metric
|
253
|
-
@client ||= ::LogStash::Outputs::
|
269
|
+
@client ||= ::LogStash::Outputs::AmazonElasticSearch::HttpClientBuilder.build(@logger, @hosts, params)
|
254
270
|
end
|
255
271
|
|
256
272
|
def close
|
@@ -1,6 +1,10 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
Gem::Specification.new do |s|
|
2
6
|
s.name = 'logstash-output-amazon_es'
|
3
|
-
s.version = '
|
7
|
+
s.version = '7.1.0'
|
4
8
|
s.licenses = ['Apache-2.0']
|
5
9
|
s.summary = "Logstash Output to Amazon Elasticsearch Service"
|
6
10
|
s.description = "Output events to Amazon Elasticsearch Service with V4 signing"
|
@@ -26,9 +30,7 @@ Gem::Specification.new do |s|
|
|
26
30
|
s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
|
27
31
|
s.add_runtime_dependency 'aws-sdk', '>= 2.3.22', '~> 2'
|
28
32
|
|
29
|
-
s.add_development_dependency 'logstash-codec-plain'
|
30
|
-
s.add_development_dependency 'logstash-devutils',
|
33
|
+
s.add_development_dependency 'logstash-codec-plain'
|
34
|
+
s.add_development_dependency 'logstash-devutils', "~> 1.3", ">= 1.3.1"
|
31
35
|
s.add_development_dependency 'flores', '~> 0'
|
32
|
-
# Still used in some specs, we should remove this ASAP
|
33
|
-
s.add_development_dependency 'elasticsearch', '~> 0'
|
34
36
|
end
|
data/spec/es_spec_helper.rb
CHANGED
@@ -1,6 +1,9 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/devutils/rspec/spec_helper"
|
2
6
|
require 'manticore'
|
3
|
-
require 'elasticsearch'
|
4
7
|
|
5
8
|
# by default exclude secure_integration tests unless requested
|
6
9
|
# normal integration specs are already excluded by devutils' spec helper
|
@@ -13,10 +16,6 @@ module ESHelper
|
|
13
16
|
"127.0.0.1"
|
14
17
|
end
|
15
18
|
|
16
|
-
def get_client
|
17
|
-
Elasticsearch::Client.new(:hosts => [get_host_port])
|
18
|
-
end
|
19
|
-
|
20
19
|
def self.es_version
|
21
20
|
RSpec.configuration.filter[:es_version] || ENV['ES_VERSION']
|
22
21
|
end
|
@@ -1,11 +1,15 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/devutils/rspec/spec_helper"
|
2
6
|
require "logstash/outputs/amazon_es"
|
3
7
|
require "logstash/outputs/amazon_es/http_client"
|
4
8
|
require "logstash/outputs/amazon_es/http_client_builder"
|
5
9
|
|
6
|
-
describe LogStash::Outputs::
|
10
|
+
describe LogStash::Outputs::AmazonElasticSearch::HttpClientBuilder do
|
7
11
|
describe "auth setup with url encodable passwords" do
|
8
|
-
let(:klass) { LogStash::Outputs::
|
12
|
+
let(:klass) { LogStash::Outputs::AmazonElasticSearch::HttpClientBuilder }
|
9
13
|
let(:user) { "foo@bar"}
|
10
14
|
let(:password) {"baz@blah" }
|
11
15
|
let(:password_secured) do
|
@@ -1,7 +1,11 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/devutils/rspec/spec_helper"
|
2
6
|
require "logstash/outputs/amazon_es/http_client"
|
3
7
|
|
4
|
-
describe LogStash::Outputs::
|
8
|
+
describe LogStash::Outputs::AmazonElasticSearch::HttpClient::ManticoreAdapter do
|
5
9
|
let(:logger) { Cabin::Channel.get }
|
6
10
|
let(:options) { {:aws_access_key_id => 'AAAAAAAAAAAAAAAAAAAA',
|
7
11
|
:aws_secret_access_key => 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'} }
|
@@ -34,12 +38,12 @@ describe LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter do
|
|
34
38
|
uri_with_path = uri.clone
|
35
39
|
uri_with_path.path = "/"
|
36
40
|
|
37
|
-
expect(::LogStash::Outputs::
|
41
|
+
expect(::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError).to receive(:new).
|
38
42
|
with(resp.code, uri_with_path, nil, resp.body).and_call_original
|
39
43
|
|
40
44
|
expect do
|
41
45
|
subject.perform_request(uri, :get, "/")
|
42
|
-
end.to raise_error(::LogStash::Outputs::
|
46
|
+
end.to raise_error(::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError)
|
43
47
|
end
|
44
48
|
end
|
45
49
|
|
@@ -1,10 +1,14 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/devutils/rspec/spec_helper"
|
2
6
|
require "logstash/outputs/amazon_es/http_client"
|
3
7
|
require "json"
|
4
8
|
|
5
|
-
describe LogStash::Outputs::
|
9
|
+
describe LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool do
|
6
10
|
let(:logger) { Cabin::Channel.get }
|
7
|
-
let(:adapter) { LogStash::Outputs::
|
11
|
+
let(:adapter) { LogStash::Outputs::AmazonElasticSearch::HttpClient::ManticoreAdapter.new(logger,
|
8
12
|
{:aws_access_key_id => 'AAAAAAAAAAAAAAAAAAAA',
|
9
13
|
:aws_secret_access_key => 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'}) }
|
10
14
|
let(:initial_urls) { [::LogStash::Util::SafeURI.new("http://localhost:9200")] }
|
@@ -1,8 +1,12 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/devutils/rspec/spec_helper"
|
2
6
|
require "logstash/outputs/amazon_es/http_client"
|
3
7
|
require "java"
|
4
8
|
|
5
|
-
describe LogStash::Outputs::
|
9
|
+
describe LogStash::Outputs::AmazonElasticSearch::HttpClient do
|
6
10
|
let(:ssl) { nil }
|
7
11
|
let(:base_options) do
|
8
12
|
opts = {
|
@@ -12,7 +16,8 @@ describe LogStash::Outputs::ElasticSearch::HttpClient do
|
|
12
16
|
:protocol => "http",
|
13
17
|
:port => 9200,
|
14
18
|
:aws_access_key_id => "AAAAAAAAAAAAAAAAAAAA",
|
15
|
-
:aws_secret_access_key => "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
|
19
|
+
:aws_secret_access_key => "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA",
|
20
|
+
:max_bulk_bytes => 20 * 1024 * 1024
|
16
21
|
}
|
17
22
|
|
18
23
|
if !ssl.nil? # Shortcut to set this
|
@@ -29,6 +34,7 @@ describe LogStash::Outputs::ElasticSearch::HttpClient do
|
|
29
34
|
let(:ipv6_hostname) { "[::1]" }
|
30
35
|
let(:ipv4_hostname) { "127.0.0.1" }
|
31
36
|
let(:port) { 9200 }
|
37
|
+
let(:max_bulk_bytes) { 20 * 1024 * 1024 }
|
32
38
|
let(:protocol) {"http"}
|
33
39
|
let(:aws_access_key_id) {"AAAAAAAAAAAAAAAAAAAA"}
|
34
40
|
let(:aws_secret_access_key) {"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}
|
@@ -163,7 +169,7 @@ describe LogStash::Outputs::ElasticSearch::HttpClient do
|
|
163
169
|
]}
|
164
170
|
|
165
171
|
context "if a message is over TARGET_BULK_BYTES" do
|
166
|
-
let(:target_bulk_bytes) {
|
172
|
+
let(:target_bulk_bytes) { 20 * 1024 * 1024 }
|
167
173
|
let(:message) { "a" * (target_bulk_bytes + 1) }
|
168
174
|
|
169
175
|
it "should be handled properly" do
|
@@ -189,7 +195,7 @@ describe LogStash::Outputs::ElasticSearch::HttpClient do
|
|
189
195
|
end
|
190
196
|
|
191
197
|
context "if one exceeds TARGET_BULK_BYTES" do
|
192
|
-
let(:target_bulk_bytes) {
|
198
|
+
let(:target_bulk_bytes) { 20 * 1024 * 1024 }
|
193
199
|
let(:message1) { "a" * (target_bulk_bytes + 1) }
|
194
200
|
it "executes two bulk_send operations" do
|
195
201
|
allow(subject).to receive(:join_bulk_responses)
|
@@ -201,7 +207,7 @@ describe LogStash::Outputs::ElasticSearch::HttpClient do
|
|
201
207
|
end
|
202
208
|
|
203
209
|
describe "sniffing" do
|
204
|
-
let(:client) { LogStash::Outputs::
|
210
|
+
let(:client) { LogStash::Outputs::AmazonElasticSearch::HttpClient.new(base_options.merge(client_opts)) }
|
205
211
|
|
206
212
|
context "with sniffing enabled" do
|
207
213
|
let(:client_opts) { {:sniffing => true, :sniffing_delay => 1 } }
|
@@ -1,9 +1,13 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/devutils/rspec/spec_helper"
|
2
6
|
require "logstash/outputs/amazon_es/http_client"
|
3
7
|
require "java"
|
4
8
|
require "json"
|
5
9
|
|
6
|
-
describe LogStash::Outputs::
|
10
|
+
describe LogStash::Outputs::AmazonElasticSearch::TemplateManager do
|
7
11
|
|
8
12
|
describe ".default_template_path" do
|
9
13
|
context "amazon_es 1.x" do
|
@@ -1,8 +1,12 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require_relative "../../../spec/es_spec_helper"
|
2
6
|
require "flores/random"
|
3
7
|
require "logstash/outputs/amazon_es"
|
4
8
|
|
5
|
-
describe LogStash::Outputs::
|
9
|
+
describe LogStash::Outputs::AmazonElasticSearch do
|
6
10
|
subject { described_class.new(options) }
|
7
11
|
let(:options) { { "aws_access_key_id" => "AAAAAAAAAAAAAAAAAAAA",
|
8
12
|
"aws_secret_access_key" => "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"} }
|
@@ -222,7 +226,7 @@ describe LogStash::Outputs::ElasticSearch do
|
|
222
226
|
context "429 errors" do
|
223
227
|
let(:event) { ::LogStash::Event.new("foo" => "bar") }
|
224
228
|
let(:error) do
|
225
|
-
::LogStash::Outputs::
|
229
|
+
::LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError.new(
|
226
230
|
429, double("url").as_null_object, double("request body"), double("response body")
|
227
231
|
)
|
228
232
|
end
|
@@ -1,3 +1,7 @@
|
|
1
|
+
# SPDX-License-Identifier: Apache-2.0
|
2
|
+
# See NOTICE for attribution details.
|
3
|
+
# Modifications Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
|
4
|
+
|
1
5
|
require "logstash/outputs/amazon_es"
|
2
6
|
require_relative "../../../spec/es_spec_helper"
|
3
7
|
|
@@ -14,7 +18,7 @@ describe "whitelisting error types in expected behavior" do
|
|
14
18
|
"aws_access_key_id" => "AAAAAAAAAAAAAAAAAAAA",
|
15
19
|
"aws_secret_access_key" => "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"} }
|
16
20
|
|
17
|
-
subject { LogStash::Outputs::
|
21
|
+
subject { LogStash::Outputs::AmazonElasticSearch.new(settings) }
|
18
22
|
|
19
23
|
before :each do
|
20
24
|
allow(subject.logger).to receive(:warn)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-output-amazon_es
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version:
|
4
|
+
version: 7.1.0
|
5
5
|
platform: java
|
6
6
|
authors:
|
7
7
|
- Amazon
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2022-01-26 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
@@ -107,7 +107,7 @@ dependencies:
|
|
107
107
|
- !ruby/object:Gem::Dependency
|
108
108
|
requirement: !ruby/object:Gem::Requirement
|
109
109
|
requirements:
|
110
|
-
- - "
|
110
|
+
- - ">="
|
111
111
|
- !ruby/object:Gem::Version
|
112
112
|
version: '0'
|
113
113
|
name: logstash-codec-plain
|
@@ -115,7 +115,7 @@ dependencies:
|
|
115
115
|
type: :development
|
116
116
|
version_requirements: !ruby/object:Gem::Requirement
|
117
117
|
requirements:
|
118
|
-
- - "
|
118
|
+
- - ">="
|
119
119
|
- !ruby/object:Gem::Version
|
120
120
|
version: '0'
|
121
121
|
- !ruby/object:Gem::Dependency
|
@@ -123,7 +123,10 @@ dependencies:
|
|
123
123
|
requirements:
|
124
124
|
- - "~>"
|
125
125
|
- !ruby/object:Gem::Version
|
126
|
-
version: '
|
126
|
+
version: '1.3'
|
127
|
+
- - ">="
|
128
|
+
- !ruby/object:Gem::Version
|
129
|
+
version: 1.3.1
|
127
130
|
name: logstash-devutils
|
128
131
|
prerelease: false
|
129
132
|
type: :development
|
@@ -131,28 +134,17 @@ dependencies:
|
|
131
134
|
requirements:
|
132
135
|
- - "~>"
|
133
136
|
- !ruby/object:Gem::Version
|
134
|
-
version: '
|
135
|
-
-
|
136
|
-
requirement: !ruby/object:Gem::Requirement
|
137
|
-
requirements:
|
138
|
-
- - "~>"
|
139
|
-
- !ruby/object:Gem::Version
|
140
|
-
version: '0'
|
141
|
-
name: flores
|
142
|
-
prerelease: false
|
143
|
-
type: :development
|
144
|
-
version_requirements: !ruby/object:Gem::Requirement
|
145
|
-
requirements:
|
146
|
-
- - "~>"
|
137
|
+
version: '1.3'
|
138
|
+
- - ">="
|
147
139
|
- !ruby/object:Gem::Version
|
148
|
-
version:
|
140
|
+
version: 1.3.1
|
149
141
|
- !ruby/object:Gem::Dependency
|
150
142
|
requirement: !ruby/object:Gem::Requirement
|
151
143
|
requirements:
|
152
144
|
- - "~>"
|
153
145
|
- !ruby/object:Gem::Version
|
154
146
|
version: '0'
|
155
|
-
name:
|
147
|
+
name: flores
|
156
148
|
prerelease: false
|
157
149
|
type: :development
|
158
150
|
version_requirements: !ruby/object:Gem::Requirement
|