logstash-output-amazon_es 7.1.0-java → 8.0.0-java

Sign up to get free protection for your applications and to get access to all the features.
Files changed (5) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +84 -7
  3. data/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb +21 -19
  4. data/logstash-output-amazon_es.gemspec +2 -2
  5. metadata +19 -26
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 827216d23ab630ee15e9ab6fd7324ef2dd506ff3ca7296ada51afa09e78b08e3
4
- data.tar.gz: 67471b7f10a922559cd3247a370f80b5989c30947bd3981ea2233c1b0f310a18
3
+ metadata.gz: b2ee2078a352ec70e02b1a7ea155920fa46321a5338988e51a16fb252f4fc3e5
4
+ data.tar.gz: 9ddbcfb3abedb40c41c27784c3c76ed8358ddda3ed57a15be569fec0a5ecbd03
5
5
  SHA512:
6
- metadata.gz: 7ce4a46376dce1d4f0cd684086b28d94c30fd1fb841e59562713b2c674b3ed01df89e6257a78612df7cad1b88569b771e2e513581981366994563fecd7d23edc
7
- data.tar.gz: b7c513e943a05a6e78cf602d66cf6139d42a36881acd23b7d7fbebb1319d61c9ceff7753b2c1303f3c29b0afcf621f80d741e013bf8e79a35cdc2561bbe76694
6
+ metadata.gz: bf71c118ab4fbb00a2b317f380fb5e0e05fd8db866f3f01b63ec3cb1ad066dae2f78f77a9e9c4980c5c824879efdff711b0bd94e130ea32e81516a47fb3f8d36
7
+ data.tar.gz: f5543338eb6057ed5c29653fedd8e2710a14b610617b2e7ce94f2ca91d4770aadcaece33ffcc7152b8f3bb103eef26ff7f6eb5c3f3d65e9c1705670486e41043
data/README.md CHANGED
@@ -1,7 +1,69 @@
1
- # Logstash Plugin
1
+ # Logstash Output Plugin
2
+
3
+
4
+ This plugin is now in maintenance mode. We will supply bug fixes and security patches for v7.2.X, older versions are no longer supported. This change is because the OpenSearch Project created a new Logstash output plugin
5
+ [logstash-output-opensearch](https://github.com/opensearch-project/logstash-output-opensearch) which ships events from
6
+ Logstash to OpenSearch 1.x and Elasticsearch 7.x clusters, and also supports SigV4 signing. Having similar functionality
7
+ plugins can be redundant, so we plan to eventually replace this logstash-output-amazon_es plugin with the logstash-output-opensearch
8
+ plugin.
9
+
10
+ To help you migrate to [logstash-output-opensearch](https://github.com/opensearch-project/logstash-output-opensearch) plugin, please
11
+ find below a brief migration guide.
12
+
13
+ ## Migrating to logstash-output-opensearch plugin
14
+
15
+
16
+ This guide provides instructions for existing users of logstash-output-amazon_es plugin to migrate to
17
+ logstash-output-opensearch plugin.
18
+
19
+ ### Configuration Changes
20
+ * The plugin name will change from `amazon_es` to `opensearch`.
21
+ * If using HTTPS this must be explicitly configured because `opensearch` plugin does not default to it like `amazon_es` does:
22
+ * The protocol must be included in `hosts` as `https` (or option `ssl` added with value `true`)
23
+ * `port` must explicitly specified as `443`
24
+ * A new parameter `auth_type` will be added to the Config to support SigV4 signing.
25
+ * The `region` parameter will move under `auth_type`.
26
+ * Credential parameters `aws_access_key_id` and `aws_secret_access_key` will move under `auth_type`.
27
+ * The `type` value for `auth_type` for SigV4 signing will be set to `aws_iam`.
28
+
29
+ For the Logstash configuration provided in [Configuration for Amazon Elasticsearch Service Output Plugin
30
+ ](#configuration-for-amazon-elasticsearch-service-output-plugin), here's a mapped example configuration for
31
+ logstash-output-opensearch plugin:
32
+
33
+ ```
34
+ output {
35
+ opensearch {
36
+ hosts => ["https://hostname:port"]
37
+ auth_type => {
38
+ type => 'aws_iam'
39
+ aws_access_key_id => 'ACCESS_KEY'
40
+ aws_secret_access_key => 'SECRET_KEY'
41
+ region => 'us-west-2'
42
+ }
43
+ index => "logstash-logs-%{+YYYY.MM.dd}"
44
+ }
45
+ }
46
+ ```
47
+
48
+ ### Installation of logstash-output-opensearch plugin
49
+ This [Installation Guide](https://opensearch.org/docs/latest/clients/logstash/index/) has instructions on installing the
50
+ logstash-output-opensearch plugin in two ways: Linux (ARM64/X64) OR Docker (ARM64/X64).
51
+
52
+ To install the latest version of logstash-output-opensearch, use the normal Logstash plugin installation command:
53
+ ```shell
54
+ bin/logstash-plugin install logstash-output-opensearch
55
+ ```
56
+
57
+ # Using the logstash-output-amazon_es plugin
58
+
59
+
60
+ The remainder of this document is for using or developing the logstash-output-amazon_es plugin.
61
+
62
+
63
+ ## Overview
2
64
 
3
65
  This is a plugin for [Logstash](https://github.com/elastic/logstash) which outputs
4
- to [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/)
66
+ to [Amazon OpenSearch Service](https://aws.amazon.com/opensearch-service/)
5
67
  (successor to Amazon Elasticsearch Service) using
6
68
  [SigV4 signing](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
7
69
 
@@ -18,9 +80,10 @@ The following table shows the versions of logstash and logstash-output-amazon_es
18
80
  | 6.0.0 | < 6.0.0 |
19
81
  | 6.4.2 | >= 6.0.0 |
20
82
  | 7.0.1 | >= 7.0.0 |
83
+ | 7.1.0 | >= 7.0.0 |
84
+ | 8.0.0 | >= 7.0.0 |
21
85
 
22
-
23
- Also, logstash-output-amazon_es plugin versions 6.4.0 and newer are tested to be compatible with Elasticsearch 6.5 and greater.
86
+ Also, logstash-output-amazon_es plugin versions 6.4.0 and newer are tested to be compatible with Elasticsearch 6.5 and greater.
24
87
 
25
88
  | logstash-output-amazon_es | Elasticsearch |
26
89
  | ------------- |----------|
@@ -42,8 +105,20 @@ flag to specify the version. For example:
42
105
  bin/logstash-plugin install --version 6.4.2 logstash-output-amazon_es
43
106
  ```
44
107
 
45
-
46
-
108
+ Starting in 8.0.0, the aws sdk version is bumped to v3. In order for all other AWS plugins to work together, please remove pre-installed plugins and install logstash-integration-aws plugin as follows. See also https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
109
+ ```
110
+ # Remove existing logstash aws plugins and install logstash-integration-aws to keep sdk dependency the same
111
+ # https://github.com/logstash-plugins/logstash-mixin-aws/issues/38
112
+ /usr/share/logstash/bin/logstash-plugin remove logstash-input-s3
113
+ /usr/share/logstash/bin/logstash-plugin remove logstash-input-sqs
114
+ /usr/share/logstash/bin/logstash-plugin remove logstash-output-s3
115
+ /usr/share/logstash/bin/logstash-plugin remove logstash-output-sns
116
+ /usr/share/logstash/bin/logstash-plugin remove logstash-output-sqs
117
+ /usr/share/logstash/bin/logstash-plugin remove logstash-output-cloudwatch
118
+
119
+ /usr/share/logstash/bin/logstash-plugin install --version 0.1.0.pre logstash-integration-aws
120
+ bin/logstash-plugin install --version 8.0.0 logstash-output-amazon_es
121
+ ```
47
122
 
48
123
  ## Configuration for Amazon Elasticsearch Service Output Plugin
49
124
 
@@ -104,7 +179,9 @@ Starting logstash-output-amazon_es v7.1.0, we have introduced the following opti
104
179
  ### 1. Prerequisites
105
180
  To get started, you can install JRuby with the Bundler gem using [RVM](https://rvm.io/rvm/install)
106
181
 
107
- rvm install jruby
182
+ ```shell
183
+ rvm install jruby-9.2.5.0
184
+ ```
108
185
 
109
186
  ### 2. Plugin Development and Testing
110
187
 
data/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb CHANGED
@@ -51,10 +51,10 @@ module LogStash; module Outputs; class AmazonElasticSearch; class HttpClient;
51
51
  if options[:proxy]
52
52
  options[:proxy] = manticore_proxy_hash(options[:proxy])
53
53
  end
54
-
54
+
55
55
  @manticore = ::Manticore::Client.new(options)
56
56
  end
57
-
57
+
58
58
  # Transform the proxy option to a hash. Manticore's support for non-hash
59
59
  # proxy options is broken. This was fixed in https://github.com/cheald/manticore/commit/34a00cee57a56148629ed0a47c329181e7319af5
60
60
  # but this is not yet released
@@ -89,12 +89,12 @@ module LogStash; module Outputs; class AmazonElasticSearch; class HttpClient;
89
89
  params[:body] = body if body
90
90
 
91
91
  if url.user
92
- params[:auth] = {
92
+ params[:auth] = {
93
93
  :user => CGI.unescape(url.user),
94
94
  # We have to unescape the password here since manticore won't do it
95
95
  # for us unless its part of the URL
96
- :password => CGI.unescape(url.password),
97
- :eager => true
96
+ :password => CGI.unescape(url.password),
97
+ :eager => true
98
98
  }
99
99
  end
100
100
 
@@ -107,16 +107,18 @@ module LogStash; module Outputs; class AmazonElasticSearch; class HttpClient;
107
107
  end
108
108
 
109
109
 
110
- key = Seahorse::Client::Http::Request.new(options={:endpoint=>url, :http_method => method.to_s.upcase,
110
+ request = Seahorse::Client::Http::Request.new(options={:endpoint=>url, :http_method => method.to_s.upcase,
111
111
  :headers => params[:headers],:body => params[:body]})
112
112
 
113
- aws_signer = Aws::Signers::V4.new(@credentials, @service_name, @region )
114
-
115
-
116
- signed_key = aws_signer.sign(key)
117
- params[:headers] = params[:headers].merge(signed_key.headers)
118
-
113
+ aws_signer = Aws::Sigv4::Signer.new(service: @service_name, region: @region, credentials_provider: @credentials)
119
114
 
115
+ signed_key = aws_signer.sign_request(
116
+ http_method: request.http_method,
117
+ url: url,
118
+ headers: params[:headers],
119
+ body: params[:body]
120
+ )
121
+ params[:headers] = params[:headers].merge(signed_key.headers)
120
122
 
121
123
  resp = @manticore.send(method.downcase, request_uri.to_s, params)
122
124
 
@@ -136,27 +138,27 @@ module LogStash; module Outputs; class AmazonElasticSearch; class HttpClient;
136
138
 
137
139
  def format_url(url, path_and_query=nil)
138
140
  request_uri = url.clone
139
-
141
+
140
142
  # We excise auth info from the URL in case manticore itself tries to stick
141
143
  # sensitive data in a thrown exception or log data
142
144
  request_uri.user = nil
143
145
  request_uri.password = nil
144
146
 
145
147
  return request_uri.to_s if path_and_query.nil?
146
-
148
+
147
149
  parsed_path_and_query = java.net.URI.new(path_and_query)
148
-
150
+
149
151
  query = request_uri.query
150
152
  parsed_query = parsed_path_and_query.query
151
-
153
+
152
154
  new_query_parts = [request_uri.query, parsed_path_and_query.query].select do |part|
153
155
  part && !part.empty? # Skip empty nil and ""
154
156
  end
155
-
157
+
156
158
  request_uri.query = new_query_parts.join("&") unless new_query_parts.empty?
157
-
159
+
158
160
  request_uri.path = "#{request_uri.path}/#{parsed_path_and_query.path}".gsub(/\/{2,}/, "/")
159
-
161
+
160
162
  request_uri
161
163
  end
162
164
 
data/logstash-output-amazon_es.gemspec CHANGED
@@ -4,7 +4,7 @@
4
4
 
5
5
  Gem::Specification.new do |s|
6
6
  s.name = 'logstash-output-amazon_es'
7
- s.version = '7.1.0'
7
+ s.version = '8.0.0'
8
8
  s.licenses = ['Apache-2.0']
9
9
  s.summary = "Logstash Output to Amazon Elasticsearch Service"
10
10
  s.description = "Output events to Amazon Elasticsearch Service with V4 signing"
@@ -28,7 +28,7 @@ Gem::Specification.new do |s|
28
28
  s.add_runtime_dependency 'stud', ['>= 0.0.17', '~> 0.0']
29
29
  s.add_runtime_dependency 'cabin', ['~> 0.6']
30
30
  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
31
- s.add_runtime_dependency 'aws-sdk', '>= 2.3.22', '~> 2'
31
+ s.add_runtime_dependency 'aws-sdk', '~> 3'
32
32
 
33
33
  s.add_development_dependency 'logstash-codec-plain'
34
34
  s.add_development_dependency 'logstash-devutils', "~> 1.3", ">= 1.3.1"
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-amazon_es
3
3
  version: !ruby/object:Gem::Version
4
- version: 7.1.0
4
+ version: 8.0.0
5
5
  platform: java
6
6
  authors:
7
7
  - Amazon
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-01-26 00:00:00.000000000 Z
11
+ date: 2022-09-15 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -20,8 +20,8 @@ dependencies:
20
20
  - !ruby/object:Gem::Version
21
21
  version: 1.0.0
22
22
  name: manticore
23
- prerelease: false
24
23
  type: :runtime
24
+ prerelease: false
25
25
  version_requirements: !ruby/object:Gem::Requirement
26
26
  requirements:
27
27
  - - ">="
@@ -33,23 +33,23 @@ dependencies:
33
33
  - !ruby/object:Gem::Dependency
34
34
  requirement: !ruby/object:Gem::Requirement
35
35
  requirements:
36
- - - ">="
37
- - !ruby/object:Gem::Version
38
- version: 0.0.17
39
36
  - - "~>"
40
37
  - !ruby/object:Gem::Version
41
38
  version: '0.0'
39
+ - - ">="
40
+ - !ruby/object:Gem::Version
41
+ version: 0.0.17
42
42
  name: stud
43
- prerelease: false
44
43
  type: :runtime
44
+ prerelease: false
45
45
  version_requirements: !ruby/object:Gem::Requirement
46
46
  requirements:
47
- - - ">="
48
- - !ruby/object:Gem::Version
49
- version: 0.0.17
50
47
  - - "~>"
51
48
  - !ruby/object:Gem::Version
52
49
  version: '0.0'
50
+ - - ">="
51
+ - !ruby/object:Gem::Version
52
+ version: 0.0.17
53
53
  - !ruby/object:Gem::Dependency
54
54
  requirement: !ruby/object:Gem::Requirement
55
55
  requirements:
@@ -57,8 +57,8 @@ dependencies:
57
57
  - !ruby/object:Gem::Version
58
58
  version: '0.6'
59
59
  name: cabin
60
- prerelease: false
61
60
  type: :runtime
61
+ prerelease: false
62
62
  version_requirements: !ruby/object:Gem::Requirement
63
63
  requirements:
64
64
  - - "~>"
@@ -74,8 +74,8 @@ dependencies:
74
74
  - !ruby/object:Gem::Version
75
75
  version: '2.99'
76
76
  name: logstash-core-plugin-api
77
- prerelease: false
78
77
  type: :runtime
78
+ prerelease: false
79
79
  version_requirements: !ruby/object:Gem::Requirement
80
80
  requirements:
81
81
  - - ">="
@@ -87,23 +87,17 @@ dependencies:
87
87
  - !ruby/object:Gem::Dependency
88
88
  requirement: !ruby/object:Gem::Requirement
89
89
  requirements:
90
- - - ">="
91
- - !ruby/object:Gem::Version
92
- version: 2.3.22
93
90
  - - "~>"
94
91
  - !ruby/object:Gem::Version
95
- version: '2'
92
+ version: '3'
96
93
  name: aws-sdk
97
- prerelease: false
98
94
  type: :runtime
95
+ prerelease: false
99
96
  version_requirements: !ruby/object:Gem::Requirement
100
97
  requirements:
101
- - - ">="
102
- - !ruby/object:Gem::Version
103
- version: 2.3.22
104
98
  - - "~>"
105
99
  - !ruby/object:Gem::Version
106
- version: '2'
100
+ version: '3'
107
101
  - !ruby/object:Gem::Dependency
108
102
  requirement: !ruby/object:Gem::Requirement
109
103
  requirements:
@@ -111,8 +105,8 @@ dependencies:
111
105
  - !ruby/object:Gem::Version
112
106
  version: '0'
113
107
  name: logstash-codec-plain
114
- prerelease: false
115
108
  type: :development
109
+ prerelease: false
116
110
  version_requirements: !ruby/object:Gem::Requirement
117
111
  requirements:
118
112
  - - ">="
@@ -128,8 +122,8 @@ dependencies:
128
122
  - !ruby/object:Gem::Version
129
123
  version: 1.3.1
130
124
  name: logstash-devutils
131
- prerelease: false
132
125
  type: :development
126
+ prerelease: false
133
127
  version_requirements: !ruby/object:Gem::Requirement
134
128
  requirements:
135
129
  - - "~>"
@@ -145,8 +139,8 @@ dependencies:
145
139
  - !ruby/object:Gem::Version
146
140
  version: '0'
147
141
  name: flores
148
- prerelease: false
149
142
  type: :development
143
+ prerelease: false
150
144
  version_requirements: !ruby/object:Gem::Requirement
151
145
  requirements:
152
146
  - - "~>"
@@ -205,8 +199,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
205
199
  - !ruby/object:Gem::Version
206
200
  version: '0'
207
201
  requirements: []
208
- rubyforge_project:
209
- rubygems_version: 2.7.6
202
+ rubygems_version: 3.0.1
210
203
  signing_key:
211
204
  specification_version: 4
212
205
  summary: Logstash Output to Amazon Elasticsearch Service