logstash-output-dynatrace 0.6.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9281b9e498b9986e45d25eaadc74358f95547551baf97132483533997edf134d
4
- data.tar.gz: 27bc3095ec54575becfaf75eb2b7bca7d186f95b2ab239713e83f8b9f58b363b
3
+ metadata.gz: 4db961537a03cb5b21240ab63ea56877ceebcf0a075c5db85feb736b787d05df
4
+ data.tar.gz: 185005382c4ac576fde60eeffa8d3f1a5e037445b3dd92b2126be22f0a3481dc
5
5
  SHA512:
6
- metadata.gz: f4863a8fae10fe23afc85d4a6e31290ba3c0628d974c57c8cec6306adc2ba1da5c73905c7597c0954cb095ea1a24c6e182d028d8d7289f05951f77c214255b48
7
- data.tar.gz: 53281ec7c6a7a8f6ac5dffae9cde5babe61d0a1b962d5c903e7e60975bb00d0528efc57c21c3b35d6f9bfc18b84c1549221c18dbe8ca3ae75e8cde764a94d7b7
6
+ metadata.gz: 0b14e9eba2edf866eff03c2fa7333ec23733bb6622dd841fcc3da9ddf5cfe5e5b1104238a2857cadc7474bc49444773ee0b5ae7cabcbbaddbefd28e4da403bb8
7
+ data.tar.gz: 334f37197001b662b86c451c1dcb1faa059f94d1d4045ad43e66590f08663e7a90ea5979bbe247d3d2d7774a0f0e386d6f97c33663e6bef70c0dc6b909557ce3
data/CHANGELOG.md CHANGED
@@ -1,3 +1,9 @@
1
+ ## 0.7.0
2
+
3
+ - Add new development dependency `rackup` for logstash 8.x compatibility
4
+ - Enable `compression` configuration to compress payloads using `gzip`
5
+ - Document `proxy` configuration
6
+
1
7
  ## 0.6.0
2
8
 
3
9
  - Disable cookie processing by default
data/README.md CHANGED
@@ -15,6 +15,7 @@
15
15
  - [`ingest_endpoint_url`](#ingest_endpoint_url)
16
16
  - [`api_key`](#api_key)
17
17
  - [`ssl_verify_none`](#ssl_verify_none)
18
+ - [`proxy`](#proxy)
18
19
  - [`enable_metric`](#enable_metric)
19
20
  - [`id`](#id)
20
21
  - [Troubleshooting issues](#troubleshooting-issues)
@@ -62,6 +63,7 @@ The following configuration options are supported by the Dynatrace output plugin
62
63
  | [`ingest_endpoint_url`](#ingest_endpoint_url) | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) | Yes |
63
64
  | [`api_key`](#api_key) | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) | Yes |
64
65
  | [`ssl_verify_none`](#ssl_verify_none) | [Boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean) | No |
66
+ | [`http_compression`](#http_compression) | [Boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean) | No |
65
67
 
66
68
 
67
69
  ### Common Options
@@ -70,7 +72,8 @@ The following configuration options are supported by all output plugins:
70
72
 
71
73
  | Setting | Input type | Required |
72
74
  | --------------------------------- | ----------------------------------------------------------------------------------------------------- | -------- |
73
- | [`codec`](#codec) | [Codec](https://www.elastic.co/guide/en/logstash/7.16/configuration-file-structure.html#codec) | No |
75
+ | [`proxy`](#proxy) | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) or [Hash](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#hash) | No |
76
+ | `codec` | [Codec](https://www.elastic.co/guide/en/logstash/7.16/configuration-file-structure.html#codec) | No |
74
77
  | [`enable_metric`](#enable_metric) | [Boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean) | No |
75
78
  | [`id`](#id) | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) | No |
76
79
 
@@ -106,9 +109,26 @@ This option may be required if you are using a self-signed certificate, an expir
106
109
  > NOTE: Starting in plugin version `0.5.0`, this option has no effect in versions of Logstash older than `8.1.0`.
107
110
  > If this functionality is required, it is recommended to update Logstash or stay at plugin version `0.4.x` or older.
108
111
 
112
+ ### `proxy`
113
+
114
+ * Value type is [string](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) or [hash](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#hash)
115
+ * Optional
116
+ * No default value
117
+ * Introduced in `logstash-output-dynatrace` version `0.5.0` (not available in older versions)
118
+
119
+ This setting configures an HTTP proxy to route exported data through.
120
+
121
+ The supported configuration options and their syntax are the same as for the [proxy option](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html#plugins-outputs-http-proxy) in the HTTP output plugin:
122
+
123
+ 1. `proxy => http://example.org:1234`
124
+ 2. `proxy => { host => "example.org" port => 80 scheme => 'http' user => 'username@host' password => 'password' }`
125
+ 3. `proxy => { url => 'http://example.org:1234' user => 'username@host' password => 'password' }`
126
+
127
+ Note that [Hashes](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#hash) in the Logstash configuration file format use spaces as delimiters between entries, not commas.
128
+
109
129
  ### `enable_metric`
110
130
 
111
- * Value type is [boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean)
131
+ * Value type is [boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean)
112
132
  * Default value is true
113
133
 
114
134
  Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.
@@ -116,7 +136,7 @@ Disable or enable metric logging for this specific plugin instance. By default w
116
136
  ### `id`
117
137
 
118
138
  * Value type is string
119
- * There is no default value for this setting.
139
+ * No default value
120
140
 
121
141
  Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 dynatrace outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.
122
142
 
@@ -128,6 +148,15 @@ output {
128
148
  }
129
149
  ```
130
150
 
151
+ ### `http_compression`
152
+
153
+ * Value type is [boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean)
154
+ * Optional
155
+ * Default value is `false`
156
+
157
+ Setting `http_compression` to `true` causes the output plugin to compress all requests to the Dynatrace API using `gzip`.
158
+ This can result in a reduction in size and transmission time of network requests at the expense of some additional CPU and memory consumption.
159
+
131
160
  ## Troubleshooting issues
132
161
 
133
162
  When troubleshooting, always try to reduce the configuration as much as possible.
@@ -21,6 +21,7 @@ require 'logstash/version'
21
21
  require 'dynatrace/version'
22
22
  require 'uri'
23
23
  require 'logstash/plugin_mixins/http_client'
24
+ require 'zlib'
24
25
 
25
26
  # These constants came from the http plugin config but we don't want them configurable
26
27
  # If encountered as response codes this plugin will retry these requests
@@ -80,6 +81,9 @@ module LogStash
80
81
  # Disable cookie support. Overridden default value from LogStash::PluginMixins::HttpClient
81
82
  config :cookies, :validate => :boolean, :default => false
82
83
 
84
+ # Set this to true if you want to enable gzip compression for your http requests
85
+ config :http_compression, :validate => :boolean, :default => false
86
+
83
87
  def register
84
88
  # ssl_verification_mode config is from mixin but ssl_verify_none is our documented config
85
89
  @ssl_verification_mode = 'none' if @ssl_verify_none
@@ -253,6 +257,12 @@ module LogStash
253
257
  def send_event(event, attempt)
254
258
  headers = make_headers
255
259
 
260
+ # Compress the body and add appropriate header
261
+ if @http_compression == true
262
+ headers["Content-Encoding"] = "gzip"
263
+ event = gzip(event)
264
+ end
265
+
256
266
  # Create an async request
257
267
  response = client.post(ingest_endpoint_url, body: event, headers: headers)
258
268
 
@@ -331,6 +341,16 @@ module LogStash
331
341
  def log_warning(message, opts)
332
342
  @logger.warn(message, opts)
333
343
  end
344
+
345
+ # gzip data
346
+ def gzip(data)
347
+ gz = StringIO.new
348
+ gz.set_encoding("BINARY")
349
+ z = Zlib::GzipWriter.new(gz)
350
+ z.write(data)
351
+ z.close
352
+ gz.string
353
+ end
334
354
  end
335
355
  end
336
356
  end
@@ -52,4 +52,7 @@ Gem::Specification.new do |s|
52
52
 
53
53
  s.add_development_dependency 'rubocop', '1.9.1'
54
54
  s.add_development_dependency 'rubocop-rake', '0.5.1'
55
+
56
+ # Pin to avoid using new Fiber-based implementation that breaks tests here
57
+ s.add_development_dependency 'rackup', "< 2.1.0"
55
58
  end
@@ -347,7 +347,7 @@ describe LogStash::Outputs::Dynatrace do
347
347
  end
348
348
 
349
349
  let(:last_request) { TestApp.last_request }
350
- let(:body) { last_request.body.read }
350
+ let(:body) { read_last_request_body(last_request) }
351
351
  let(:content_type) { last_request.env['CONTENT_TYPE'] }
352
352
 
353
353
  it 'should receive the request' do
@@ -409,11 +409,11 @@ describe LogStash::Outputs::Dynatrace do
409
409
  include_examples('integration tests')
410
410
  end
411
411
 
412
- # describe "integration test with gzip compression" do
413
- # include_examples("integration tests") do
414
- # let(:base_config) { { "http_compression" => true } }
415
- # end
416
- # end
412
+ describe "integration test with gzip compression" do
413
+ include_examples("integration tests") do
414
+ let(:base_config) { { "http_compression" => true } }
415
+ end
416
+ end
417
417
 
418
418
  describe 'retryable error in termination' do
419
419
  let(:ingest_endpoint_url) { "http://localhost:#{port - 1}/invalid" }
@@ -437,10 +437,10 @@ describe LogStash::Outputs::Dynatrace do
437
437
  end
438
438
 
439
439
  RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starting web server with TLS
440
- @@default_server_settings = TestApp.server_settings.dup
440
+ let(:default_server_settings) { TestApp.server_settings.dup }
441
441
 
442
442
  before do
443
- TestApp.server_settings = @@default_server_settings.merge(webrick_config)
443
+ TestApp.server_settings = default_server_settings.merge(webrick_config)
444
444
 
445
445
  TestApp.last_request = nil
446
446
 
@@ -465,7 +465,7 @@ RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starti
465
465
  rescue StandardError
466
466
  nil
467
467
  end
468
- TestApp.server_settings = @@default_server_settings
468
+ TestApp.server_settings = default_server_settings
469
469
  end
470
470
 
471
471
  let(:ssl_cert_host) { 'localhost' }
@@ -483,7 +483,7 @@ RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starti
483
483
  after { subject.close }
484
484
 
485
485
  let(:last_request) { TestApp.last_request }
486
- let(:last_request_body) { last_request.body.read }
486
+ let(:last_request_body) { read_last_request_body(last_request) }
487
487
 
488
488
  let(:event) { LogStash::Event.new('message' => 'hello!') }
489
489
 
@@ -550,3 +550,10 @@ RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starti
550
550
  end
551
551
  end
552
552
  end
553
+
554
+ # Pre-emptively rewind the retrieval of `last_request.body` - for form based endpoints, the body
555
+ # is placed in params, and body is empty, requiring a `rewind` for the body to be available for comparison
556
+ def read_last_request_body(last_request)
557
+ last_request.body.rewind
558
+ last_request.body.read
559
+ end
data/spec/spec_helper.rb CHANGED
@@ -23,6 +23,8 @@ require 'webrick'
23
23
  require 'webrick/https'
24
24
  require 'openssl'
25
25
 
26
+ require "supports/compressed_requests"
27
+
26
28
  PORT = rand(65_535 - 1024) + 1025
27
29
 
28
30
  module LogStash
@@ -52,6 +54,9 @@ end
52
54
  # == Sinatra has ended his set (crowd applauds)
53
55
  #
54
56
  class TestApp < Sinatra::Base
57
+ # on the fly uncompress gzip content
58
+ use CompressedRequests
59
+
55
60
  set :environment, :production
56
61
  set :sessions, false
57
62
 
@@ -0,0 +1,42 @@
1
+ # encoding: utf-8
2
+ #
3
+ # based on relistan's rack handler
4
+ # out of the box rack only gives use the rack deflater handler to return compressed
5
+ # response, this gist offer the inverse and should work on all rack based app like sinatra or rails.
6
+ #
7
+ # original source: https://gist.github.com/relistan/2109707
8
+ require "zlib"
9
+
10
+ class CompressedRequests
11
+ def initialize(app)
12
+ @app = app
13
+ end
14
+
15
+ def method_handled?(env)
16
+ ['POST', 'PUT'].include? env['REQUEST_METHOD']
17
+ end
18
+
19
+ def encoding_handled?(env)
20
+ ['gzip', 'deflate'].include? env['HTTP_CONTENT_ENCODING']
21
+ end
22
+
23
+ def call(env)
24
+ if method_handled?(env) && encoding_handled?(env)
25
+ extracted = decode(env['rack.input'], env['HTTP_CONTENT_ENCODING'])
26
+
27
+ env.delete('HTTP_CONTENT_ENCODING')
28
+ env['CONTENT_LENGTH'] = extracted.bytesize
29
+ env['rack.input'] = StringIO.new(extracted)
30
+ end
31
+
32
+ status, headers, response = @app.call(env)
33
+ return [status, headers, response]
34
+ end
35
+
36
+ def decode(input, content_encoding)
37
+ case content_encoding
38
+ when 'gzip' then Zlib::GzipReader.new(input).read
39
+ when 'deflate' then Zlib::Inflate.inflate(input.read)
40
+ end
41
+ end
42
+ end
data/version.yaml CHANGED
@@ -1 +1 @@
1
- logstash-output-dynatrace: '0.6.0'
1
+ logstash-output-dynatrace: '0.7.0'
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-dynatrace
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.6.0
4
+ version: 0.7.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Dynatrace Open Source Engineering
8
- autorequire:
8
+ autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2024-02-27 00:00:00.000000000 Z
11
+ date: 2025-01-08 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: logstash-codec-json
@@ -148,6 +148,20 @@ dependencies:
148
148
  - - '='
149
149
  - !ruby/object:Gem::Version
150
150
  version: 0.5.1
151
+ - !ruby/object:Gem::Dependency
152
+ name: rackup
153
+ requirement: !ruby/object:Gem::Requirement
154
+ requirements:
155
+ - - "<"
156
+ - !ruby/object:Gem::Version
157
+ version: 2.1.0
158
+ type: :development
159
+ prerelease: false
160
+ version_requirements: !ruby/object:Gem::Requirement
161
+ requirements:
162
+ - - "<"
163
+ - !ruby/object:Gem::Version
164
+ version: 2.1.0
151
165
  description: |2
152
166
  This gem is a Logstash plugin required to be installed on top of the Logstash
153
167
  core pipeline using `$LS_HOME/bin/logstash-plugin install logstash-output-dynatrace`.
@@ -168,6 +182,7 @@ files:
168
182
  - logstash-output-dynatrace.gemspec
169
183
  - spec/outputs/dynatrace_spec.rb
170
184
  - spec/spec_helper.rb
185
+ - spec/supports/compressed_requests.rb
171
186
  - version.yaml
172
187
  homepage: https://github.com/dynatrace-oss/logstash-output-dynatrace
173
188
  licenses:
@@ -175,7 +190,7 @@ licenses:
175
190
  metadata:
176
191
  logstash_plugin: 'true'
177
192
  logstash_group: output
178
- post_install_message:
193
+ post_install_message:
179
194
  rdoc_options: []
180
195
  require_paths:
181
196
  - lib
@@ -191,10 +206,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
191
206
  version: '0'
192
207
  requirements: []
193
208
  rubygems_version: 3.1.6
194
- signing_key:
209
+ signing_key:
195
210
  specification_version: 4
196
211
  summary: A logstash output plugin for sending logs to the Dynatrace Generic log ingest
197
212
  API v2
198
213
  test_files:
199
214
  - spec/outputs/dynatrace_spec.rb
200
215
  - spec/spec_helper.rb
216
+ - spec/supports/compressed_requests.rb