logstash-output-dynatrace 0.5.1 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 69fbd3982ad32849fe3d185e3edd44f9c3d071cb3f74c31d2f6d32a1e7841df5
-  data.tar.gz: 24bfb2c6599648c80295860e90eb8e6600d75ad4af8ccd5252f42b46b3beb0fc
+  metadata.gz: 4db961537a03cb5b21240ab63ea56877ceebcf0a075c5db85feb736b787d05df
+  data.tar.gz: 185005382c4ac576fde60eeffa8d3f1a5e037445b3dd92b2126be22f0a3481dc
 SHA512:
-  metadata.gz: ec160bc4be7fcdeed4704cba0c084d74cfb812de161a02f72ce4178dd25232505951d9237b23f433f0b195213ad49d6d36e9ac757f7970c0014b8920728536c0
-  data.tar.gz: 3ae1b197743a9b72e2e6530a78187f2d0acf822e4baf3835625ad3d348d62593cd7e9e7100b15475a4ccfb535409d835a652a92d534553abc9685dc391055665
+  metadata.gz: 0b14e9eba2edf866eff03c2fa7333ec23733bb6622dd841fcc3da9ddf5cfe5e5b1104238a2857cadc7474bc49444773ee0b5ae7cabcbbaddbefd28e4da403bb8
+  data.tar.gz: 334f37197001b662b86c451c1dcb1faa059f94d1d4045ad43e66590f08663e7a90ea5979bbe247d3d2d7774a0f0e386d6f97c33663e6bef70c0dc6b909557ce3

data/CHANGELOG.md

@@ -1,3 +1,21 @@
+## 0.7.0
+
+- Add new development dependency `rackup` for logstash 8.x compatibility
+- Enable `compression` configuration to compress payloads using `gzip`
+- Document `proxy` configuration
+
+## 0.6.0
+
+- Disable cookie processing by default
+
+## 0.5.1
+
+- Split large batches into smaller requests in order to meet Dynatrace API payload size limitations
+
+## 0.5.0
+
+- Rewrite plugin using http client mixin
+
 ## 0.4.0
 
 - Add` OpenSSL::SSL::SSLError` to the list of retried exception

data/README.md

@@ -1,3 +1,4 @@
+<!-- omit in toc -->
 # Logstash Dynatrace output plugin
 
 [![Travis Build Status](https://api.travis-ci.com/dynatrace-oss/logstash-output-dynatrace.svg?branch=main)](https://app.travis-ci.com/dynatrace-oss/logstash-output-dynatrace)
@@ -14,13 +15,13 @@
   - [`ingest_endpoint_url`](#ingest_endpoint_url)
   - [`api_key`](#api_key)
   - [`ssl_verify_none`](#ssl_verify_none)
-  - [`codec`](#codec)
+  - [`proxy`](#proxy)
   - [`enable_metric`](#enable_metric)
   - [`id`](#id)
 - [Troubleshooting issues](#troubleshooting-issues)
   - [Enable Debug Logs](#enable-debug-logs)
 
-A [Logstash](https://github.com/elastic/logstash) output plugin for sending logs to the Dynatrace [Generic log ingest API v2](https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring/log-monitoring-v2/post-log-ingest/).
+A [Logstash](https://github.com/elastic/logstash) output plugin for sending logs to the Dynatrace [Generic log ingest API v2](https://docs.dynatrace.com/docs/shortlink/api-log-monitoring-v2-post-ingest).
 Please review the documentation for this API before using the plugin.
 
 ## Installation Prerequisites
@@ -62,6 +63,7 @@ The following configuration options are supported by the Dynatrace output plugin
 | [`ingest_endpoint_url`](#ingest_endpoint_url) | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string)   | Yes      |
 | [`api_key`](#api_key)                         | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string)   | Yes      |
 | [`ssl_verify_none`](#ssl_verify_none)         | [Boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean) | No       |
+| [`http_compression`](#http_compression)         | [Boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean) | No       |
 
 
 ### Common Options
@@ -70,7 +72,8 @@ The following configuration options are supported by all output plugins:
 
 | Setting                           | Input type                                                                                            | Required |
 | --------------------------------- | ----------------------------------------------------------------------------------------------------- | -------- |
-| [`codec`](#codec)                 | [Codec](https://www.elastic.co/guide/en/logstash/7.16/configuration-file-structure.html#codec)        | No       |
+| [`proxy`](#proxy)                 | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) or [Hash](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#hash) | No |
+|  `codec`                          | [Codec](https://www.elastic.co/guide/en/logstash/7.16/configuration-file-structure.html#codec)        | No       |
 | [`enable_metric`](#enable_metric) | [Boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean) | No       |
 | [`id`](#id)                       | [String](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string)   | No       |
 
@@ -81,7 +84,7 @@ The following configuration options are supported by all output plugins:
 * Value type is [string](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string)
 * Required
 
-This is the full URL of the [Generic log ingest API v2](https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring/log-monitoring-v2/post-log-ingest/) endpoint on your ActiveGate.
+This is the full URL of the [Generic log ingest API v2](https://docs.dynatrace.com/docs/shortlink/api-log-monitoring-v2-post-ingest/) endpoint on your ActiveGate.
 Example: `"ingest_endpoint_url" => "https://abc123456.live.dynatrace.com/api/v2/logs/ingest"`
 
 ### `api_key`
@@ -89,7 +92,7 @@ Example: `"ingest_endpoint_url" => "https://abc123456.live.dynatrace.com/api/v2/
 * Value type is [string](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string)
 * Required
 
-This is the [Dynatrace API token](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication/) which will be used to authenticate log ingest requests.
+This is the [Dynatrace API token](https://docs.dynatrace.com/docs/shortlink/api-authentication) which will be used to authenticate log ingest requests.
 It requires the `logs.ingest` (Ingest Logs) scope to be set and it is recommended to limit scope to only this one.
 Example: `"api_key" => "dt0c01.4XLO3..."`
 
@@ -106,9 +109,26 @@ This option may be required if you are using a self-signed certificate, an expir
 > NOTE: Starting in plugin version `0.5.0`, this option has no effect in versions of Logstash older than `8.1.0`.
 > If this functionality is required, it is recommended to update Logstash or stay at plugin version `0.4.x` or older.
 
+### `proxy`
+
+* Value type is [string](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#string) or [hash](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#hash)
+* Optional
+* No default value
+* Introduced in `logstash-output-dynatrace` version `0.5.0` (not available in older versions)
+
+This setting configures an HTTP proxy to route exported data through.
+
+The supported configuration options and their syntax are the same as for the [proxy option](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-http.html#plugins-outputs-http-proxy) in the HTTP output plugin:
+
+1. `proxy => http://example.org:1234`
+2. `proxy => { host => "example.org" port => 80 scheme => 'http' user => 'username@host' password => 'password' }`
+3. `proxy => { url =>  'http://example.org:1234' user => 'username@host' password => 'password' }`
+
+Note that [Hashes](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#hash) in the Logstash configuration file format use spaces as delimiters between entries, not commas.
+
 ### `enable_metric`
 
- * Value type is [boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean)
+* Value type is [boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean)
 * Default value is true
 
 Disable or enable metric logging for this specific plugin instance. By default we record all the metrics we can, but you can disable metrics collection for a specific plugin.
@@ -116,7 +136,7 @@ Disable or enable metric logging for this specific plugin instance. By default w
 ### `id`
 
 * Value type is string
-* There is no default value for this setting.
+* No default value
 
 Add a unique ID to the plugin configuration. If no ID is specified, Logstash will generate one. It is strongly recommended to set this ID in your configuration. This is particularly useful when you have two or more plugins of the same type. For example, if you have 2 dynatrace outputs. Adding a named ID in this case will help in monitoring Logstash when using the monitoring APIs.
 
@@ -128,6 +148,15 @@ output {
 }
 ```
 
+### `http_compression`
+
+* Value type is [boolean](https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html#boolean)
+* Optional
+* Default value is `false`
+
+Setting `http_compression` to `true` causes the output plugin to compress all requests to the Dynatrace API using `gzip`.
+This can result in a reduction in size and transmission time of network requests at the expense of some additional CPU and memory consumption.
+
 ## Troubleshooting issues
 
 When troubleshooting, always try to reduce the configuration as much as possible.

data/lib/logstash/outputs/dynatrace.rb

@@ -21,6 +21,7 @@ require 'logstash/version'
 require 'dynatrace/version'
 require 'uri'
 require 'logstash/plugin_mixins/http_client'
+require 'zlib'
 
 # These constants came from the http plugin config but we don't want them configurable
 # If encountered as response codes this plugin will retry these requests
@@ -77,6 +78,12 @@ module LogStash
       # Maximum size payload to send to the Dynatrace API in Bytes. Batches of events which would be larger than max_payload_size when serialized will be split into smaller batches of events.
       config :max_payload_size, validate: :number, default: 4_500_000
 
+      # Disable cookie support. Overridden default value from LogStash::PluginMixins::HttpClient
+      config :cookies, :validate => :boolean, :default => false
+
+      # Set this to true if you want to enable gzip compression for your http requests
+      config :http_compression, :validate => :boolean, :default => false
+
       def register
         # ssl_verification_mode config is from mixin but ssl_verify_none is our documented config
         @ssl_verification_mode = 'none' if @ssl_verify_none
@@ -250,6 +257,12 @@ module LogStash
       def send_event(event, attempt)
         headers = make_headers
 
+        # Compress the body and add appropriate header
+        if @http_compression == true
+          headers["Content-Encoding"] = "gzip"
+          event = gzip(event)
+        end
+
         # Create an async request
         response = client.post(ingest_endpoint_url, body: event, headers: headers)
 
@@ -328,6 +341,16 @@ module LogStash
       def log_warning(message, opts)
         @logger.warn(message, opts)
       end
+
+      # gzip data
+      def gzip(data)
+        gz = StringIO.new
+        gz.set_encoding("BINARY")
+        z = Zlib::GzipWriter.new(gz)
+        z.write(data)
+        z.close
+        gz.string
+      end
     end
   end
 end

data/logstash-output-dynatrace.gemspec

@@ -52,4 +52,7 @@ Gem::Specification.new do |s|
 
   s.add_development_dependency 'rubocop', '1.9.1'
   s.add_development_dependency 'rubocop-rake', '0.5.1'
+
+  # Pin to avoid using new Fiber-based implementation that breaks tests here
+  s.add_development_dependency 'rackup', "< 2.1.0"
 end

data/spec/outputs/dynatrace_spec.rb

@@ -347,7 +347,7 @@ describe LogStash::Outputs::Dynatrace do
       end
 
       let(:last_request) { TestApp.last_request }
-      let(:body) { last_request.body.read }
+      let(:body) { read_last_request_body(last_request) }
       let(:content_type) { last_request.env['CONTENT_TYPE'] }
 
       it 'should receive the request' do
@@ -409,11 +409,11 @@ describe LogStash::Outputs::Dynatrace do
     include_examples('integration tests')
   end
 
-  # describe "integration test with gzip compression" do
-  #   include_examples("integration tests") do
-  #     let(:base_config) { { "http_compression" => true } }
-  #   end
-  # end
+  describe "integration test with gzip compression" do
+    include_examples("integration tests") do
+      let(:base_config) { { "http_compression" => true } }
+    end
+  end
 
   describe 'retryable error in termination' do
     let(:ingest_endpoint_url) { "http://localhost:#{port - 1}/invalid" }
@@ -437,10 +437,10 @@ describe LogStash::Outputs::Dynatrace do
 end
 
 RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starting web server with TLS
-  @@default_server_settings = TestApp.server_settings.dup
+  let(:default_server_settings) { TestApp.server_settings.dup }
 
   before do
-    TestApp.server_settings = @@default_server_settings.merge(webrick_config)
+    TestApp.server_settings = default_server_settings.merge(webrick_config)
 
     TestApp.last_request = nil
 
@@ -465,7 +465,7 @@ RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starti
     rescue StandardError
       nil
     end
-    TestApp.server_settings = @@default_server_settings
+    TestApp.server_settings = default_server_settings
   end
 
   let(:ssl_cert_host) { 'localhost' }
@@ -483,7 +483,7 @@ RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starti
   after  { subject.close }
 
   let(:last_request) { TestApp.last_request }
-  let(:last_request_body) { last_request.body.read }
+  let(:last_request_body) { read_last_request_body(last_request) }
 
   let(:event) { LogStash::Event.new('message' => 'hello!') }
 
@@ -550,3 +550,10 @@ RSpec.describe LogStash::Outputs::Dynatrace do # different block as we're starti
     end
   end
 end
+
+# Pre-emptively rewind the retrieval of `last_request.body` - for form based endpoints, the body
+# is placed in params, and body is empty, requiring a `rewind` for the body to be available for comparison
+def read_last_request_body(last_request)
+  last_request.body.rewind
+  last_request.body.read
+end

data/spec/spec_helper.rb

@@ -23,6 +23,8 @@ require 'webrick'
 require 'webrick/https'
 require 'openssl'
 
+require "supports/compressed_requests"
+
 PORT = rand(65_535 - 1024) + 1025
 
 module LogStash
@@ -52,6 +54,9 @@ end
 # == Sinatra has ended his set (crowd applauds)
 #
 class TestApp < Sinatra::Base
+  # on the fly uncompress gzip content
+  use CompressedRequests
+
   set :environment, :production
   set :sessions, false
 

data/spec/supports/compressed_requests.rb

@@ -0,0 +1,42 @@
+# encoding: utf-8
+#
+# based on relistan's rack handler
+# out of the box rack only gives use the rack deflater handler to return compressed
+# response, this gist offer the inverse and should work on all rack based app like sinatra or rails.
+#
+# original source: https://gist.github.com/relistan/2109707
+require "zlib"
+
+class CompressedRequests
+  def initialize(app)
+    @app = app
+  end
+
+  def method_handled?(env)
+    ['POST', 'PUT'].include? env['REQUEST_METHOD']
+  end
+
+  def encoding_handled?(env)
+    ['gzip', 'deflate'].include? env['HTTP_CONTENT_ENCODING']
+  end
+
+  def call(env)
+    if method_handled?(env) && encoding_handled?(env)
+      extracted = decode(env['rack.input'], env['HTTP_CONTENT_ENCODING'])
+
+      env.delete('HTTP_CONTENT_ENCODING')
+      env['CONTENT_LENGTH'] = extracted.bytesize
+      env['rack.input'] = StringIO.new(extracted)
+    end
+
+    status, headers, response = @app.call(env)
+    return [status, headers, response]
+  end
+
+  def decode(input, content_encoding)
+    case content_encoding
+      when 'gzip' then Zlib::GzipReader.new(input).read
+      when 'deflate' then Zlib::Inflate.inflate(input.read)
+    end
+  end
+end

data/version.yaml

@@ -1 +1 @@
-logstash-output-dynatrace: '0.5.1'
+logstash-output-dynatrace: '0.7.0'

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-dynatrace
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.7.0
 platform: ruby
 authors:
 - Dynatrace Open Source Engineering
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-09-26 00:00:00.000000000 Z
+date: 2025-01-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-codec-json
@@ -148,6 +148,20 @@ dependencies:
     - - '='
       - !ruby/object:Gem::Version
         version: 0.5.1
+- !ruby/object:Gem::Dependency
+  name: rackup
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 2.1.0
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 2.1.0
 description: |2
       This gem is a Logstash plugin required to be installed on top of the Logstash
       core pipeline using `$LS_HOME/bin/logstash-plugin install logstash-output-dynatrace`.
@@ -168,6 +182,7 @@ files:
 - logstash-output-dynatrace.gemspec
 - spec/outputs/dynatrace_spec.rb
 - spec/spec_helper.rb
+- spec/supports/compressed_requests.rb
 - version.yaml
 homepage: https://github.com/dynatrace-oss/logstash-output-dynatrace
 licenses:
@@ -175,7 +190,7 @@ licenses:
 metadata:
   logstash_plugin: 'true'
   logstash_group: output
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -191,10 +206,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.1.6
-signing_key: 
+signing_key:
 specification_version: 4
 summary: A logstash output plugin for sending logs to the Dynatrace Generic log ingest
   API v2
 test_files:
 - spec/outputs/dynatrace_spec.rb
 - spec/spec_helper.rb
+- spec/supports/compressed_requests.rb