fluent-plugin-datadog 0.14.4 → 0.15.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 926fa9b18f9ddafc71acdb33f6a35f9a8f5c8c349aa738c6772f37f4e678c27e
-  data.tar.gz: 334c768913c3bde058cba88d88bd10e343ff92d9d98be29b34efabfb94a3caf3
+  metadata.gz: a972214999befc7ccaf2a687d6a5f08007b381c3722998523e58d1058515a802
+  data.tar.gz: 1be2db9c9b926019eb917aa4973ffde6243fb2d013072b74965c2e4d32dd21ab
 SHA512:
-  metadata.gz: 8d9919de155b21d18ac85eac0479565fec145468fa169a41b56a12d974ce4326e689cd1794e32f24561639d30561d7c56ef45abd81559191c1d257ff16bab7f4
-  data.tar.gz: 6449bf1033f5a217acf0598426e27aa82dbe546977eab926540bc82ff773a74a726ead6eaa0804b753ec26d42177c4cd843224bf160beca148cc653238b820e0
+  metadata.gz: 71b1a131838a953f3f352ee5132a4443b1a7f21f56c031d3f8e13c9409874e34a1c3a79c96e0a964bcb2c7b205d1f02ad2f3a22208087bf39358d456d8f7b8ad
+  data.tar.gz: 7082da59a80ea386ae4e4152755fc48fcdb5931dc53f80d9c41eecab2d4aa7f3a0f0e12428403dff48d5b91358b7d947b04cbc764c136561cb6abb9285118b09

data/README.md

@@ -19,6 +19,7 @@ If you installed the td-agent instead
     /usr/sbin/td-agent-gem install fluent-plugin-datadog
 
 ## Usage
+
 ### Configure the output plugin
 
 To match events and send them to Datadog, simply add the following code to your configuration file.
@@ -74,10 +75,9 @@ Produces the following event:
 }
 ```
 
-### fluent-plugin-datadog properties
-Let's go deeper on the plugin configuration.
+### Advanced configuration
 
-As fluent-plugin-datadog is an output_buffer, you can set all output_buffer properties like it's describe in the [fluentd documentation](http://docs.fluentd.org/articles/output-plugin-overview#buffered-output-parameters "documentation").
+As `fluent-plugin-datadog` is a buffered output plugin, you can set all of the buffered output-specific properties as described in the [fluentd documentation](https://docs.fluentd.org/v/0.12/output#buffered-output-parameters).
 
 |  Property   |  Description                                                             |  Default value |
 |-------------|--------------------------------------------------------------------------|----------------|
@@ -100,8 +100,10 @@ As fluent-plugin-datadog is an output_buffer, you can set all output_buffer prop
 | **dd_hostname** | Used by Datadog to identify the host submitting the logs. | `hostname -f` |
 | **service** | Used by Datadog to correlate between logs, traces and metrics. | nil |
 | **port** | Proxy port when logs are not directly forwarded to Datadog and ssl is not used | 80 |
-| **host** | Proxy endpoint when logs are not directly forwarded to Datadog | http-intake.logs.datadoghq.com |
+| **site** | The Datadog [site](https://docs.datadoghq.com/getting_started/site/) to send logs to. Used to derive the default `host` when no explicit `host` is provided. Valid values: `datadoghq.com`, `datadoghq.eu`, `us3.datadoghq.com`, `us5.datadoghq.com`, `ap1.datadoghq.com`, `ddog-gov.com`. | datadoghq.com |
+| **host** | Proxy endpoint when logs are not directly forwarded to Datadog. When unset, the default is derived from `site` as `http-intake.logs.<site>`. An explicitly configured `host` always wins over `site`. | (derived from `site`) |
 | **http_proxy** | HTTP proxy, only takes effect if HTTP forwarding is enabled (`use_http`). Defaults to `HTTP_PROXY`/`http_proxy` env vars. | nil |
+| **delete_extracted_tag_attributes** | When true, removes `kubernetes` and `docker` attributes from log records after extracting them as tags. Useful to avoid duplicate data in Datadog logs UI. | false |
 
 ### Docker and Kubernetes tags
 
@@ -115,6 +117,8 @@ If your logs contain any of the following attributes, it will automatically be a
 * kubernetes.pod_name
 * docker.container_id
 
+**Note:** By default, these values will appear twice in the Datadog logs UI, once as tags (e.g., `container_name:myapp`) and once as attributes (e.g., `@kubernetes.container_name`). If you prefer to avoid this duplication, set `delete_extracted_tag_attributes` to `true` in your configuration. This will remove the `kubernetes` and `docker` attributes from the log record after the tags have been extracted.
+
 If the Datadog Agent collect them automatically, FluentD requires a plugin for this. We recommend using [fluent-plugin-kubernetes_metadata_filter](https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter) to collect Docker and Kubernetes metadata.
 
 Configuration example:
@@ -146,13 +150,29 @@ Configuration example:
 
 ## Build
 
-To build a new version of this plugin and push it to RubyGems:
+To build the gem locally:
 
-- Update the version in the .gemspec file accordingly
 - `rake build` to build the gem file
-- `rake release` to push the new gem to RubyGems
 
-**Note**: The latest command will fail without appropriate credentials configured. You can set those credentials by running the following command:
+## Releasing
+
+This gem is published to RubyGems via a GitHub Actions [Trusted Publishing](https://guides.rubygems.org/trusted-publishing/) workflow. No API keys or local credentials are needed.
+
+To release a new version:
+
+1. Update the version in `lib/fluent/plugin/version.rb`
+2. Update `CHANGELOG.md`
+3. Merge to `master`
+4. Go to **Actions** > **Publish gem** > **Run workflow**
+5. Run with `push` unchecked first (dry run) to verify the build
+6. Run again with `push` checked to publish to RubyGems
+7. The `rubygems.org` environment gate will ask for approval before publishing
+
+## Development Environment
+
+This repository includes the files to run it in a dev container in VS Code. To use it:
 
-`curl -u <USERNAME> https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials`, it will ask for your password.
+1. Install VS Code and Docker.
+2. Open the project in VS Code with the Dev Containers extension.
 
+VS Code will build and start the container automatically.

data/fluent-plugin-datadog.gemspec

@@ -28,9 +28,9 @@ Gem::Specification.new do |spec|
 
   spec.add_development_dependency "bundler", "~> 2.1"
   spec.add_development_dependency "test-unit", '~> 3.1'
-  spec.add_development_dependency "rake", "~> 12.0"
-  spec.add_development_dependency "yajl-ruby", "~> 1.2"
   spec.add_development_dependency 'webmock', "~> 3.6.0"
+  spec.add_development_dependency "ruby-debug-ide"
+  spec.add_development_dependency "rake", "~> 12.0"
 
   spec.metadata      = {
     'bug_tracker_uri'   => 'https://github.com/DataDog/fluent-plugin-datadog/issues',

data/lib/fluent/plugin/out_datadog.rb

@@ -5,7 +5,7 @@
 
 require "socket"
 require "openssl"
-require "yajl"
+require "json"
 require "zlib"
 require "fluent/plugin/output"
 
@@ -24,7 +24,9 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
   DD_MAX_BATCH_SIZE = 5000000
   DD_TRUNCATION_SUFFIX = "...TRUNCATED..."
 
-  DD_DEFAULT_HTTP_ENDPOINT = "http-intake.logs.datadoghq.com"
+  DD_DEFAULT_SITE = "datadoghq.com"
+  DD_DEFAULT_HTTP_HOST_PREFIX = "http-intake.logs."
+  DD_DEFAULT_HTTP_ENDPOINT = "#{DD_DEFAULT_HTTP_HOST_PREFIX}#{DD_DEFAULT_SITE}".freeze
   DD_DEFAULT_TCP_ENDPOINT = "intake.logs.datadoghq.com"
 
   helpers :compat_parameters
@@ -43,15 +45,25 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
   config_param :dd_source, :string, :default => nil
   config_param :dd_tags, :string, :default => nil
   config_param :dd_hostname, :string, :default => nil
+  config_param :delete_extracted_tag_attributes, :bool, :default => false
+
+  # Datadog site used to derive the default intake host. Valid values include:
+  # "datadoghq.com" (default), "datadoghq.eu", "us3.datadoghq.com",
+  # "us5.datadoghq.com", "ap1.datadoghq.com", "ddog-gov.com". Any value
+  # explicitly set for `host` takes precedence over the site-derived default.
+  config_param :site, :string, :default => DD_DEFAULT_SITE
 
   # Connection settings
-  config_param :host, :string, :default => DD_DEFAULT_HTTP_ENDPOINT
+  # `host` defaults to nil so we can tell whether the user explicitly set it.
+  # When nil, the host is derived from `site` during `configure`.
+  config_param :host, :string, :default => nil
   config_param :use_ssl, :bool, :default => true
   config_param :port, :integer, :default => 80
   config_param :ssl_port, :integer, :default => 443
   config_param :max_retries, :integer, :default => -1
   config_param :max_backoff, :integer, :default => 30
   config_param :use_http, :bool, :default => true
+  config_param :custom_headers, :hash, :default => {}
   config_param :use_compression, :bool, :default => true
   config_param :compression_level, :integer, :default => 6
   config_param :no_ssl_validation, :bool, :default => false
@@ -75,12 +87,29 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
   def configure(conf)
     compat_parameters_convert(conf, :buffer)
     super
-    return if @dd_hostname
 
-    if not @use_http and @host == DD_DEFAULT_HTTP_ENDPOINT
-      @host = DD_DEFAULT_TCP_ENDPOINT
+    # Derive default host from `site` only for HTTP transport.
+    # TCP users with a non-default site must set `host` explicitly;
+    # TCP users with no host and the default site get the legacy TCP endpoint.
+    if @host.nil? || @host.empty?
+      if @use_http
+        @host = "#{DD_DEFAULT_HTTP_HOST_PREFIX}#{@site}"
+      elsif @site == DD_DEFAULT_SITE
+        @host = DD_DEFAULT_TCP_ENDPOINT
+      else
+        # TCP + non-default site: we cannot safely derive a TCP intake hostname
+        # from `site` (the HTTP-intake prefix is HTTP-only), and leaving @host
+        # nil would surface as a cryptic connect-time error. Fail fast at
+        # configure time with an actionable message.
+        raise Fluent::ConfigError,
+          "`host` is required when `use_http false` is combined with a non-default `site` " \
+          "(`site #{@site.inspect}`). Set `host` explicitly to your TCP intake (e.g. " \
+          "`intake.logs.#{@site}`)."
+      end
     end
 
+    return if @dd_hostname
+
     # Set dd_hostname if not already set (can be set when using fluentd as aggregator)
     @dd_hostname = %x[hostname -f 2> /dev/null].strip
     @dd_hostname = Socket.gethostname if @dd_hostname.empty?
@@ -98,7 +127,7 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
 
   def start
     super
-    @client = new_client(log, @api_key, @use_http, @use_ssl, @no_ssl_validation, @host, @ssl_port, @port, @http_proxy, @use_compression, @force_v1_routes)
+    @client = new_client(log, @api_key, @use_http, @use_ssl, @no_ssl_validation, @host, @ssl_port, @port, @http_proxy, @custom_headers, @use_compression, @force_v1_routes)
   end
 
   def shutdown
@@ -117,10 +146,10 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
     # is compatible with Time.at below.
     record = enrich_record(tag, time.to_f, record)
     if @use_http
-      record = Yajl.dump(record)
+      record = JSON.generate(record)
     else
       if @use_json
-        record = "#{api_key} #{Yajl.dump(record)}"
+        record = "#{api_key} #{JSON.generate(record)}"
       else
         record = "#{api_key} #{record}"
       end
@@ -146,8 +175,9 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
           process_tcp_event(record[0], @max_retries, @max_backoff, DD_MAX_BATCH_SIZE)
         end
       end
-    rescue Exception => e
-      log.error("Uncaught processing exception in datadog forwarder #{e.message}")
+    rescue StandardError => e
+      log.error("Processing exception in datadog forwarder #{e.class}: #{e.message}")
+      raise
     end
   end
 
@@ -253,6 +283,12 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
         record["ddtags"] = record["ddtags"] + "," + container_tags
       end
     end
+
+    if @delete_extracted_tag_attributes
+      record.delete('kubernetes')
+      record.delete('docker')
+    end
+
     record
   end
 
@@ -270,9 +306,9 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
   end
 
   # Build a new transport client
-  def new_client(logger, api_key, use_http, use_ssl, no_ssl_validation, host, ssl_port, port, http_proxy, use_compression, force_v1_routes)
+  def new_client(logger, api_key, use_http, use_ssl, no_ssl_validation, host, ssl_port, port, http_proxy, custom_headers, use_compression, force_v1_routes)
     if use_http
-      DatadogHTTPClient.new logger, use_ssl, no_ssl_validation, host, ssl_port, port, http_proxy, use_compression, api_key, force_v1_routes
+      DatadogHTTPClient.new logger, use_ssl, no_ssl_validation, host, ssl_port, port, http_proxy, custom_headers, use_compression, api_key, force_v1_routes
     else
       DatadogTCPClient.new logger, use_ssl, no_ssl_validation, host, ssl_port, port
     end
@@ -293,6 +329,10 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
           retries += 1
           retry
         end
+        # Bounded retries exhausted: re-raise so the caller (and, ultimately,
+        # Fluentd core's buffer retry) is signalled instead of the failure being
+        # swallowed and the chunk silently dropped.
+        raise
       end
     end
 
@@ -310,7 +350,29 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
     require 'net/http'
     require 'net/http/persistent'
 
-    def initialize(logger, use_ssl, no_ssl_validation, host, ssl_port, port, http_proxy, use_compression, api_key, force_v1_routes = false)
+    # Transient network exceptions that warrant a retry. This mirrors the set
+    # Ruby's Net::HTTP retries for idempotent requests (see
+    # Net::HTTP#max_retries=), which notably does NOT include POST, plus the
+    # connection-establishment errors net-http-persistent wraps in its own
+    # Error. Because our log POSTs are non-idempotent, Net::HTTP will not retry
+    # them for us, so we classify these ourselves and route them through
+    # send_retries (and, on exhaustion, up to Fluentd core).
+    RETRYABLE_NETWORK_EXCEPTIONS = [
+      Net::OpenTimeout,
+      Net::ReadTimeout,
+      EOFError,
+      IOError,
+      SocketError,
+      Errno::ECONNRESET,
+      Errno::ECONNREFUSED,
+      Errno::ECONNABORTED,
+      Errno::EPIPE,
+      Errno::ETIMEDOUT,
+      OpenSSL::SSL::SSLError,
+      Net::HTTP::Persistent::Error,
+    ].freeze
+
+    def initialize(logger, use_ssl, no_ssl_validation, host, ssl_port, port, http_proxy, custom_headers, use_compression, api_key, force_v1_routes = false)
       @logger = logger
       protocol = use_ssl ? "https" : "http"
       port = use_ssl ? ssl_port : port
@@ -328,6 +390,9 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
       logger.info("Starting HTTP connection to #{protocol}://#{host}:#{port.to_s} with compression " + (use_compression ? "enabled" : "disabled") + (force_v1_routes ? " using v1 routes" : " using v2 routes"))
       @client = Net::HTTP::Persistent.new name: "fluent-plugin-datadog-logcollector", proxy: proxy_uri
       @client.verify_mode = OpenSSL::SSL::VERIFY_NONE if no_ssl_validation
+      custom_headers.each do |key, value|
+        @client.override_headers[key] = value
+      end
       unless force_v1_routes
         @client.override_headers["DD-API-KEY"] = api_key
         @client.override_headers["DD-EVP-ORIGIN"] = "fluent"
@@ -346,7 +411,13 @@ class Fluent::DatadogOutput < Fluent::Plugin::Output
     def send(payload)
       request = Net::HTTP::Post.new @uri.request_uri
       request.body = payload
-      response = @client.request @uri, request
+      begin
+        response = @client.request @uri, request
+      rescue *RETRYABLE_NETWORK_EXCEPTIONS => e
+        # Transient network failure before we ever saw a response. Net::HTTP
+        # won't retry a POST for us, so surface it as retryable.
+        raise RetryableError.new "Unable to send payload, transient network error: #{e.class}: #{e.message}"
+      end
       res_code = response.code.to_i
       # on a backend error or on an http 429, retry with backoff
       if res_code >= 500 || res_code == 429

data/lib/fluent/plugin/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module DatadogFluentPlugin
-  VERSION = '0.14.4'
+  VERSION = '0.15.1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-datadog
 version: !ruby/object:Gem::Version
-  version: 0.14.4
+  version: 0.15.1
 platform: ruby
 authors:
 - Datadog Solutions Team
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-07-15 00:00:00.000000000 Z
+date: 2026-09-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -73,47 +73,47 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '3.1'
 - !ruby/object:Gem::Dependency
-  name: rake
+  name: webmock
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '12.0'
+        version: 3.6.0
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '12.0'
+        version: 3.6.0
 - !ruby/object:Gem::Dependency
-  name: yajl-ruby
+  name: ruby-debug-ide
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.2'
+        version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
-        version: '1.2'
+        version: '0'
 - !ruby/object:Gem::Dependency
-  name: webmock
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 3.6.0
+        version: '12.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 3.6.0
+        version: '12.0'
 description:
 email:
 - support@datadoghq.com
@@ -152,7 +152,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.3.15
+rubygems_version: 3.5.22
 signing_key:
 specification_version: 4
 summary: Datadog output plugin for Fluent event collector