fluent-plugin-opentelemetry 0.5.3 → 0.5.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2e1331e92688a03b2da70e3712257f4dfd4ae1cec7c5da354084f6fc883ae5cf
-  data.tar.gz: 403339bc82d1c09823afc7a5c2b16199cafab92300a0e56eb446fb33e55e1407
+  metadata.gz: dad225b751029746a86f4bdeae0b9f415d0756c25d9c560e7606ac11c13ade0b
+  data.tar.gz: '091d4d6113b42822cc47e6cd386aa10e2044f9a146d401e437eefe21b11df10e'
 SHA512:
-  metadata.gz: d3cc4c75562d9698c668b02e172d8eeb2c15de4233cb1ebe55607e6daeb18efe19df1bd20f2b0c586f79e07d63301bcf880221d904ebcf11c34047b7a00fd12d
-  data.tar.gz: 83d8d1e66e74c93a23f46d7580591b72d1bd3dfd064365ce262057890cb8aff0377cae4cf7c88ad810ed94842a5eb7c91e526109efd50fcff7d131c6afb4f3a7
+  metadata.gz: d5919a65f7d4adf3491c16cc3a098283bb69f7fd84a09e88730d15df7919c878838eee465d3315f5f762acd323f2a0983c2445f74b36384526ea1a1dc2e04f51
+  data.tar.gz: fe3517def463d5c8ae2c559df0e42438d518ae4d74f0ccfc6e900c570ca74b1cca2d1b31a97da33e79d566ba304884c5646f13a46869d3ea8147ba209143d1c9

data/CHANGELOG.md

@@ -1,5 +1,13 @@
 ## [Unreleased]
 
+## [0.5.4] - 2026-09-04
+
+Fixes:
+
+- Apply <transport tls> ca_path to HTTP connections (#55)
+- Apply <transport tls> settings to gRPC connections (#54)
+- out_opentelemetry: skip invalid records instead of failing the chunk (#53)
+
 ## [0.5.3] - 2026-06-25
 
 Fixes:

data/README.md

@@ -1,5 +1,8 @@
 # fluent-plugin-opentelemetry
 
+[![Gem Version](https://badge.fury.io/rb/fluent-plugin-opentelemetry.svg)](https://badge.fury.io/rb/fluent-plugin-opentelemetry)
+[![Test](https://github.com/fluent-plugins-nursery/fluent-plugin-opentelemetry/actions/workflows/test.yml/badge.svg)](https://github.com/fluent-plugins-nursery/fluent-plugin-opentelemetry/actions/workflows/test.yml)
+
 [Fluentd](https://fluentd.org/) input/output plugin to forward [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-proto) data.
 
 ## Installation
@@ -89,6 +92,11 @@ It needs to install `grpc` gem manually to use this feature.
 
 Refer [Config: Transport Section](https://docs.fluentd.org/configuration/transport-section)
 
+When the `<grpc>` section is configured, `<transport tls>` applies to the gRPC listener as well. gRPC uses `ca_path`, `cert_path`, `private_key_path`, `private_key_passphrase` and `client_cert_auth`, and the other TLS parameters have no effect on the gRPC listener. `client_cert_auth` requires `ca_path`, because gRPC has no default trust store for client certificates.
+
+> [!WARNING]
+> This changes the behavior of an existing configuration that has both `<transport tls>` and `<grpc>`. The gRPC listener used to accept plaintext regardless of `<transport tls>`, and it now requires TLS. Plaintext OTLP/gRPC clients have to be switched to TLS.
+
 #### Example
 
 ```
@@ -204,9 +212,11 @@ It needs to install `grpc` gem manually to use this feature.
 
 | parameter              | type    | description                                    | default |
 |------------------------|---------|------------------------------------------------|---------|
+| ca_path                | string  | Specifies the path of CA Certificate file used to verify the server | `nil`   |
 | cert_path              | string  | Specifies the path of Certificate file         | `nil`   |
 | private_key_path       | string  | Specifies the path of Private Key file         | `nil`   |
 | private_key_passphrase | string  | Specifies the public CA private key passphrase | `nil`   |
+| insecure               | bool    | Skips the verification of the server certificate | `false` |
 
 | parameter   | type | description                                                 | available values               | default |
 |-------------|------|-------------------------------------------------------------|--------------------------------|---------|
@@ -215,6 +225,18 @@ It needs to install `grpc` gem manually to use this feature.
 
 Refer [Config: Transport Section](https://docs.fluentd.org/configuration/transport-section)
 
+The HTTP connection uses `ca_path`, `cert_path`, `private_key_path`, `private_key_passphrase`, `min_version`, `max_version` and `insecure`, and the other TLS parameters have no effect on it. `ca_path` is added to the default trusted CAs of the system instead of replacing them.
+
+Fluentd rejects a `<transport tls>` section that configures no certificate at all, so pair `ca_path` with `insecure true` to verify the server without presenting a client certificate. On the HTTP connection, `insecure true` skips the verification only when `ca_path` is omitted.
+
+> [!WARNING]
+> This changes the behavior of an existing configuration that has both `ca_path` and `insecure true`. The HTTP connection used to skip the verification, and it now verifies the server certificate and its hostname. Drop `ca_path` to keep the previous behavior.
+
+When the `<grpc>` section is configured, `<transport tls>` applies to the gRPC connection as well. gRPC uses `ca_path`, `cert_path`, `private_key_path` and `private_key_passphrase`, and the other TLS parameters have no effect on the gRPC connection. `insecure` does not weaken the gRPC connection: the server certificate is verified against `ca_path`, or against the default root certificates of gRPC when `ca_path` is omitted.
+
+> [!WARNING]
+> This changes the behavior of an existing configuration that has both `<transport tls>` and `<grpc>`. The gRPC connection used to be plaintext regardless of `<transport tls>`, and it now uses TLS, so the collector has to accept it.
+
 #### `<buffer>` section
 
 | parameter        | type    | description                                                     | default    |

data/lib/fluent/plugin/in_opentelemetry.rb

@@ -2,6 +2,7 @@
 
 require "fluent/plugin/input"
 require "fluent/plugin/opentelemetry/constant"
+require "fluent/plugin/opentelemetry/grpc_tls"
 require "fluent/plugin/opentelemetry/http_input_handler"
 require "fluent/plugin_helper/http_server"
 require "fluent/plugin_helper/thread"
@@ -57,6 +58,8 @@ module Fluent::Plugin
       unless [@http_config, @grpc_config].any?
         raise Fluent::ConfigError, "Please configure either <http> or <grpc> section, or both."
       end
+
+      Opentelemetry::GrpcTLS.validate_server!(@transport_config) if @grpc_config
     end
 
     def start
@@ -78,8 +81,8 @@ module Fluent::Plugin
       end
 
       if @grpc_config
+        @grpc_handler = Opentelemetry::GrpcInputHandler.new(@grpc_config, @transport_config, log)
         thread_create(:in_opentelemetry_grpc_server) do
-          @grpc_handler = Opentelemetry::GrpcInputHandler.new(@grpc_config, log)
           @grpc_handler.run(
             logs: lambda { |record|
               router.emit(tag_for(Opentelemetry::RECORD_TYPE_LOGS), Fluent::EventTime.now, { "type" => Opentelemetry::RECORD_TYPE_LOGS, "message" => record })

data/lib/fluent/plugin/opentelemetry/grpc_input_handler.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "fluent/plugin/opentelemetry/grpc_tls"
 require "fluent/plugin/opentelemetry/response"
 require "opentelemetry/proto/collector/logs/v1/logs_service_services_pb"
 require "opentelemetry/proto/collector/metrics/v1/metrics_service_services_pb"
@@ -49,14 +50,15 @@ module Fluent::Plugin::Opentelemetry
       end
     end
 
-    def initialize(grpc_config, logger)
+    def initialize(grpc_config, transport_config, logger)
       @grpc_config = grpc_config
       @logger = logger
+      @credentials = Fluent::Plugin::Opentelemetry::GrpcTLS.server_credentials(transport_config)
     end
 
     def run(logs:, metrics:, traces:)
       @server = GRPC::RpcServer.new(interceptors: [ExceptionInterceptor.new])
-      @server.add_http2_port("#{@grpc_config.bind}:#{@grpc_config.port}", :this_port_is_insecure)
+      @server.add_http2_port("#{@grpc_config.bind}:#{@grpc_config.port}", @credentials)
 
       logs_handler = ServiceHandler::Logs.new
       logs_handler.callback = lambda { |request|

data/lib/fluent/plugin/opentelemetry/grpc_output_handler.rb

@@ -8,6 +8,7 @@ require "opentelemetry/proto/collector/trace/v1/trace_service_pb"
 require "opentelemetry/proto/collector/trace/v1/trace_service_services_pb"
 
 require "fluent/plugin/opentelemetry/constant"
+require "fluent/plugin/opentelemetry/grpc_tls"
 require "google/protobuf"
 
 class Fluent::Plugin::Opentelemetry::GrpcOutputHandler
@@ -56,10 +57,11 @@ class Fluent::Plugin::Opentelemetry::GrpcOutputHandler
     channel_args["grpc.keepalive_time_ms"] = grpc_config.keepalive_time * 1000
     channel_args["grpc.keepalive_timeout_ms"] = grpc_config.keepalive_timeout * 1000
     channel_args["grpc.keepalive_permit_without_calls"] = 1
+    credentials = Fluent::Plugin::Opentelemetry::GrpcTLS.channel_credentials(@transport_config)
     @services = {
-      Fluent::Plugin::Opentelemetry::RECORD_TYPE_LOGS => ServiceStub::Logs.new(@grpc_config.endpoint, :this_channel_is_insecure, channel_args: channel_args),
-      Fluent::Plugin::Opentelemetry::RECORD_TYPE_METRICS => ServiceStub::Metrics.new(@grpc_config.endpoint, :this_channel_is_insecure, channel_args: channel_args),
-      Fluent::Plugin::Opentelemetry::RECORD_TYPE_TRACES => ServiceStub::Traces.new(@grpc_config.endpoint, :this_channel_is_insecure, channel_args: channel_args)
+      Fluent::Plugin::Opentelemetry::RECORD_TYPE_LOGS => ServiceStub::Logs.new(@grpc_config.endpoint, credentials, channel_args: channel_args),
+      Fluent::Plugin::Opentelemetry::RECORD_TYPE_METRICS => ServiceStub::Metrics.new(@grpc_config.endpoint, credentials, channel_args: channel_args),
+      Fluent::Plugin::Opentelemetry::RECORD_TYPE_TRACES => ServiceStub::Traces.new(@grpc_config.endpoint, credentials, channel_args: channel_args)
     }
   end
 

data/lib/fluent/plugin/opentelemetry/grpc_tls.rb

@@ -0,0 +1,71 @@
+# frozen_string_literal: true
+
+require "fluent/plugin/opentelemetry/constant"
+
+require "openssl"
+
+module Fluent::Plugin::Opentelemetry
+  module GrpcTLS
+    module_function
+
+    def validate_server!(transport_config)
+      return unless tls?(transport_config)
+
+      if transport_config.cert_path.nil? || transport_config.private_key_path.nil?
+        raise Fluent::ConfigError, "<transport tls> cert_path and private_key_path are required when <grpc> is used"
+      end
+
+      if transport_config.client_cert_auth && transport_config.ca_path.nil?
+        raise Fluent::ConfigError, "<transport tls> client_cert_auth requires ca_path when <grpc> is used"
+      end
+
+      [transport_config.ca_path, transport_config.cert_path, transport_config.private_key_path].compact.each do |path|
+        raise Fluent::ConfigError, "<transport tls> cannot read '#{path}'" unless File.readable?(path)
+      end
+    end
+
+    def channel_credentials(transport_config)
+      return :this_channel_is_insecure unless tls?(transport_config)
+
+      GRPC::Core::ChannelCredentials.new(
+        read_pem(transport_config.ca_path),
+        read_private_key(transport_config),
+        read_pem(transport_config.cert_path)
+      )
+    end
+
+    def server_credentials(transport_config)
+      return :this_port_is_insecure unless tls?(transport_config)
+
+      GRPC::Core::ServerCredentials.new(
+        read_pem(transport_config.ca_path),
+        [{ private_key: read_private_key(transport_config), cert_chain: read_pem(transport_config.cert_path) }],
+        transport_config.client_cert_auth
+      )
+    end
+
+    def tls?(transport_config)
+      transport_config.protocol == :tls
+    end
+
+    def read_pem(path)
+      return nil unless path
+
+      File.read(path)
+    rescue SystemCallError, IOError => e
+      raise Fluent::ConfigError, "<transport tls> failed to read '#{path}': #{e.message}"
+    end
+
+    def read_private_key(transport_config)
+      pem = read_pem(transport_config.private_key_path)
+      return nil unless pem
+
+      OpenSSL::PKey.read(pem, transport_config.private_key_passphrase.to_s).to_pem
+    rescue OpenSSL::PKey::PKeyError => e
+      raise Fluent::ConfigError,
+            "<transport tls> failed to load private_key_path '#{transport_config.private_key_path}': #{e.message}"
+    end
+
+    private_class_method :tls?, :read_pem, :read_private_key
+  end
+end

data/lib/fluent/plugin/opentelemetry/http_input_handler.rb

@@ -119,6 +119,7 @@ class Fluent::Plugin::Opentelemetry::HttpInputHandler
   end
 
   BYTES_TO_READ = 64 * 1024
+  private_constant :BYTES_TO_READ
 
   def decompress(compressed_data, limit:)
     io = StringIO.new(compressed_data)

data/lib/fluent/plugin/opentelemetry/http_output_handler.rb

@@ -19,6 +19,7 @@ class Fluent::Plugin::Opentelemetry::HttpOutputHandler
       tls_settings[:client_cert] = @transport_config.cert_path
       tls_settings[:client_key] = @transport_config.private_key_path
       tls_settings[:client_key_pass] = @transport_config.private_key_passphrase
+      tls_settings[:ssl_ca_file] = @transport_config.ca_path unless @transport_config.ca_path.to_s.empty?
       tls_settings[:ssl_min_version] = Fluent::Plugin::Opentelemetry::TLS_VERSIONS_MAP[@transport_config.min_version]
       tls_settings[:ssl_max_version] = Fluent::Plugin::Opentelemetry::TLS_VERSIONS_MAP[@transport_config.max_version]
     end
@@ -29,7 +30,7 @@ class Fluent::Plugin::Opentelemetry::HttpOutputHandler
       connect_timeout: http_config.connect_timeout
     }
 
-    Excon.defaults[:ssl_verify_peer] = false if @transport_config.insecure
+    tls_settings[:ssl_verify_peer] = false if @transport_config.insecure && tls_settings[:ssl_ca_file].nil?
     @connections = {
       Fluent::Plugin::Opentelemetry::RECORD_TYPE_LOGS => Excon.new(http_logs_endpoint, proxy: @http_config.proxy, persistent: true, **tls_settings, **timeout_settings),
       Fluent::Plugin::Opentelemetry::RECORD_TYPE_METRICS => Excon.new(http_metrics_endpoint, proxy: @http_config.proxy, persistent: true, **tls_settings, **timeout_settings),

data/lib/fluent/plugin/opentelemetry/version.rb

@@ -3,7 +3,7 @@
 module Fluent
   module Plugin
     module Opentelemetry
-      VERSION = "0.5.3"
+      VERSION = "0.5.4"
     end
   end
 end

data/lib/fluent/plugin/out_opentelemetry.rb

@@ -90,7 +90,7 @@ module Fluent::Plugin
     end
 
     def write(chunk)
-      BatchProcessor.build_export_requests(chunk).each do |export_request|
+      BatchProcessor.build_export_requests(chunk, log).each do |export_request|
         if @grpc_handler
           @grpc_handler.export(export_request)
         else
@@ -106,22 +106,72 @@ module Fluent::Plugin
         Opentelemetry::RECORD_TYPE_TRACES => "resourceSpans"
       }.freeze
 
-      def self.build_export_requests(chunk)
+      class InvalidRecords
+        MAX_SAMPLE_BYTESIZE = 100
+        MAX_SAMPLES = 3
+        private_constant :MAX_SAMPLE_BYTESIZE, :MAX_SAMPLES
+
+        attr_reader :count
+
+        def initialize
+          @count = 0
+          @samples = []
+        end
+
+        def add(sample = nil)
+          @count += 1
+          return if sample.nil? || @samples.size >= MAX_SAMPLES
+
+          text = sample.byteslice(0, MAX_SAMPLE_BYTESIZE).scrub("")
+          @samples << text unless @samples.include?(text)
+        end
+
+        def to_s
+          @samples.empty? ? @count.to_s : "#{@count} (#{@samples.join(' | ')})"
+        end
+      end
+
+      def self.build_export_requests(chunk, logger)
         requests = {
           Opentelemetry::RECORD_TYPE_LOGS => {},
           Opentelemetry::RECORD_TYPE_METRICS => {},
           Opentelemetry::RECORD_TYPE_TRACES => {}
         }
+        invalid = Hash.new { |hash, key| hash[key] = InvalidRecords.new }
 
         chunk.each do |_, record| # rubocop:disable Style/HashEachMethods
           record_type = record["type"]
           resource_key = RESOURCE_KEY_MAP[record_type]
-          record["message"] = JSON.parse(record["message"])
-          resource_hash = record["message"][resource_key][0]["resource"].hash
+          unless resource_key
+            invalid["unknown type"].add(record_type.inspect)
+            next
+          end
+
+          begin
+            record["message"] = JSON.parse(record["message"])
+          rescue JSON::ParserError, TypeError => e
+            invalid["broken message"].add(e.message)
+            next
+          end
+
+          resources = record["message"][resource_key] if record["message"].is_a?(Hash)
+          unless resources.is_a?(Array) && resources.first.is_a?(Hash)
+            invalid["no #{resource_key}"].add
+            next
+          end
+
+          resource_hash = resources.first["resource"].hash
           if requests[record_type][resource_hash].nil?
             requests[record_type][resource_hash] = record
           else
-            requests[record_type][resource_hash]["message"][resource_key].concat(record["message"][resource_key])
+            requests[record_type][resource_hash]["message"][resource_key].concat(resources)
+          end
+        end
+
+        unless invalid.empty?
+          logger.warn do
+            details = invalid.map { |reason, records| "#{reason}=#{records}" }
+            "Skipped invalid records (total: #{invalid.values.sum(&:count)}): #{details.join(', ')}"
           end
         end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-opentelemetry
 version: !ruby/object:Gem::Version
-  version: 0.5.3
+  version: 0.5.4
 platform: ruby
 authors:
 - Shizuo Fujita
@@ -112,6 +112,7 @@ files:
 - lib/fluent/plugin/opentelemetry/constant.rb
 - lib/fluent/plugin/opentelemetry/grpc_input_handler.rb
 - lib/fluent/plugin/opentelemetry/grpc_output_handler.rb
+- lib/fluent/plugin/opentelemetry/grpc_tls.rb
 - lib/fluent/plugin/opentelemetry/http_input_handler.rb
 - lib/fluent/plugin/opentelemetry/http_output_handler.rb
 - lib/fluent/plugin/opentelemetry/request.rb
@@ -154,7 +155,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.0.18
 specification_version: 4
 summary: Fluentd input/output plugin to forward OpenTelemetry Protocol data.
 test_files: []