dogstatsd-ruby 5.4.0 → 5.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d780ad7a840c021ee3cb31a9e8b512ef7ccff4b82b14be9e300206d3a07cc61e
-  data.tar.gz: 2da798fe3a84f313c5c84655136597c562e3dc1bfaa24f8ba6441ef722c7cc45
+  metadata.gz: 20363c18876c0641ab23a8cee8d28c22de3e4210420e398d639d6a7b384da168
+  data.tar.gz: 851d63e75192dd81451f7a94abba7efe5c81e9c4afdc2dce8102a76e44a10d9a
 SHA512:
-  metadata.gz: 8f92d7dab8099d571c291e26497020e53087ccdbeecfd84d955778f484112f9b94fb0f4ec7d043b9efabd5d249333168fa44e4fe6e54ba4c9541447b92983ceb
-  data.tar.gz: 8dadc086c84a821d4a1b1824a25ea4861bdf0b6a36cfcf8c50a8d06e19d6853c35974dbefb03fedc9ccd7a3f57ff13a19d206c5eef152c1aa5ffbc1f789da1bc
+  metadata.gz: 72f565700a7ddf32ca56a686617e840c53ad167770be812a66ad6af5d1641c96f89796fe147c1565c38c7d83be14fc638e0f8ce76a00e62e980f43a510826d18
+  data.tar.gz: 659bbd4b5b669a11bf62f723387d0263d76a243bda5c6b5de101700b20ad2991d6fd6535832857acdd15b0f809faff80cb1e9f459e1ddacb3262a6be27c4785f

data/README.md

@@ -81,8 +81,6 @@ Version v5.x of `dogstatsd-ruby` is using a sender thread for flushing. This pro
 
 If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
 
-If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
-
 Applications that run into issues but can't apply these recommendations should use the `single_thread` mode which disables the use of the sender thread.
 Here is how to instantiate a client in this mode:
 
@@ -185,7 +183,7 @@ There is also an implicit message which closes the queue which will cause the se
 statsd = Datadog::Statsd.new('localhost', 8125)
 ```
 
-The message queue's maximum size (in messages) is given by the `sender_queue_size` argument, and has appropriate defaults for UDP (2048) and UDS (512).
+The message queue's maximum size (in messages) is given by the `sender_queue_size` argument, and has appropriate defaults for UDP (2048), UDS (512) and `single_thread: true` (1).
 
 The `buffer_flush_interval`, if enabled, is implemented with an additional thread which manages the timing of those flushes.  This additional thread is used even if `single_thread: true`.
 
@@ -211,6 +209,16 @@ By default, instances of `Datadog::Statsd` are thread-safe and we recommend that
 
 When using the `single_thread: true` mode, instances of `Datadog::Statsd` are still thread-safe, but you may run into contention on heavily-threaded applications, so we don’t recommend (for performance reasons) reusing these instances.
 
+### Delaying serialization
+
+By default, message serialization happens synchronously whenever stat methods such as `#increment` gets called, blocking the caller. If the blocking is impacting your program's performance, you may want to consider the `delay_serialization: true` mode.
+
+The `delay_serialization: true` mode delays the serialization of metrics to avoid the wait when submitting metrics. Serialization will still have to happen at some point, but it might be postponed until a more convenient time, such as after an HTTP request has completed.
+
+In `single_thread: true` mode, you'll probably want to set `sender_queue_size:` from it's default of `1` to some greater value, so that it can benefit from `delay_serialization: true`. Messages will then be queued unserialized in the sender queue and processed normally whenever `sender_queue_size` is reached or `#flush` is called. You might set `sender_queue_size: Float::INFINITY` to allow for an unbounded queue that will only be processed on explicit `#flush`.
+
+In `single_thread: false` mode, `delay_serialization: true`, will cause serialization to happen inside the sender thread.
+
 ## Versioning
 
 This Ruby gem is using [Semantic Versioning](https://guides.rubygems.org/patterns/#semantic-versioning) but please note that supported Ruby versions can change in a minor release of this library.

data/lib/datadog/statsd/connection.rb

@@ -9,7 +9,7 @@ module Datadog
       end
 
       def reset_telemetry
-        telemetry.reset
+        telemetry.reset if telemetry
       end
 
       # not thread safe: `Sender` instances that use this are required to properly synchronize or sequence calls to this method

data/lib/datadog/statsd/connection_cfg.rb

@@ -23,12 +23,25 @@ module Datadog
 
       private
 
+      ERROR_MESSAGE = "Valid environment variables combination for connection configuration:\n" +
+                      "  - DD_DOGSTATSD_URL for UDP or UDS connection.\n" +
+                      "     Example for UDP: DD_DOGSTATSD_URL='udp://localhost:8125'\n" +
+                      "     Example for UDS: DD_DOGSTATSD_URL='unix:///path/to/unix.sock'\n" +
+                      "  or\n" +
+                      "  - DD_AGENT_HOST and DD_DOGSTATSD_PORT for an UDP connection. E.g. DD_AGENT_HOST='localhost' DD_DOGSTATSD_PORT=8125\n" +
+                      "  or\n" +
+                      "  - DD_DOGSTATSD_SOCKET for an UDS connection: E.g. DD_DOGSTATSD_SOCKET='/path/to/unix.sock'\n" +
+                      " Note that DD_DOGSTATSD_URL has priority on other environment variables."
+
       DEFAULT_HOST = '127.0.0.1'
       DEFAULT_PORT = 8125
 
+      UDP_PREFIX = 'udp://'
+      UDS_PREFIX = 'unix://'
+
       def initialize_with_constructor_args(host: nil, port: nil, socket_path: nil)
         try_initialize_with(host: host, port: port, socket_path: socket_path,
-          not_both_error_message: 
+          error_message: 
             "Both UDP: (host/port #{host}:#{port}) and UDS (socket_path #{socket_path}) " +
             "constructor arguments were given. Use only one or the other.",
           )
@@ -36,13 +49,11 @@ module Datadog
 
       def initialize_with_env_vars()
         try_initialize_with(
+          dogstatsd_url: ENV['DD_DOGSTATSD_URL'],
           host: ENV['DD_AGENT_HOST'],
           port: ENV['DD_DOGSTATSD_PORT'] && ENV['DD_DOGSTATSD_PORT'].to_i,
           socket_path: ENV['DD_DOGSTATSD_SOCKET'],
-          not_both_error_message:
-            "Both UDP (DD_AGENT_HOST/DD_DOGSTATSD_PORT #{ENV['DD_AGENT_HOST']}:#{ENV['DD_DOGSTATSD_PORT']}) " +
-            "and UDS (DD_DOGSTATSD_SOCKET #{ENV['DD_DOGSTATSD_SOCKET']}) environment variables are set. " +
-            "Set only one or the other.",
+          error_message: ERROR_MESSAGE,
         )
       end
 
@@ -50,9 +61,13 @@ module Datadog
         try_initialize_with(host: DEFAULT_HOST, port: DEFAULT_PORT)
       end
 
-      def try_initialize_with(host: nil, port: nil, socket_path: nil, not_both_error_message: "")
+      def try_initialize_with(dogstatsd_url: nil, host: nil, port: nil, socket_path: nil, error_message: ERROR_MESSAGE)
         if (host || port) && socket_path
-          raise ArgumentError, not_both_error_message
+          raise ArgumentError, error_message
+        end
+
+        if dogstatsd_url
+          host, port, socket_path = parse_dogstatsd_url(str: dogstatsd_url.to_s)
         end
 
         if host || port 
@@ -71,6 +86,40 @@ module Datadog
 
         return false
       end
+
+      def parse_dogstatsd_url(str:)
+        # udp socket connection
+
+        if str.start_with?(UDP_PREFIX)
+          dogstatsd_url = str[UDP_PREFIX.size..str.size]
+          host = nil
+          port = nil
+
+          if dogstatsd_url.include?(":")
+            parts = dogstatsd_url.split(":")
+            if parts.size > 2
+              raise ArgumentError, "Error: DD_DOGSTATSD_URL wrong format for an UDP connection. E.g. 'udp://localhost:8125'"
+            end
+
+            host = parts[0]
+            port = parts[1].to_i
+          else
+            host = dogstatsd_url
+          end
+
+          return host, port, nil
+        end
+
+        # unix socket connection
+
+        if str.start_with?(UDS_PREFIX)
+          return nil, nil, str[UDS_PREFIX.size..str.size]
+        end
+
+        # malformed value
+
+        raise ArgumentError, "Error: DD_DOGSTATSD_URL has been provided but is not starting with 'udp://' nor 'unix://'"
+      end
     end
   end
 end

data/lib/datadog/statsd/forwarder.rb

@@ -21,15 +21,19 @@ module Datadog
 
         single_thread: false,
 
-        logger: nil
+        logger: nil,
+
+        serializer:
       )
         @transport_type = connection_cfg.transport_type
 
-        if telemetry_flush_interval
-          @telemetry = Telemetry.new(telemetry_flush_interval,
+        @telemetry = if telemetry_flush_interval
+          Telemetry.new(telemetry_flush_interval,
             global_tags: global_tags,
             transport_type: @transport_type
           )
+        else
+          nil
         end
 
         @connection = connection_cfg.make_connection(logger: logger, telemetry: telemetry)
@@ -50,8 +54,10 @@ module Datadog
           max_payload_size: buffer_max_payload_size,
           max_pool_size: buffer_max_pool_size || DEFAULT_BUFFER_POOL_SIZE,
           overflowing_stategy: buffer_overflowing_stategy,
+          serializer: serializer
         )
 
+        sender_queue_size ||= 1 if single_thread
         sender_queue_size ||= (@transport_type == :udp ?
                                UDP_DEFAULT_SENDER_QUEUE_SIZE : UDS_DEFAULT_SENDER_QUEUE_SIZE)
 
@@ -59,7 +65,8 @@ module Datadog
           SingleThreadSender.new(
             buffer,
             logger: logger,
-            flush_interval: buffer_flush_interval) :
+            flush_interval: buffer_flush_interval,
+            queue_size: sender_queue_size) :
           Sender.new(
             buffer,
             logger: logger,

data/lib/datadog/statsd/message_buffer.rb

@@ -8,7 +8,8 @@ module Datadog
       def initialize(connection,
         max_payload_size: nil,
         max_pool_size: DEFAULT_BUFFER_POOL_SIZE,
-        overflowing_stategy: :drop
+        overflowing_stategy: :drop,
+        serializer:
       )
         raise ArgumentError, 'max_payload_size keyword argument must be provided' unless max_payload_size
         raise ArgumentError, 'max_pool_size keyword argument must be provided' unless max_pool_size
@@ -17,12 +18,19 @@ module Datadog
         @max_payload_size = max_payload_size
         @max_pool_size = max_pool_size
         @overflowing_stategy = overflowing_stategy
+        @serializer = serializer
 
         @buffer = String.new
         clear_buffer
       end
 
       def add(message)
+        # Serializes the message if it hasn't been already. Part of the
+        # delay_serialization feature.
+        if message.is_a?(Array)
+          message = @serializer.to_stat(*message[0], **message[1])
+        end
+
         message_size = message.bytesize
 
         return nil unless message_size > 0 # to avoid adding empty messages to the buffer

data/lib/datadog/statsd/sender.rb

@@ -20,8 +20,10 @@ module Datadog
         @mx = Mutex.new
         @queue_class = queue_class
         @thread_class = thread_class
-        if flush_interval
-          @flush_timer = Datadog::Statsd::Timer.new(flush_interval) { flush(sync: true) }
+        @flush_timer = if flush_interval
+          Datadog::Statsd::Timer.new(flush_interval) { flush(sync: true) }
+        else
+          nil
         end
       end
 
@@ -82,7 +84,10 @@ module Datadog
         if message_queue.length <= @queue_size
           message_queue << message
         else
-          @telemetry.dropped_queue(packets: 1, bytes: message.bytesize) if @telemetry
+          if @telemetry
+            bytesize = message.respond_to?(:bytesize) ? message.bytesize : 0
+            @telemetry.dropped_queue(packets: 1, bytes: bytesize)
+          end
         end
       end
 
@@ -102,10 +107,11 @@ module Datadog
         # to close the sender nor trying to continue to `#add` more message
         # into the sender.
         def stop(join_worker: true)
+          @flush_timer.stop if @flush_timer
+
           message_queue = @message_queue
           message_queue.close if message_queue
 
-          @flush_timer.stop if @flush_timer
           sender_thread = @sender_thread
           sender_thread.join if sender_thread && join_worker
         end
@@ -114,10 +120,11 @@ module Datadog
         # to close the sender nor trying to continue to `#add` more message
         # into the sender.
         def stop(join_worker: true)
+          @flush_timer.stop if @flush_timer
+
           message_queue = @message_queue
           message_queue << :close if message_queue
 
-          @flush_timer.stop if @flush_timer
           sender_thread = @sender_thread
           sender_thread.join if sender_thread && join_worker
         end

data/lib/datadog/statsd/single_thread_sender.rb

@@ -7,12 +7,16 @@ module Datadog
     # It is using current Process.PID to check it is the result of a recent fork
     # and it is reseting the MessageBuffer if that's the case.
     class SingleThreadSender
-      def initialize(message_buffer, logger: nil, flush_interval: nil)
+      def initialize(message_buffer, logger: nil, flush_interval: nil, queue_size: 1)
         @message_buffer = message_buffer
         @logger = logger
         @mx = Mutex.new
-        if flush_interval
-          @flush_timer = Datadog::Statsd::Timer.new(flush_interval) { flush }
+        @message_queue_size = queue_size
+        @message_queue = []
+        @flush_timer = if flush_interval
+          Datadog::Statsd::Timer.new(flush_interval) { flush }
+        else
+          nil
         end
         # store the pid for which this sender has been created
         update_fork_pid
@@ -24,15 +28,21 @@ module Datadog
           # not send, they belong to the parent process, let's clear the buffer.
           if forked?
             @message_buffer.reset
+            @message_queue.clear
             @flush_timer.start if @flush_timer && @flush_timer.stop?
             update_fork_pid
           end
-          @message_buffer.add(message)
+
+          @message_queue << message
+          if @message_queue.size >= @message_queue_size
+            drain_message_queue
+          end
         }
       end
 
       def flush(*)
         @mx.synchronize {
+          drain_message_queue
           @message_buffer.flush()
         }
       end
@@ -51,6 +61,12 @@ module Datadog
 
       private
 
+      def drain_message_queue
+        while msg = @message_queue.shift
+          @message_buffer.add(msg)
+        end
+      end
+
       # below are "fork management" methods to be able to clean the MessageBuffer
       # if it detects that it is running in a unknown PID.
 

data/lib/datadog/statsd/timer.rb

@@ -9,6 +9,7 @@ module Datadog
         @interval = interval
         @callback = callback
         @stop = true
+        @thread = nil
       end
 
       def start

data/lib/datadog/statsd/version.rb

@@ -4,6 +4,6 @@ require_relative 'connection'
 
 module Datadog
   class Statsd
-    VERSION = '5.4.0'
+    VERSION = '5.6.0'
   end
 end

data/lib/datadog/statsd.rb

@@ -76,11 +76,12 @@ module Datadog
     # @option [Logger] logger for debugging
     # @option [Integer] buffer_max_payload_size max bytes to buffer
     # @option [Integer] buffer_max_pool_size max messages to buffer
-    # @option [Integer] sender_queue_size size of the sender queue in number of buffers (multi-thread only)
+    # @option [Integer] sender_queue_size size of the sender queue in number of buffers
     # @option [Numeric] buffer_flush_interval interval in second to flush buffer
     # @option [String] socket_path unix socket path
     # @option [Float] default sample rate if not overridden
     # @option [Boolean] single_thread flushes the metrics on the main thread instead of in a companion thread
+    # @option [Boolean] delay_serialization delays stat serialization
     def initialize(
       host = nil,
       port = nil,
@@ -100,6 +101,7 @@ module Datadog
       logger: nil,
 
       single_thread: false,
+      delay_serialization: false,
 
       telemetry_enable: true,
       telemetry_flush_interval: DEFAULT_TELEMETRY_FLUSH_INTERVAL
@@ -112,6 +114,7 @@ module Datadog
       @prefix = @namespace ? "#{@namespace}.".freeze : nil
       @serializer = Serialization::Serializer.new(prefix: @prefix, global_tags: tags)
       @sample_rate = sample_rate
+      @delay_serialization = delay_serialization
 
       @forwarder = Forwarder.new(
         connection_cfg: ConnectionCfg.new(
@@ -133,6 +136,7 @@ module Datadog
         sender_queue_size: sender_queue_size,
 
         telemetry_flush_interval: telemetry_enable ? telemetry_flush_interval : nil,
+        serializer: serializer
       )
     end
 
@@ -237,6 +241,26 @@ module Datadog
       send_stats(stat, value, DISTRIBUTION_TYPE, opts)
     end
 
+    # Reports execution time of the provided block as a distribution.
+    #
+    # If the block fails, the stat is still reported, then the error
+    # is reraised
+    #
+    # @param [String] stat stat name.
+    # @param [Numeric] value distribution value.
+    # @param [Hash] opts the options to create the metric with
+    # @option opts [Numeric] :sample_rate sample rate, 1 for always
+    # @option opts [Array<String>] :tags An array of tags
+    # @example Report the time (in ms) taken to activate an account
+    #   $statsd.distribution_time('account.activate') { @account.activate! }
+    def distribution_time(stat, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
+      start = now
+      yield
+    ensure
+      distribution(stat, ((now - start) * 1000).round, opts)
+    end
+
     # Sends a timing (in ms) for the given stat to the statsd server. The
     # sample_rate determines what percentage of the time this report is sent. The
     # statsd server then uses the sample_rate to correctly track the average
@@ -405,7 +429,12 @@ module Datadog
       sample_rate = opts[:sample_rate] || @sample_rate || 1
 
       if sample_rate == 1 || opts[:pre_sampled] || rand <= sample_rate
-        full_stat = serializer.to_stat(stat, delta, type, tags: opts[:tags], sample_rate: sample_rate)
+        full_stat =
+          if @delay_serialization
+            [[stat, delta, type], {tags: opts[:tags], sample_rate: sample_rate}]
+          else
+            serializer.to_stat(stat, delta, type, tags: opts[:tags], sample_rate: sample_rate)
+          end
 
         forwarder.send_message(full_stat)
       end

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: dogstatsd-ruby
 version: !ruby/object:Gem::Version
-  version: 5.4.0
+  version: 5.6.0
 platform: ruby
 authors:
 - Rein Henrichs
 - Karim Bogtob
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-03-01 00:00:00.000000000 Z
+date: 2023-07-10 00:00:00.000000000 Z
 dependencies: []
 description: A Ruby DogStatsd client
 email: code@datadoghq.com
@@ -44,9 +44,9 @@ licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/DataDog/dogstatsd-ruby/issues
-  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.4.0/CHANGELOG.md
-  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.4.0
-  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.4.0
+  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.6.0/CHANGELOG.md
+  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.6.0
+  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.6.0
 post_install_message: |2+
 
   If you are upgrading from v4.x of the dogstatsd-ruby library, note the major change to the threading model:
@@ -66,8 +66,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.22
-signing_key: 
+rubygems_version: 3.2.3
+signing_key:
 specification_version: 4
 summary: A Ruby DogStatsd client
 test_files: []
+...