dogstatsd-ruby 5.0.1 → 5.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c77f82b5b9a858517a937a5b2db1a1f890a80c94ca63e60f12e256ab28f7192d
-  data.tar.gz: e95ee401174c084edb117068f73d1c72f08e602ea9b51363e57e53c176072472
+  metadata.gz: 539ec3d4e02bd8784df4d032b0f8e96ebb2a582b510ff5e27c6e7cfff4515926
+  data.tar.gz: d8ecf4c6724e491aaea6d738cc255891b55119987b1ca5665087d506552135ee
 SHA512:
-  metadata.gz: 88d54866c8693d2dab18a1370d6ba5951a42ec4ed62615c6194548dd5faa109731840187e2a7ddaf591f08d07064c8e173df74728f182fe5cb5593c896503e19
-  data.tar.gz: d8021d0b6b21efe7ab14841665b01819c397d4419b247615d88b7685b332b1c8bb9a13d05fc22d21baf5d709d5452f9a9a3d932438addfbf1238e8ab5656a8e5
+  metadata.gz: d615efd205f7858c55b9864e1d001d607dcb39a47a26c87f5e27402b25834cf750fb3904d10d667c63a437587d38ba315d7a3c609b7709bc28a8bc8138d30eef
+  data.tar.gz: 254bf33695141f521d55d5f5b9c4d6b3abf2408b6b231263949094cbc2c9f531bf3ff2aea92ae0f11d505bbb8d25a789a2c0da76d9a065b713f9dc9b5b9d836d

data/README.md

@@ -22,22 +22,83 @@ To instantiate a DogStatsd client:
 # Import the library
 require 'datadog/statsd'
 
-# Create a DogStatsD client instance.
+# Create a DogStatsD client instance
 statsd = Datadog::Statsd.new('localhost', 8125)
+# ...
+# release resources used by the client instance
+statsd.close()
 ```
 Or if you want to connect over Unix Domain Socket:
 ```ruby
 # Connection over Unix Domain Socket
 statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
+# ...
+# release resources used by the client instance
+statsd.close()
 ```
 
-Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby#client-instantiation-parameters).
+Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?code-lang=ruby#client-instantiation-parameters).
+
+### Migrating from v4.x to v5.x
+
+If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
+change concerning you is the new [threading model](#threading-model):
+
+In practice, it means two things:
+
+1. Now that the client is buffering metrics before sending them, you have to call `Datadog::Statsd#flush(sync: true)` if you want synchronous behavior. In most cases, this is not needed, as the sender thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
+
+2. You have to make sure you are either:
+
+  * Using a singleton instance of the DogStatsD client instead of creating a new instance whenever you need one; this will let the buffering mechanism flush metrics regularly
+  * Or properly disposing of the DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close`
+
+If you have issues with the sender thread or the buffering mode, you can instantiate a client that behaves exactly as in v4.x (i.e. no sender thread and flush on every metric submission):
+
+```ruby
+# Create a DogStatsD client instance using UDP
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true, buffer_max_pool_size: 1)
+# ...
+statsd.close()
+```
+
+or
+
+```ruby
+# Create a DogStatsD client instance using UDS
+statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file', single_thread: true, buffer_max_pool_size: 1)
+# ...
+statsd.close()
+```
+
+### v5.x Common Pitfalls
+
+Version v5.x of `dogstatsd-ruby` is using a sender thread for flushing. This provides better performance, but you need to consider the following pitfalls:
+
+1. Applications that use `fork` after having created the dogstatsd instance: the child process will automatically spawn a new sender thread to flush metrics.
+
+2. Applications that create multiple instances of the client without closing them: it is important to `#close` all instances to free the thread and the socket they are using otherwise you will leak those resources.
+
+If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
+
+If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
+
+Applications that run into issues but can't apply these recommendations should use the `single_thread` mode which disables the use of the sender thread.
+Here is how to instantiate a client in this mode:
+
+```ruby
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true)
+# ...
+# release resources used by the client instance and flush last metrics
+statsd.close()
+```
 
 ### Origin detection over UDP
 
-Origin detection is a method to detect which pod DogStatsD packets are coming from in order to add the pod's tags to the tag list.
+Origin detection is a method to detect which pod DogStatsD packets are coming from, in order to add the pod's tags to the tag list.
+
+To enable origin detection over UDP, add the following lines to your application manifest:
 
-To enable origin detection over UDP, add the following lines to your application manifest
 ```yaml
 env:
   - name: DD_ENTITY_ID
@@ -45,49 +106,109 @@ env:
       fieldRef:
         fieldPath: metadata.uid
 ```
+
 The DogStatsD client attaches an internal tag, `entity_id`. The value of this tag is the content of the `DD_ENTITY_ID` environment variable, which is the pod’s UID.
 
 ## Usage
 
-In order to use DogStatsD metrics, events, and Service Checks the Agent must be [running and available](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby).
+In order to use DogStatsD metrics, events, and Service Checks the Datadog Agent must be [running and available](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby).
 
 ### Metrics
 
-After the client is created, you can start sending custom metrics to Datadog. See the dedicated [Metric Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby) to see how to submit all supported metric types to Datadog with working code examples:
+After the client is created, you can start sending custom metrics to Datadog. See the dedicated [Metric Submission: DogStatsD documentation](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby) to see how to submit all supported metric types to Datadog with working code examples:
 
-* [Submit a COUNT metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#count).
-* [Submit a GAUGE metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#gauge).
-* [Submit a SET metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#set)
-* [Submit a HISTOGRAM metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#histogram)
-* [Submit a DISTRIBUTION metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#distribution)
+* [Submit a COUNT metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#count).
+* [Submit a GAUGE metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#gauge).
+* [Submit a SET metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#set)
+* [Submit a HISTOGRAM metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#histogram)
+* [Submit a DISTRIBUTION metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#distribution)
 
-Some options are suppported when submitting metrics, like [applying a Sample Rate to your metrics](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-submission-options) or [tagging your metrics with your custom tags](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-tagging). Find all the available functions to report metrics in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
+Some options are suppported when submitting metrics, like [applying a Sample Rate to your metrics](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-submission-options) or [tagging your metrics with your custom tags](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-tagging). Find all the available functions to report metrics in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
 
 ### Events
 
-After the client is created, you can start sending events to your Datadog Event Stream. See the dedicated [Event Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/events/dogstatsd/?tab=ruby) to see how to submit an event to Datadog your Event Stream.
+After the client is created, you can start sending events to your Datadog Event Stream. See the dedicated [Event Submission: DogStatsD documentation](https://docs.datadoghq.com/events/guides/dogstatsd/?code-lang=ruby) to see how to submit an event to Datadog your Event Stream.
 
 ### Service Checks
 
 After the client is created, you can start sending Service Checks to Datadog. See the dedicated [Service Check Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/?tab=ruby) to see how to submit a Service Check to Datadog.
 
-### Maximum packets size in high-throughput scenarios
+### Maximum packet size in high-throughput scenarios
 
 In order to have the most efficient use of this library in high-throughput scenarios,
-default values for the maximum packets size have already been set for both UDS (8192 bytes)
-and UDP (1432 bytes) in order to have the best usage of the underlying network.
-However, if you perfectly know your network and you know that a different value for the maximum packets
-size should be used, you can set it with the parameter `buffer_max_payload_size`. Example:
+recommended values for the maximum packet size have already been set for both UDS (8192 bytes)
+and UDP (1432 bytes).
+
+However, if are in control of your network and want to use a different value for the maximum packet
+size, you can do it by setting the `buffer_max_payload_size` parameter:
 
 ```ruby
-# Create a DogStatsD client instance.
 statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
+# ...
+statsd.close()
 ```
 
+## Threading model
+
+Starting with version 5.0, `dogstatsd-ruby` employs a new threading model where one instance of `Datadog::Statsd` can be shared between threads and where data sending is non-blocking (asynchronous).
+
+When you instantiate a `Datadog::Statsd`, a sender thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class. You can make use of `single_thread: true` to disable this behavior.
+
+This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). Instantiating a lot of statsd clients without calling `#close` after they are not needed anymore will most likely lead to threads being leaked.
+
+The sender thread has the following logic (from `Datadog::Statsd::Sender#send_loop`):
+
+```
+while the sender message queue is not closed do
+  read message from sender message queue
+
+  if message is a Control message to flush
+    flush buffer in connection
+  else if message is a Control message to synchronize
+    synchronize with calling thread
+  else
+    add message to the buffer
+  end
+end while
+```
+
+There are three different kinds of messages:
+
+1. a control message to flush the buffer in the connection
+2. a control message to synchronize any thread with the sender thread
+3. a message to append to the buffer
+
+There is also an implicit message which closes the queue which will cause the sender thread to finish processing and exit.
+
+### Usual workflow
+
+You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
+
+### Flushing
+
+When calling `Datadog::Statsd#flush`, a specific control message (`:flush`) is sent to the sender thread. When the sender thread receives it, it flushes its internal buffer into the connection.
+
+### Rendez-vous
+
+It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronous flush using `Datadog::Statsd#flush(sync: true)`.
+
+Doing so means the caller thread is blocked and waiting until the data has been flushed by the sender thread.
+
+This is useful when preparing to exit the application or when checking unit tests.
+
+### Thread-safety
+
+By default, instances of `Datadog::Statsd` are thread-safe and we recommend that a single instance be reused by all application threads (even in applications that employ forking). The sole exception is the `#close` method — this method is not yet thread safe (work in progress here [#209](https://github.com/DataDog/dogstatsd-ruby/pull/209)).
+
+When using the `single_thread: true` mode, instances of `Datadog::Statsd` are still thread-safe, but you may run into contention on heavily-threaded applications, so we don’t recommend (for performance reasons) reusing these instances.
+
+## Versioning
+
+This Ruby gem is using [Semantic Versioning](https://guides.rubygems.org/patterns/#semantic-versioning) but please note that supported Ruby versions can change in a minor release of this library. As much as possible, we will add a "future deprecation" message in the minor release preceding the one dropping the support.
+
 ## Credits
 
-dogstatsd-ruby is forked from Rein Henrichs [original Statsd
-client](https://github.com/reinh/statsd).
+dogstatsd-ruby is forked from Rein Henrichs' [original Statsd client](https://github.com/reinh/statsd).
 
 Copyright (c) 2011 Rein Henrichs. See LICENSE.txt for
 further details.

data/lib/datadog/statsd/connection.rb

@@ -8,16 +8,11 @@ module Datadog
         @logger = logger
       end
 
-      # Close the underlying socket
-      def close
-        begin
-          @socket && @socket.close if instance_variable_defined?(:@socket)
-        rescue StandardError => boom
-          logger.error { "Statsd: #{boom.class} #{boom}" } if logger
-        end
-        @socket = nil
+      def reset_telemetry
+        telemetry.reset
       end
 
+      # not thread safe: `Sender` instances that use this are required to properly synchronize or sequence calls to this method
       def write(payload)
         logger.debug { "Statsd: #{payload}" } if logger
 
@@ -36,6 +31,7 @@ module Datadog
           retries += 1
           begin
             close
+            connect
             retry
           rescue StandardError => e
             boom = e
@@ -48,11 +44,16 @@ module Datadog
       end
 
       private
+
       attr_reader :telemetry
       attr_reader :logger
 
-      def socket
-        @socket ||= connect
+      def connect
+        raise 'Should be implemented by subclass'
+      end
+
+      def close
+        raise 'Should be implemented by subclass'
       end
     end
   end

data/lib/datadog/statsd/forwarder.rb

@@ -18,6 +18,8 @@ module Datadog
         telemetry_flush_interval: nil,
         global_tags: [],
 
+        single_thread: false,
+
         logger: nil
       )
         @transport_type = socket_path.nil? ? :udp : :uds
@@ -47,13 +49,12 @@ module Datadog
           raise ArgumentError, "buffer_max_payload_size is not high enough to use telemetry (tags=(#{global_tags.inspect}))"
         end
 
-        @buffer = MessageBuffer.new(@connection,
+        buffer = MessageBuffer.new(@connection,
           max_payload_size: buffer_max_payload_size,
           max_pool_size: buffer_max_pool_size || DEFAULT_BUFFER_POOL_SIZE,
           overflowing_stategy: buffer_overflowing_stategy,
         )
-
-        @sender = Sender.new(buffer)
+        @sender = (single_thread ? SingleThreadSender : Sender).new(buffer, logger: logger)
         @sender.start
       end
 
@@ -97,7 +98,6 @@ module Datadog
       end
 
       private
-      attr_reader :buffer
       attr_reader :sender
       attr_reader :connection
 

data/lib/datadog/statsd/message_buffer.rb

@@ -19,7 +19,7 @@ module Datadog
         @overflowing_stategy = overflowing_stategy
 
         @buffer = String.new
-        @message_count = 0
+        clear_buffer
       end
 
       def add(message)
@@ -42,16 +42,20 @@ module Datadog
         true
       end
 
+      def reset
+        clear_buffer
+        connection.reset_telemetry
+      end
+
       def flush
         return if buffer.empty?
 
         connection.write(buffer)
-
-        buffer.clear
-        @message_count = 0
+        clear_buffer
       end
 
       private
+
       attr :max_payload_size
       attr :max_pool_size
 
@@ -66,6 +70,11 @@ module Datadog
         false
       end
 
+      def clear_buffer
+        buffer.clear
+        @message_count = 0
+      end
+
       def preemptive_flush?
         @message_count == max_pool_size || buffer.bytesize > bytesize_threshold
       end

data/lib/datadog/statsd/sender.rb

@@ -2,22 +2,45 @@
 
 module Datadog
   class Statsd
+    # Sender is using a companion thread to flush and pack messages
+    # in a `MessageBuffer`.
+    # The communication with this thread is done using a `Queue`.
+    # If the thread is dead, it is starting a new one to avoid having a blocked
+    # Sender with no companion thread to communicate with (most of the time, having
+    # a dead companion thread means that a fork just happened and that we are
+    # running in the child process).
     class Sender
       CLOSEABLE_QUEUES = Queue.instance_methods.include?(:close)
 
-      def initialize(message_buffer)
+      def initialize(message_buffer, logger: nil)
         @message_buffer = message_buffer
+        @logger = logger
+        @mx = Mutex.new
       end
 
       def flush(sync: false)
-        raise ArgumentError, 'Start sender first' unless message_queue
-
-        message_queue.push(:flush)
+        # keep a copy around in case another thread is calling #stop while this method is running
+        current_message_queue = message_queue
+
+        # don't try to flush if there is no message_queue instantiated or
+        # no companion thread running
+        if !current_message_queue
+          @logger.debug { "Statsd: can't flush: no message queue ready" } if @logger
+          return
+        end
+        if !sender_thread.alive?
+          @logger.debug { "Statsd: can't flush: no sender_thread alive" } if @logger
+          return
+        end
 
+        current_message_queue.push(:flush)
         rendez_vous if sync
       end
 
       def rendez_vous
+        # could happen if #start hasn't be called
+        return unless message_queue
+
         # Initialize and get the thread's sync queue
         queue = (Thread.current[:statsd_sync_queue] ||= Queue.new)
         # tell sender-thread to notify us in the current
@@ -31,19 +54,39 @@ module Datadog
       def add(message)
         raise ArgumentError, 'Start sender first' unless message_queue
 
+        # if the thread does not exist, we assume we are running in a forked process,
+        # empty the message queue and message buffers (these messages belong to
+        # the parent process) and spawn a new companion thread.
+        if !sender_thread.alive?
+          @mx.synchronize {
+            # a call from another thread has already re-created
+            # the companion thread before this one acquired the lock
+            break if sender_thread.alive?
+            @logger.debug { "Statsd: companion thread is dead, re-creating one" } if @logger
+
+            message_queue.close if CLOSEABLE_QUEUES
+            @message_queue = nil
+            message_buffer.reset
+            start
+          }
+        end
+
         message_queue << message
       end
 
       def start
         raise ArgumentError, 'Sender already started' if message_queue
 
-        # initialize message queue for background thread
+        # initialize a new message queue for the background thread
         @message_queue = Queue.new
         # start background thread
         @sender_thread = Thread.new(&method(:send_loop))
       end
 
       if CLOSEABLE_QUEUES
+        # when calling stop, make sure that no other threads is trying
+        # to close the sender nor trying to continue to `#add` more message
+        # into the sender.
         def stop(join_worker: true)
           message_queue = @message_queue
           message_queue.close if message_queue
@@ -52,6 +95,9 @@ module Datadog
           sender_thread.join if sender_thread && join_worker
         end
       else
+        # when calling stop, make sure that no other threads is trying
+        # to close the sender nor trying to continue to `#add` more message
+        # into the sender.
         def stop(join_worker: true)
           message_queue = @message_queue
           message_queue << :close if message_queue
@@ -64,7 +110,6 @@ module Datadog
       private
 
       attr_reader :message_buffer
-
       attr_reader :message_queue
       attr_reader :sender_thread
 

data/lib/datadog/statsd/single_thread_sender.rb

@@ -0,0 +1,62 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    # The SingleThreadSender is a sender synchronously buffering messages
+    # in a `MessageBuffer`.
+    # It is using current Process.PID to check it is the result of a recent fork
+    # and it is reseting the MessageBuffer if that's the case.
+    class SingleThreadSender
+      def initialize(message_buffer, logger: nil)
+        @message_buffer = message_buffer
+        @logger = logger
+        @mx = Mutex.new
+        # store the pid for which this sender has been created
+        update_fork_pid
+      end
+
+      def add(message)
+        @mx.synchronize {
+          # we have just forked, meaning we have messages in the buffer that we should
+          # not send, they belong to the parent process, let's clear the buffer.
+          if forked?
+            @message_buffer.reset
+            update_fork_pid
+          end
+          @message_buffer.add(message)
+        }
+      end
+
+      def flush(*)
+        @mx.synchronize {
+          @message_buffer.flush()
+        }
+      end
+
+      # Compatibility with `Sender`
+      def start()
+      end
+
+      # Compatibility with `Sender`
+      def stop()
+      end
+
+      # Compatibility with `Sender`
+      def rendez_vous()
+      end
+
+      private
+
+      # below are "fork management" methods to be able to clean the MessageBuffer
+      # if it detects that it is running in a unknown PID.
+
+      def forked?
+        Process.pid != @fork_pid
+      end
+
+      def update_fork_pid
+        @fork_pid = Process.pid
+      end
+    end
+  end
+end

data/lib/datadog/statsd/threaded_sender.rb

@@ -0,0 +1,132 @@
+# frozen_string_literal: true
+
+module Datadog
+  class FlushQueue < Queue
+  end
+  class CloseQueue < Queue
+  end
+  class Statsd
+    # Sender is using a background thread to flush and pack messages
+    # in a `MessageBuffer`.
+    # The communication with this thread is done using a `Queue`.
+    # If the thread is dead, it is starting a new one to avoid having a blocked
+    # Sender with no background thread to communicate with (most of the time,
+    # having a dead background thread means that a fork just happened and that we
+    # are running in the child process).
+    class Sender
+      CLOSEABLE_QUEUES = Queue.instance_methods.include?(:close)
+
+      def initialize(message_buffer, logger: nil)
+        @message_buffer = message_buffer
+        @logger = logger
+
+        # communication and synchronization with the background thread
+        # @mux is also used to not having multiple threads fighting for
+        # closing the Sender or creating a new background thread
+        @channel = Queue.new
+        @mux = Mutex.new
+
+        @is_closed = false
+
+        # start background thread immediately
+        @sender_thread = Thread.new(&method(:send_loop))
+      end
+
+      def flush(sync: false)
+        @mux.synchronize {
+          # we don't want to send a flush action to the bg thread if:
+          # - there is no bg thread running
+          # - the sender has been closed
+          return if !sender_thread.alive? || @is_closed
+
+          if sync
+             # blocking flush
+             blocking_queue = FlushQueue.new
+             channel << blocking_queue
+             blocking_queue.pop # wait for the bg thread to finish its work
+             blocking_queue.close if CLOSEABLE_QUEUES
+           else
+             # asynchronous flush
+             channel << :flush
+           end
+         }
+      end
+
+      def add(message)
+        return if @is_closed # don't send a message to the bg thread if the sender has been closed
+
+        # the bg thread is not running anymore, this is happening if the main process has forked and
+        # we are running in the child, we will spawn a bg thread and reset buffers (containing parents' messages)
+        if !sender_thread.alive?
+          @mux.synchronize {
+            return if @is_closed
+            # test if a call from another thread has already re-created
+            # the background thread before this one acquired the lock
+            break if sender_thread.alive?
+
+            # re-create the channel of communication since we will spawn a new bg thread
+            channel.close if CLOSEABLE_QUEUES
+            @channel = Queue.new
+            message_buffer.reset # don't use messages appended by another fork
+            @sender_thread = Thread.new(&method(:send_loop))
+          }
+        end
+
+        channel << message
+      end
+
+      # Compatibility with `Sender`
+      def start()
+      end
+
+      def stop()
+        return if @is_closed
+        # use this lock to both: not having another thread stopping this instance nor
+        # having a #add call creating a new thread
+        @mux.synchronize {
+          @is_closed = true
+          if sender_thread.alive? # no reasons to stop the bg thread is none is running already
+            blocking_queue = CloseQueue.new
+            channel << blocking_queue
+            blocking_queue.pop # wait for the bg thread to finish its work
+            blocking_queue.close if CLOSEABLE_QUEUES
+            sender_thread.join(3) # wait for completion, timeout after 3 seconds
+            # TODO(remy): should I close `channel` here?
+          end
+        }
+      end
+
+      private
+
+      attr_reader :message_buffer
+      attr_reader :channel
+      attr_reader :mux
+      attr_reader :sender_thread
+
+      def send_loop
+        until (message = channel.pop).nil? && (CLOSEABLE_QUEUES && channel.closed?)
+          # skip if message is nil, e.g. when the channel is empty and closed
+          next unless message
+
+          case message
+          # if a FlushQueue is received, the background thread has to flush the message
+          # buffer and to send an :unblock to let the caller know that it has finished
+          when FlushQueue
+            message_buffer.flush
+            message << :unblock
+          # if a :flush is received, the background thread has to flush asynchronously
+          when :flush
+            message_buffer.flush
+          # if a CloseQueue is received, the background thread has to do a last flush
+          # and to send an :unblock to let the caller know that it has finished
+          when CloseQueue
+            message << :unblock
+            return
+          else
+            message_buffer.add(message)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/datadog/statsd/udp_connection.rb

@@ -19,18 +19,30 @@ module Datadog
 
         @host = host || ENV.fetch('DD_AGENT_HOST', DEFAULT_HOST)
         @port = port || ENV.fetch('DD_DOGSTATSD_PORT', DEFAULT_PORT).to_i
+        @socket = nil
+      end
+
+      def close
+        @socket.close if @socket
+        @socket = nil
       end
 
       private
 
       def connect
-        UDPSocket.new.tap do |socket|
-          socket.connect(host, port)
-        end
+        close if @socket
+
+        @socket = UDPSocket.new
+        @socket.connect(host, port)
       end
 
+      # send_message is writing the message in the socket, it may create the socket if nil
+      # It is not thread-safe but since it is called by either the Sender bg thread or the
+      # SingleThreadSender (which is using a mutex while Flushing), only one thread must call
+      # it at a time.
       def send_message(message)
-        socket.send(message, 0)
+        connect unless @socket
+        @socket.send(message, 0)
       end
     end
   end

data/lib/datadog/statsd/uds_connection.rb

@@ -14,20 +14,31 @@ module Datadog
         super(**kwargs)
 
         @socket_path = socket_path
+        @socket = nil
+      end
+
+      def close
+        @socket.close if @socket
+        @socket = nil
       end
 
       private
 
       def connect
-        socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM)
-        socket.connect(Socket.pack_sockaddr_un(@socket_path))
-        socket
+        close if @socket
+
+        @socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM)
+        @socket.connect(Socket.pack_sockaddr_un(@socket_path))
       end
 
+      # send_message is writing the message in the socket, it may create the socket if nil
+      # It is not thread-safe but since it is called by either the Sender bg thread or the
+      # SingleThreadSender (which is using a mutex while Flushing), only one thread must call
+      # it at a time.
       def send_message(message)
-        socket.sendmsg_nonblock(message)
+        connect unless @socket
+        @socket.sendmsg_nonblock(message)
       rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::ENOENT => e
-        @socket = nil
         # TODO: FIXME: This error should be considered as a retryable error in the
         # Connection class. An even better solution would be to make BadSocketError inherit
         # from a specific retryable error class in the Connection class.

data/lib/datadog/statsd/version.rb

@@ -4,6 +4,6 @@ require_relative 'connection'
 
 module Datadog
   class Statsd
-    VERSION = '5.0.1'
+    VERSION = '5.3.1'
   end
 end

data/lib/datadog/statsd.rb

@@ -8,8 +8,12 @@ require_relative 'statsd/uds_connection'
 require_relative 'statsd/message_buffer'
 require_relative 'statsd/serialization'
 require_relative 'statsd/sender'
+require_relative 'statsd/single_thread_sender'
 require_relative 'statsd/forwarder'
 
+$deprecation_message_mutex = Mutex.new
+$deprecation_message_done = false
+
 # = Datadog::Statsd: A DogStatsd client (https://www.datadoghq.com)
 #
 # @example Set up a global Statsd client for a server on localhost:8125
@@ -70,6 +74,7 @@ module Datadog
     # @option [Integer] buffer_max_pool_size max messages to buffer
     # @option [String] socket_path unix socket path
     # @option [Float] default sample rate if not overridden
+    # @option [Boolean] single_thread flushes the metrics on the main thread instead of in a companion thread
     def initialize(
       host = nil,
       port = nil,
@@ -85,6 +90,8 @@ module Datadog
 
       logger: nil,
 
+      single_thread: false,
+
       telemetry_enable: true,
       telemetry_flush_interval: DEFAULT_TELEMETRY_FLUSH_INTERVAL
     )
@@ -97,6 +104,19 @@ module Datadog
       @serializer = Serialization::Serializer.new(prefix: @prefix, global_tags: tags)
       @sample_rate = sample_rate
 
+      # deprecation message for ruby < 2.1.0 users as we will drop support for ruby 2.0
+      # in dogstatsd-ruby 5.4.0
+      # TODO(remy): remove this message and the two global vars used in dogstatd-ruby 5.4.0
+      if RUBY_VERSION < '2.1.0' && $deprecation_message_mutex.try_lock && !$deprecation_message_done
+        if logger != nil
+          logger.warn { "deprecation: dogstatsd-ruby will drop support of Ruby < 2.1.0 in a next minor release" }
+        else
+          puts("warning: deprecation: dogstatsd-ruby will drop support of Ruby < 2.1.0 in a next minor release")
+        end
+        $deprecation_message_done = true
+        $deprecation_message_mutex.unlock
+      end
+
       @forwarder = Forwarder.new(
         host: host,
         port: port,
@@ -105,6 +125,8 @@ module Datadog
         global_tags: tags,
         logger: logger,
 
+        single_thread: single_thread,
+
         buffer_max_payload_size: buffer_max_payload_size,
         buffer_max_pool_size: buffer_max_pool_size,
         buffer_overflowing_stategy: buffer_overflowing_stategy,
@@ -320,7 +342,10 @@ module Datadog
     end
 
     # Close the underlying socket
-    def close
+    #
+    # @param [Boolean, true] flush Should we flush the metrics before closing
+    def close(flush: true)
+      flush(sync: true) if flush
       forwarder.close
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dogstatsd-ruby
 version: !ruby/object:Gem::Version
-  version: 5.0.1
+  version: 5.3.1
 platform: ruby
 authors:
 - Rein Henrichs
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-04-09 00:00:00.000000000 Z
+date: 2021-10-21 00:00:00.000000000 Z
 dependencies: []
 description: A Ruby DogStatsd client
 email: code@datadoghq.com
@@ -32,7 +32,9 @@ files:
 - lib/datadog/statsd/serialization/service_check_serializer.rb
 - lib/datadog/statsd/serialization/stat_serializer.rb
 - lib/datadog/statsd/serialization/tag_serializer.rb
+- lib/datadog/statsd/single_thread_sender.rb
 - lib/datadog/statsd/telemetry.rb
+- lib/datadog/statsd/threaded_sender.rb
 - lib/datadog/statsd/udp_connection.rb
 - lib/datadog/statsd/uds_connection.rb
 - lib/datadog/statsd/version.rb
@@ -41,9 +43,9 @@ licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/DataDog/dogstatsd-ruby/issues
-  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.0.1/CHANGELOG.md
-  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.0.1
-  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.0.1
+  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.3.1/CHANGELOG.md
+  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.3.1
+  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.3.1
 post_install_message:
 rdoc_options: []
 require_paths: