dogstatsd-ruby 4.8.2 → 5.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dca98cecf21fceb3e5a4c4e76ceb8be8ffa2032ad317d63d1b3df7817dd60ef7
-  data.tar.gz: 0ef534d77693c3f1efbd8e8c8e1399f54ee6664e1044a16c913e4f9fa302e3d7
+  metadata.gz: 9abfaabd9be58a320dd8cc8d1d4998ac287daa08e68ab894054cf2910f5f3133
+  data.tar.gz: 88ebf4ef472f7663897c06fa3e5753a17c12722bc1a58f8aa4e9fb0a3ace07c6
 SHA512:
-  metadata.gz: d95681cd4c29de465efb5f6429227716d5cf4c6d1c57d1d5f6ba06cae2297b6ac06a961763a800b7997e6bf0011fa6f922826fdea7842fde536943c151c8e050
-  data.tar.gz: 0b560868db6815d18283ae938536c234d169e5aba4e1d602ee97759b36ad19fab626297b56cecf07b426c14de211e4bab43a010474e51a347387f82cd0c9cabe
+  metadata.gz: 199527bd5e9d39d94be3cabf28dc5f4d8913c4d3f746a6018a22c9ed13440c34d77fc11b363f47f9e2fd325974005fdc3567a223de4669b388ced99c3cd0750f
+  data.tar.gz: 0da25bafef540e4d36be8cae9c1fbb1f11c6aea94415828941aa9c34877db75166574f41b5e1a0829c71285f958162133d0f592a34619ce066e14c97ea86fd4d

data/README.md

@@ -24,15 +24,88 @@ require 'datadog/statsd'
 
 # Create a DogStatsD client instance.
 statsd = Datadog::Statsd.new('localhost', 8125)
+...
+# release resources used by the client instance
+statsd.close()
 ```
 Or if you want to connect over Unix Domain Socket:
 ```ruby
 # Connection over Unix Domain Socket
 statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
+...
+# release resources used by the client instance
+statsd.close()
 ```
 
 Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby#client-instantiation-parameters).
 
+### Migrating from v4.x to v5.x
+
+If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
+change concerning you is the new threading model (please see section Threading model):
+
+In practice, it means two things:
+
+1. Now that the client is buffering metrics before sending them, you have to manually
+call the method `Datadog::Statsd#flush` if you want to force the sending of metrics. Note that the companion thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
+
+2. You have to make sure you are either:
+
+  * using singletons instances of the DogStatsD client and not allocating one each time you need one, letting the buffering mechanism flush metrics, it's still a bad solution if the process later forks (see related section below). Or,
+  * properly closing your DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close` to release the resources used by the instance and to close the socket
+
+If you have issues with the companion thread or the buffering mode, you can instantiate a client that behaves exactly as in v4.x (i.e. no companion thread and flush on every metric submission):
+
+```ruby
+# Import the library
+require 'datadog/statsd'
+
+# Create a DogStatsD client instance using UDP
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true, buffer_max_payload_size: 1)
+...
+# to close the instance is not necessary in this case since metrics are flushed on submission
+# but it is still a good practice and it explicitely closes the socket
+statsd.close()
+```
+
+or
+
+```ruby
+# Import the library
+require 'datadog/statsd'
+
+# Create a DogStatsD client instance using UDS
+statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file', single_thread: true, buffer_max_payload_size: 1)
+...
+# to close the instance is not necessary in this case since metrics are flushed on submission
+# but it is still a good practice and it explicitely closes the socket
+statsd.close()
+```
+
+### v5.x Common Pitfalls
+
+Version v5.x of `dogstatsd-ruby` is using a companion thread for preemptive flushing, it brings better performances for application having a high-throughput of statsd metrics, but it comes with new pitfalls:
+
+    * Applications forking after having created the dogstatsd instance: forking a process can't duplicate the existing threads, meaning that one of the processes won't have a companion thread to flush the metrics and will lead to missing metrics.
+    * Applications creating a lot of different instances of the client without closing them: it is important to close the instance to free the thread and the socket it is using or it will lead to thread leaks.
+
+If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
+
+If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
+
+Applications that are in these situations but can't apply these recommendations should enable the `single_thread` mode which does not use a companion thread. Here is how to instantiate a client in this mode:
+
+```ruby
+# Import the library
+require 'datadog/statsd'
+
+# Create a DogStatsD client instance.
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true)
+...
+# release resources used by the client instance and flush last metrics
+statsd.close()
+```
+
 ### Origin detection over UDP
 
 Origin detection is a method to detect which pod DogStatsD packets are coming from in order to add the pod's tags to the tag list.
@@ -84,9 +157,61 @@ size should be used, you can set it with the parameter `buffer_max_payload_size`
 statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
 ```
 
+## Threading model
+
+On versions greater than 5.0, we changed the threading model of the library so that one instance of `Datadog::Statsd` could be shared between threads and so that the writes in the socket are non blocking.
+
+When you instantiate a `Datadog::Statsd`, a companion thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class. Please use `single_thread: true` while creating an instance if you don't want to or can't use a companion thread.
+
+This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). It also means that allocating a lot of statsd clients without closing them properly when not used anymore
+could lead to a thread leak (even though they will be sleeping, blocked on IO).
+The communication between the current thread is managed through a standard Ruby Queue.
+
+The sender thread has the following logic (Code present in the method `Datadog::Statsd::Sender#send_loop`):
+
+```
+while the sender message queue is not closed do
+  read message from sender message queue
+
+  if message is a Control message to flush
+    flush buffer in connection
+  else if message is a Control message to synchronize
+    synchronize with calling thread
+  else
+    add message to the buffer
+  end
+end while
+```
+
+Most of the time, the sender thread is blocked and sleeping when doing a blocking read from the sender message queue.
+
+We can see that there is 3 different kind of messages:
+
+* a control message to flush the buffer in the connection
+* a control message to synchronize any thread with the sender thread
+* a message to append to the buffer
+
+There is also an implicit message which is closing the queue as it will stop blocking read from the message queue (if happening) and thus, stop the sender thread.
+
+### Usual workflow
+
+You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
+
+### Flushing
+
+When calling a flush, a specific control message (the `:flush` symbol) is sent to the sender thread. When finding it, it flushes its internal buffer into the connection.
+
+### Rendez-vous
+
+It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronized flush (calling `Datadog::Statsd#flush` with the `sync: true` option).
+
+This means the current thread is going to sleep and wait for a Queue which is given to the sender thread. When the sender thread reads this queue from its own message queue, it puts a placeholder message in it so that it wakes up the calling thread.
+
+This is useful when closing the application or when checking unit tests.
+
 ## Credits
 
-dogstatsd-ruby is forked from Rien Henrichs [original Statsd
+dogstatsd-ruby is forked from Rein Henrichs [original Statsd
 client](https://github.com/reinh/statsd).
 
 Copyright (c) 2011 Rein Henrichs. See LICENSE.txt for

data/lib/datadog/statsd.rb

@@ -5,8 +5,11 @@ require_relative 'statsd/version'
 require_relative 'statsd/telemetry'
 require_relative 'statsd/udp_connection'
 require_relative 'statsd/uds_connection'
-require_relative 'statsd/batch'
+require_relative 'statsd/message_buffer'
 require_relative 'statsd/serialization'
+require_relative 'statsd/sender'
+require_relative 'statsd/single_thread_sender'
+require_relative 'statsd/forwarder'
 
 # = Datadog::Statsd: A DogStatsd client (https://www.datadoghq.com)
 #
@@ -26,12 +29,17 @@ require_relative 'statsd/serialization'
 #   statsd = Datadog::Statsd.new 'localhost', 8125, tags: 'tag1:true'
 module Datadog
   class Statsd
+    class Error < StandardError
+    end
+
     OK       = 0
     WARNING  = 1
     CRITICAL = 2
     UNKNOWN  = 3
 
-    DEFAULT_BUFFER_SIZE = 8 * 1_024
+    UDP_DEFAULT_BUFFER_SIZE = 1_432
+    UDS_DEFAULT_BUFFER_SIZE = 8_192
+    DEFAULT_BUFFER_POOL_SIZE = Float::INFINITY
     MAX_EVENT_SIZE = 8 * 1_024
     # minimum flush interval for the telemetry in seconds
     DEFAULT_TELEMETRY_FLUSH_INTERVAL = 10
@@ -51,67 +59,64 @@ module Datadog
       serializer.global_tags
     end
 
-    # Buffer containing the statsd message before they are sent in batch
-    attr_reader :buffer
-
-    # Maximum buffer size in bytes before it is flushed
-    attr_reader :max_buffer_bytes
-
     # Default sample rate
     attr_reader :sample_rate
 
-    # Connection
-    attr_reader :connection
-
     # @param [String] host your statsd host
     # @param [Integer] port your statsd port
     # @option [String] namespace set a namespace to be prepended to every metric name
     # @option [Array<String>|Hash] tags tags to be added to every metric
     # @option [Logger] logger for debugging
-    # @option [Integer] max_buffer_bytes max bytes to buffer when using #batch
+    # @option [Integer] buffer_max_payload_size max bytes to buffer
+    # @option [Integer] buffer_max_pool_size max messages to buffer
     # @option [String] socket_path unix socket path
     # @option [Float] default sample rate if not overridden
+    # @option [Boolean] single_thread flushes the metrics on the main thread instead of in a companion thread
     def initialize(
       host = nil,
       port = nil,
+      socket_path: nil,
+
       namespace: nil,
       tags: nil,
-      max_buffer_bytes: DEFAULT_BUFFER_SIZE,
-      socket_path: nil,
-      logger: nil,
       sample_rate: nil,
-      disable_telemetry: false,
+
+      buffer_max_payload_size: nil,
+      buffer_max_pool_size: nil,
+      buffer_overflowing_stategy: :drop,
+
+      logger: nil,
+
+      single_thread: false,
+
+      telemetry_enable: true,
       telemetry_flush_interval: DEFAULT_TELEMETRY_FLUSH_INTERVAL
     )
       unless tags.nil? || tags.is_a?(Array) || tags.is_a?(Hash)
-        raise ArgumentError, 'tags must be a Array<String> or a Hash'
+        raise ArgumentError, 'tags must be an array of string tags or a Hash'
       end
 
       @namespace = namespace
       @prefix = @namespace ? "#{@namespace}.".freeze : nil
-
       @serializer = Serialization::Serializer.new(prefix: @prefix, global_tags: tags)
+      @sample_rate = sample_rate
 
-      transport_type = socket_path.nil? ? :udp : :uds
+      @forwarder = Forwarder.new(
+        host: host,
+        port: port,
+        socket_path: socket_path,
 
-      @telemetry = Telemetry.new(disable_telemetry, telemetry_flush_interval,
         global_tags: tags,
-        transport_type: transport_type
-      )
-
-      @connection = case transport_type
-                    when :udp
-                      UDPConnection.new(host, port, logger, telemetry)
-                    when :uds
-                      UDSConnection.new(socket_path, logger, telemetry)
-                    end
+        logger: logger,
 
-      @logger = logger
+        single_thread: single_thread,
 
-      @sample_rate = sample_rate
+        buffer_max_payload_size: buffer_max_payload_size,
+        buffer_max_pool_size: buffer_max_pool_size,
+        buffer_overflowing_stategy: buffer_overflowing_stategy,
 
-      # we reduce max_buffer_bytes by a the rough estimate of the telemetry payload
-      @batch = Batch.new(connection, (max_buffer_bytes - telemetry.estimate_max_size))
+        telemetry_flush_interval: telemetry_enable ? telemetry_flush_interval : nil,
+      )
     end
 
     # yield a new instance to a block and close it when done
@@ -270,9 +275,9 @@ module Datadog
     # @example Report a critical service check status
     #   $statsd.service_check('my.service.check', Statsd::CRITICAL, :tags=>['urgent'])
     def service_check(name, status, opts = EMPTY_OPTIONS)
-      telemetry.sent(service_checks: 1)
+      telemetry.sent(service_checks: 1) if telemetry
 
-      send_stat(serializer.to_service_check(name, status, opts))
+      forwarder.send_message(serializer.to_service_check(name, status, opts))
     end
 
     # This end point allows you to post events to the stream. You can tag them, set priority and even aggregate them with other events.
@@ -290,17 +295,25 @@ module Datadog
     # @option opts [String, nil] :priority ('normal') Can be "normal" or "low"
     # @option opts [String, nil] :source_type_name (nil) Assign a source type to the event
     # @option opts [String, nil] :alert_type ('info') Can be "error", "warning", "info" or "success".
+    # @option opts [Boolean, false] :truncate_if_too_long (false) Truncate the event if it is too long
     # @option opts [Array<String>] :tags tags to be added to every metric
     # @example Report an awful event:
     #   $statsd.event('Something terrible happened', 'The end is near if we do nothing', :alert_type=>'warning', :tags=>['end_of_times','urgent'])
     def event(title, text, opts = EMPTY_OPTIONS)
-      telemetry.sent(events: 1)
+      telemetry.sent(events: 1) if telemetry
 
-      send_stat(serializer.to_event(title, text, opts))
+      forwarder.send_message(serializer.to_event(title, text, opts))
     end
 
-    # Send several metrics in the same UDP Packet
-    # They will be buffered and flushed when the block finishes
+    # Send several metrics in the same packet.
+    # They will be buffered and flushed when the block finishes.
+    #
+    # This method exists for compatibility with v4.x versions, it is not needed
+    # anymore since the batching is now automatically done internally.
+    # It also means that an automatic flush could occur if the buffer is filled
+    # during the execution of the batch block.
+    #
+    # This method is DEPRECATED and will be removed in future v6.x API.
     #
     # @example Send several metrics in one packet:
     #   $statsd.batch do |s|
@@ -308,19 +321,50 @@ module Datadog
     #      s.increment('page.views')
     #    end
     def batch
-      @batch.open do
-        yield self
-      end
+      yield self
+      flush(sync: true)
     end
 
     # Close the underlying socket
-    def close
-      connection.close
+    #
+    # @param [Boolean, true] flush Should we flush the metrics before closing
+    def close(flush: true)
+      flush(sync: true) if flush
+      forwarder.close
+    end
+
+    def sync_with_outbound_io
+      forwarder.sync_with_outbound_io
+    end
+
+    # Flush the buffer into the connection
+    def flush(flush_telemetry: false, sync: false)
+      forwarder.flush(flush_telemetry: flush_telemetry, sync: sync)
+    end
+
+    def telemetry
+      forwarder.telemetry
+    end
+
+    def host
+      forwarder.host
+    end
+
+    def port
+      forwarder.port
+    end
+
+    def socket_path
+      forwarder.socket_path
+    end
+
+    def transport_type
+      forwarder.transport_type
     end
 
     private
     attr_reader :serializer
-    attr_reader :telemetry
+    attr_reader :forwarder
 
     PROCESS_TIME_SUPPORTED = (RUBY_VERSION >= '2.1.0')
     EMPTY_OPTIONS = {}.freeze
@@ -336,22 +380,14 @@ module Datadog
     end
 
     def send_stats(stat, delta, type, opts = EMPTY_OPTIONS)
-      telemetry.sent(metrics: 1)
+      telemetry.sent(metrics: 1) if telemetry
 
       sample_rate = opts[:sample_rate] || @sample_rate || 1
 
       if sample_rate == 1 || rand <= sample_rate
         full_stat = serializer.to_stat(stat, delta, type, tags: opts[:tags], sample_rate: sample_rate)
 
-        send_stat(full_stat)
-      end
-    end
-
-    def send_stat(message)
-      if @batch.open?
-        @batch.add(message)
-      else
-        @connection.write(message)
+        forwarder.send_message(full_stat)
       end
     end
   end

data/lib/datadog/statsd/connection.rb

@@ -3,8 +3,9 @@
 module Datadog
   class Statsd
     class Connection
-      def initialize(telemetry)
+      def initialize(telemetry: nil, logger: nil)
         @telemetry = telemetry
+        @logger = logger
       end
 
       # Close the underlying socket
@@ -20,15 +21,11 @@ module Datadog
       def write(payload)
         logger.debug { "Statsd: #{payload}" } if logger
 
-        flush_telemetry = telemetry.flush?
-
-        payload += telemetry.flush if flush_telemetry
-
         send_message(payload)
 
-        telemetry.reset if flush_telemetry
+        telemetry.sent(packets: 1, bytes: payload.length) if telemetry
 
-        telemetry.sent(packets: 1, bytes: payload.length)
+        true
       rescue StandardError => boom
         # Try once to reconnect if the socket has been closed
         retries ||= 1
@@ -45,7 +42,7 @@ module Datadog
           end
         end
 
-        telemetry.dropped(packets: 1, bytes: payload.length)
+        telemetry.dropped(packets: 1, bytes: payload.length) if telemetry
         logger.error { "Statsd: #{boom.class} #{boom}" } if logger
         nil
       end

data/lib/datadog/statsd/forwarder.rb

@@ -0,0 +1,122 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class Forwarder
+      attr_reader :telemetry
+      attr_reader :transport_type
+
+      def initialize(
+        host: nil,
+        port: nil,
+        socket_path: nil,
+
+        buffer_max_payload_size: nil,
+        buffer_max_pool_size: nil,
+        buffer_overflowing_stategy: :drop,
+
+        telemetry_flush_interval: nil,
+        global_tags: [],
+
+        single_thread: false,
+
+        logger: nil
+      )
+        @transport_type = socket_path.nil? ? :udp : :uds
+
+        if telemetry_flush_interval
+          @telemetry = Telemetry.new(telemetry_flush_interval,
+            global_tags: global_tags,
+            transport_type: transport_type
+          )
+        end
+
+        @connection = case transport_type
+                      when :udp
+                        UDPConnection.new(host, port, logger: logger, telemetry: telemetry)
+                      when :uds
+                        UDSConnection.new(socket_path, logger: logger, telemetry: telemetry)
+                      end
+
+        # Initialize buffer
+        buffer_max_payload_size ||= (transport_type == :udp ? UDP_DEFAULT_BUFFER_SIZE : UDS_DEFAULT_BUFFER_SIZE)
+
+        if buffer_max_payload_size <= 0
+          raise ArgumentError, 'buffer_max_payload_size cannot be <= 0'
+        end
+
+        unless telemetry.nil? || telemetry.would_fit_in?(buffer_max_payload_size)
+          raise ArgumentError, "buffer_max_payload_size is not high enough to use telemetry (tags=(#{global_tags.inspect}))"
+        end
+
+        @buffer = MessageBuffer.new(@connection,
+          max_payload_size: buffer_max_payload_size,
+          max_pool_size: buffer_max_pool_size || DEFAULT_BUFFER_POOL_SIZE,
+          overflowing_stategy: buffer_overflowing_stategy,
+        )
+
+        @sender = single_thread ? SingleThreadSender.new(buffer) : Sender.new(buffer)
+        @sender.start
+      end
+
+      def send_message(message)
+        sender.add(message)
+
+        tick_telemetry
+      end
+
+      def sync_with_outbound_io
+        sender.rendez_vous
+      end
+
+      def flush(flush_telemetry: false, sync: false)
+        do_flush_telemetry if telemetry && flush_telemetry
+
+        sender.flush(sync: sync)
+      end
+
+      def host
+        return nil unless transport_type == :udp
+
+        connection.host
+      end
+
+      def port
+        return nil unless transport_type == :udp
+
+        connection.port
+      end
+
+      def socket_path
+        return nil unless transport_type == :uds
+
+        connection.socket_path
+      end
+
+      def close
+        sender.stop
+        connection.close
+      end
+
+      private
+      attr_reader :buffer
+      attr_reader :sender
+      attr_reader :connection
+
+      def do_flush_telemetry
+        telemetry_snapshot = telemetry.flush
+        telemetry.reset
+
+        telemetry_snapshot.each do |message|
+          sender.add(message)
+        end
+      end
+
+      def tick_telemetry
+        return nil unless telemetry
+
+        do_flush_telemetry if telemetry.should_flush?
+      end
+    end
+  end
+end

data/lib/datadog/statsd/message_buffer.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class MessageBuffer
+      PAYLOAD_SIZE_TOLERANCE = 0.05
+
+      def initialize(connection,
+        max_payload_size: nil,
+        max_pool_size: DEFAULT_BUFFER_POOL_SIZE,
+        overflowing_stategy: :drop
+      )
+        raise ArgumentError, 'max_payload_size keyword argument must be provided' unless max_payload_size
+        raise ArgumentError, 'max_pool_size keyword argument must be provided' unless max_pool_size
+
+        @connection = connection
+        @max_payload_size = max_payload_size
+        @max_pool_size = max_pool_size
+        @overflowing_stategy = overflowing_stategy
+
+        @buffer = String.new
+        @message_count = 0
+      end
+
+      def add(message)
+        message_size = message.bytesize
+
+        return nil unless message_size > 0 # to avoid adding empty messages to the buffer
+        return nil unless ensure_sendable!(message_size)
+
+        flush if should_flush?(message_size)
+
+        buffer << "\n" unless buffer.empty?
+        buffer << message
+
+        @message_count += 1
+
+        # flush when we're pretty sure that we won't be able
+        # to add another message to the buffer
+        flush if preemptive_flush?
+
+        true
+      end
+
+      def flush
+        return if buffer.empty?
+
+        connection.write(buffer)
+
+        buffer.clear
+        @message_count = 0
+      end
+
+      private
+      attr :max_payload_size
+      attr :max_pool_size
+
+      attr :overflowing_stategy
+
+      attr :connection
+      attr :buffer
+
+      def should_flush?(message_size)
+        return true if buffer.bytesize + 1 + message_size >= max_payload_size
+
+        false
+      end
+
+      def preemptive_flush?
+        @message_count == max_pool_size || buffer.bytesize > bytesize_threshold
+      end
+
+      def ensure_sendable!(message_size)
+        return true if message_size <= max_payload_size
+
+        if overflowing_stategy == :raise
+          raise Error, 'Message too big for payload limit'
+        end
+
+        false
+      end
+
+      def bytesize_threshold
+        @bytesize_threshold ||= (max_payload_size - PAYLOAD_SIZE_TOLERANCE * max_payload_size).to_i
+      end
+    end
+  end
+end

data/lib/datadog/statsd/sender.rb

@@ -0,0 +1,117 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class Sender
+      CLOSEABLE_QUEUES = Queue.instance_methods.include?(:close)
+
+      def initialize(message_buffer)
+        @message_buffer = message_buffer
+      end
+
+      def flush(sync: false)
+        # don't try to flush if there is no message_queue instantiated
+        return unless message_queue
+
+        message_queue.push(:flush)
+
+        rendez_vous if sync
+      end
+
+      def rendez_vous
+        # Initialize and get the thread's sync queue
+        queue = (Thread.current[:statsd_sync_queue] ||= Queue.new)
+        # tell sender-thread to notify us in the current
+        # thread's queue
+        message_queue.push(queue)
+        # wait for the sender thread to send a message
+        # once the flush is done
+        queue.pop
+      end
+
+      def add(message)
+        raise ArgumentError, 'Start sender first' unless message_queue
+
+        message_queue << message
+      end
+
+      def start
+        raise ArgumentError, 'Sender already started' if message_queue
+
+        # initialize message queue for background thread
+        @message_queue = Queue.new
+        # start background thread
+        @sender_thread = Thread.new(&method(:send_loop))
+      end
+
+      if CLOSEABLE_QUEUES
+        def stop(join_worker: true)
+          message_queue = @message_queue
+          message_queue.close if message_queue
+
+          sender_thread = @sender_thread
+          sender_thread.join if sender_thread && join_worker
+        end
+      else
+        def stop(join_worker: true)
+          message_queue = @message_queue
+          message_queue << :close if message_queue
+
+          sender_thread = @sender_thread
+          sender_thread.join if sender_thread && join_worker
+        end
+      end
+
+      private
+
+      attr_reader :message_buffer
+
+      attr_reader :message_queue
+      attr_reader :sender_thread
+
+      if CLOSEABLE_QUEUES
+        def send_loop
+          until (message = message_queue.pop).nil? && message_queue.closed?
+            # skip if message is nil, e.g. when message_queue
+            # is empty and closed
+            next unless message
+
+            case message
+            when :flush
+              message_buffer.flush
+            when Queue
+              message.push(:go_on)
+            else
+              message_buffer.add(message)
+            end
+          end
+
+          @message_queue = nil
+          @sender_thread = nil
+        end
+      else
+        def send_loop
+          loop do
+            message = message_queue.pop
+
+            next unless message
+
+            case message
+            when :close
+              break
+            when :flush
+              message_buffer.flush
+            when Queue
+              message.push(:go_on)
+            else
+              message_buffer.add(message)
+            end
+          end
+
+          @message_queue = nil
+          @sender_thread = nil
+        end
+      end
+    end
+  end
+end

data/lib/datadog/statsd/serialization/event_serializer.rb

@@ -48,7 +48,11 @@ module Datadog
             end
 
             if event.bytesize > MAX_EVENT_SIZE
-              raise "Event #{title} payload is too big (more that 8KB), event discarded"
+              if options[:truncate_if_too_long]
+                event.slice!(MAX_EVENT_SIZE..event.length)
+              else
+                raise "Event #{title} payload is too big (more that 8KB), event discarded"
+              end
             end
           end
         end

data/lib/datadog/statsd/serialization/tag_serializer.rb

@@ -13,7 +13,11 @@ module Datadog
 
           # Convert to tag list and set
           @global_tags = to_tags_list(global_tags)
-          @global_tags_formatted = @global_tags.join(',') if @global_tags.any?
+          if @global_tags.any?
+            @global_tags_formatted = @global_tags.join(',')
+          else
+            @global_tags_formatted = nil
+          end
         end
 
         def format(message_tags)

data/lib/datadog/statsd/single_thread_sender.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class SingleThreadSender
+      def initialize(message_buffer)
+        @message_buffer = message_buffer
+      end
+
+      def add(message)
+        @message_buffer.add(message)
+      end
+
+      def flush(*)
+        @message_buffer.flush()
+      end
+
+      # Compatibility with `Sender`
+      def start()
+      end
+
+      # Compatibility with `Sender`
+      def stop()
+      end
+
+      # Compatibility with `Sender`
+      def rendez_vous()
+      end
+    end
+  end
+end

data/lib/datadog/statsd/telemetry.rb

@@ -11,10 +11,11 @@ module Datadog
       attr_reader :bytes_dropped
       attr_reader :packets_sent
       attr_reader :packets_dropped
-      attr_reader :estimate_max_size
 
-      def initialize(disabled, flush_interval, global_tags: [], transport_type: :udp)
-        @disabled = disabled
+      # Rough estimation of maximum telemetry message size without tags
+      MAX_TELEMETRY_MESSAGE_SIZE_WT_TAGS = 50 # bytes
+
+      def initialize(flush_interval, global_tags: [], transport_type: :udp)
         @flush_interval = flush_interval
         @global_tags = global_tags
         @transport_type = transport_type
@@ -27,15 +28,10 @@ module Datadog
           client_version: VERSION,
           client_transport: transport_type,
         ).format(global_tags)
+      end
 
-        # estimate_max_size is an estimation or the maximum size of the
-        # telemetry payload. Since we don't want our packet to go over
-        # 'max_buffer_bytes', we have to adjust with the size of the telemetry
-        # (and any tags used). The telemetry payload size will change depending
-        # on the actual value of metrics: metrics received, packet dropped,
-        # etc. This is why we add a 63bytes margin: 9 bytes for each of the 7
-        # telemetry metrics.
-        @estimate_max_size = disabled ? 0 : flush.length + 9 * 7
+      def would_fit_in?(max_buffer_payload_size)
+        MAX_TELEMETRY_MESSAGE_SIZE_WT_TAGS + serialized_tags.size < max_buffer_payload_size
       end
 
       def reset
@@ -63,27 +59,29 @@ module Datadog
         @packets_dropped += packets
       end
 
-      def flush?
+      def should_flush?
         @next_flush_time < now_in_s
       end
 
       def flush
-        return '' if @disabled
-
-        # using shorthand syntax to reduce the garbage collection
-        %Q(
-datadog.dogstatsd.client.metrics:#{@metrics}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.events:#{@events}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.service_checks:#{@service_checks}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.bytes_sent:#{@bytes_sent}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.bytes_dropped:#{@bytes_dropped}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.packets_sent:#{@packets_sent}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.packets_dropped:#{@packets_dropped}|#{COUNTER_TYPE}|##{serialized_tags})
+        [
+          sprintf(pattern, 'metrics', @metrics),
+          sprintf(pattern, 'events', @events),
+          sprintf(pattern, 'service_checks', @service_checks),
+          sprintf(pattern, 'bytes_sent', @bytes_sent),
+          sprintf(pattern, 'bytes_dropped', @bytes_dropped),
+          sprintf(pattern, 'packets_sent', @packets_sent),
+          sprintf(pattern, 'packets_dropped', @packets_dropped),
+        ]
       end
 
       private
       attr_reader :serialized_tags
 
+      def pattern
+        @pattern ||= "datadog.dogstatsd.client.%s:%d|#{COUNTER_TYPE}|##{serialized_tags}"
+      end
+
       if Kernel.const_defined?('Process') && Process.respond_to?(:clock_gettime)
         def now_in_s
           Process.clock_gettime(Process::CLOCK_MONOTONIC, :second)

data/lib/datadog/statsd/udp_connection.rb

@@ -14,11 +14,11 @@ module Datadog
       # StatsD port. Defaults to 8125.
       attr_reader :port
 
-      def initialize(host, port, logger, telemetry)
-        super(telemetry)
+      def initialize(host, port, **kwargs)
+        super(**kwargs)
+
         @host = host || ENV.fetch('DD_AGENT_HOST', DEFAULT_HOST)
         @port = port || ENV.fetch('DD_DOGSTATSD_PORT', DEFAULT_PORT).to_i
-        @logger = logger
       end
 
       private

data/lib/datadog/statsd/uds_connection.rb

@@ -10,10 +10,10 @@ module Datadog
       # DogStatsd unix socket path
       attr_reader :socket_path
 
-      def initialize(socket_path, logger, telemetry)
-        super(telemetry)
+      def initialize(socket_path, **kwargs)
+        super(**kwargs)
+
         @socket_path = socket_path
-        @logger = logger
       end
 
       private

data/lib/datadog/statsd/version.rb

@@ -4,6 +4,6 @@ require_relative 'connection'
 
 module Datadog
   class Statsd
-    VERSION = '4.8.2'
+    VERSION = '5.2.0'
   end
 end

metadata

@@ -1,16 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: dogstatsd-ruby
 version: !ruby/object:Gem::Version
-  version: 4.8.2
+  version: 5.2.0
 platform: ruby
 authors:
 - Rein Henrichs
+- Karim Bogtob
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-11-16 00:00:00.000000000 Z
+date: 2021-07-01 00:00:00.000000000 Z
 dependencies: []
-description: A Ruby DogStastd client
+description: A Ruby DogStatsd client
 email: code@datadoghq.com
 executables: []
 extensions: []
@@ -21,14 +22,17 @@ files:
 - LICENSE.txt
 - README.md
 - lib/datadog/statsd.rb
-- lib/datadog/statsd/batch.rb
 - lib/datadog/statsd/connection.rb
+- lib/datadog/statsd/forwarder.rb
+- lib/datadog/statsd/message_buffer.rb
+- lib/datadog/statsd/sender.rb
 - lib/datadog/statsd/serialization.rb
 - lib/datadog/statsd/serialization/event_serializer.rb
 - lib/datadog/statsd/serialization/serializer.rb
 - lib/datadog/statsd/serialization/service_check_serializer.rb
 - lib/datadog/statsd/serialization/stat_serializer.rb
 - lib/datadog/statsd/serialization/tag_serializer.rb
+- lib/datadog/statsd/single_thread_sender.rb
 - lib/datadog/statsd/telemetry.rb
 - lib/datadog/statsd/udp_connection.rb
 - lib/datadog/statsd/uds_connection.rb
@@ -38,9 +42,9 @@ licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/DataDog/dogstatsd-ruby/issues
-  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v4.8.2/CHANGELOG.md
-  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/4.8.2
-  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v4.8.2
+  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.2.0/CHANGELOG.md
+  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.2.0
+  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.2.0
 post_install_message:
 rdoc_options: []
 require_paths:

data/lib/datadog/statsd/batch.rb

@@ -1,56 +0,0 @@
-# frozen_string_literal: true
-
-module Datadog
-  class Statsd
-    class Batch
-      def initialize(connection, max_buffer_bytes)
-        @connection = connection
-        @max_buffer_bytes = max_buffer_bytes
-        @depth = 0
-        reset
-      end
-
-      def open
-        @depth += 1
-
-        yield
-      ensure
-        @depth -= 1
-        flush if !open?
-      end
-
-      def open?
-        @depth > 0
-      end
-
-      def add(message)
-        message_bytes = message.bytesize
-
-        unless @buffer_bytes == 0
-          if @buffer_bytes + 1 + message_bytes >= @max_buffer_bytes
-            flush
-          else
-            @buffer << "\n"
-            @buffer_bytes += 1
-          end
-        end
-
-        @buffer << message
-        @buffer_bytes += message_bytes
-      end
-
-      def flush
-        return if @buffer_bytes == 0
-        @connection.write(@buffer)
-        reset
-      end
-
-      private
-
-      def reset
-        @buffer = String.new
-        @buffer_bytes = 0
-      end
-    end
-  end
-end