dogstatsd-ruby 4.8.1 → 5.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9657b1d27b4367d96457d57faaed215c8ed07343d5d0803549b9078bd9713fbe
-  data.tar.gz: ad7c7fb2638790a3fe0300c23d81e87c33dc0fa9f7d496bc164e148e67e18992
+  metadata.gz: 264d78a169508453151e7a5c3260c2299ad05c727dc9c72cfa52f8be721809d0
+  data.tar.gz: 8e0726168a20f7efcb2bdcf72ab3d748184cd652de4fd25871f9fdf676020baf
 SHA512:
-  metadata.gz: a6b5a0217129bdb2bb2e7da14711395775b2d8897f74ffd8bd9925dc6a5b8dd3503fe7ee8427e1840617197de251d5b46abe99284cf8df41928ad9b69399c814
-  data.tar.gz: c129849337907f82ddbeba16484371e18aa1f155d5fac46b1a62be31b178ceddcb3c47d04bf6ce59b1ea485770050918377dec4422d78f8b2c0d0e894794182f
+  metadata.gz: 1875c615a043db68c69d855af2b454e45ef41af1d7e1f726a6067c35e260eb0dbc9b706a0588608e31bd5087e500b1b77b1b604f3dc22750f1376b6583b78c9a
+  data.tar.gz: 36f60d62312f6844efa9ce1b53a86cde380140582e009b5e5bf07d3a4a859e4d762e92a2ebcfe38a668145255f82f72c84504caabb93f25475e895a5166a9969

data/README.md

@@ -24,15 +24,49 @@ require 'datadog/statsd'
 
 # Create a DogStatsD client instance.
 statsd = Datadog::Statsd.new('localhost', 8125)
+...
+# release resources used by the client instance
+statsd.close()
 ```
 Or if you want to connect over Unix Domain Socket:
 ```ruby
 # Connection over Unix Domain Socket
 statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
+...
+# release resources used by the client instance
+statsd.close()
 ```
 
 Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby#client-instantiation-parameters).
 
+### Migrating from v4.x to v5.x
+
+If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
+change concerning you is the new threading model (please see section Threading model):
+
+In practice, it means two things:
+
+1. Now that the client is buffering metrics before sending them, you have to manually
+call the method `Datadog::Statsd#flush` if you want to force the sending of metrics. Note that the companion thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
+
+2. You have to make sure you are either:
+
+  * using singletons instances of the DogStatsD client and not allocating one each time you need one, letting the buffering mechanism flush metrics, or,
+  * properly closing your DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close` to release the resources used by the instance and to close the socket
+
+### v5.x Common Pitfalls
+
+Version v5.x of `dogstatsd-ruby` is using a companion thread for preemptive flushing, it brings better performances for application having a high-throughput of statsd metrics, but it comes with new pitfalls:
+
+    * Applications forking after having created the dogstatsd instance: forking a process can't duplicate the existing threads, meaning that one of the processes won't have a companion thread to flush the metrics and will lead to missing metrics.
+    * Applications creating a lot of different instances of the client without closing them: it is important to close the instance to free the thread and the socket it is using or it will lead to thread leaks.
+
+If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
+
+If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
+
+Applications that are in these situations and can't apply these recommendations should pin dogstatsd-ruby v4.x using `gem 'dogstatsd-ruby', '~> 4.0'`. Note that v4.x will continue to be maintained until a future v5.x version can more easily fit these use cases.
+
 ### Origin detection over UDP
 
 Origin detection is a method to detect which pod DogStatsD packets are coming from in order to add the pod's tags to the tag list.
@@ -71,9 +105,74 @@ After the client is created, you can start sending events to your Datadog Event
 
 After the client is created, you can start sending Service Checks to Datadog. See the dedicated [Service Check Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/?tab=ruby) to see how to submit a Service Check to Datadog.
 
+### Maximum packets size in high-throughput scenarios
+
+In order to have the most efficient use of this library in high-throughput scenarios,
+default values for the maximum packets size have already been set for both UDS (8192 bytes)
+and UDP (1432 bytes) in order to have the best usage of the underlying network.
+However, if you perfectly know your network and you know that a different value for the maximum packets
+size should be used, you can set it with the parameter `buffer_max_payload_size`. Example:
+
+```ruby
+# Create a DogStatsD client instance.
+statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
+```
+
+## Threading model
+
+On versions greater than 5.0, we changed the threading model of the library so that one instance of `Datadog::Statsd` could be shared between threads and so that the writes in the socket are non blocking.
+
+When you instantiate a `Datadog::Statsd`, a companion thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class.
+
+This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). It also means that allocating a lot of statsd clients without closing them properly when not used anymore
+could lead to a thread leak (even though they will be sleeping, blocked on IO).
+The communication between the current thread is managed through a standard Ruby Queue.
+
+The sender thread has the following logic (Code present in the method `Datadog::Statsd::Sender#send_loop`):
+
+```
+while the sender message queue is not closed do
+  read message from sender message queue
+
+  if message is a Control message to flush
+    flush buffer in connection
+  else if message is a Control message to synchronize
+    synchronize with calling thread
+  else
+    add message to the buffer
+  end
+end while
+```
+
+Most of the time, the sender thread is blocked and sleeping when doing a blocking read from the sender message queue.
+
+We can see that there is 3 different kind of messages:
+
+* a control message to flush the buffer in the connection
+* a control message to synchronize any thread with the sender thread
+* a message to append to the buffer
+
+There is also an implicit message which is closing the queue as it will stop blocking read from the message queue (if happening) and thus, stop the sender thread.
+
+### Usual workflow
+
+You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
+
+### Flushing
+
+When calling a flush, a specific control message (the `:flush` symbol) is sent to the sender thread. When finding it, it flushes its internal buffer into the connection.
+
+### Rendez-vous
+
+It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronized flush (calling `Datadog::Statsd#flush` with the `sync: true` option).
+
+This means the current thread is going to sleep and wait for a Queue which is given to the sender thread. When the sender thread reads this queue from its own message queue, it puts a placeholder message in it so that it wakes up the calling thread.
+
+This is useful when closing the application or when checking unit tests.
+
 ## Credits
 
-dogstatsd-ruby is forked from Rien Henrichs [original Statsd
+dogstatsd-ruby is forked from Rein Henrichs [original Statsd
 client](https://github.com/reinh/statsd).
 
 Copyright (c) 2011 Rein Henrichs. See LICENSE.txt for

data/lib/datadog/statsd.rb

@@ -5,8 +5,10 @@ require_relative 'statsd/version'
 require_relative 'statsd/telemetry'
 require_relative 'statsd/udp_connection'
 require_relative 'statsd/uds_connection'
-require_relative 'statsd/batch'
+require_relative 'statsd/message_buffer'
 require_relative 'statsd/serialization'
+require_relative 'statsd/sender'
+require_relative 'statsd/forwarder'
 
 # = Datadog::Statsd: A DogStatsd client (https://www.datadoghq.com)
 #
@@ -26,12 +28,17 @@ require_relative 'statsd/serialization'
 #   statsd = Datadog::Statsd.new 'localhost', 8125, tags: 'tag1:true'
 module Datadog
   class Statsd
+    class Error < StandardError
+    end
+
     OK       = 0
     WARNING  = 1
     CRITICAL = 2
     UNKNOWN  = 3
 
-    DEFAULT_BUFFER_SIZE = 8 * 1_024
+    UDP_DEFAULT_BUFFER_SIZE = 1_432
+    UDS_DEFAULT_BUFFER_SIZE = 8_192
+    DEFAULT_BUFFER_POOL_SIZE = Float::INFINITY
     MAX_EVENT_SIZE = 8 * 1_024
     # minimum flush interval for the telemetry in seconds
     DEFAULT_TELEMETRY_FLUSH_INTERVAL = 10
@@ -51,67 +58,59 @@ module Datadog
       serializer.global_tags
     end
 
-    # Buffer containing the statsd message before they are sent in batch
-    attr_reader :buffer
-
-    # Maximum buffer size in bytes before it is flushed
-    attr_reader :max_buffer_bytes
-
     # Default sample rate
     attr_reader :sample_rate
 
-    # Connection
-    attr_reader :connection
-
     # @param [String] host your statsd host
     # @param [Integer] port your statsd port
     # @option [String] namespace set a namespace to be prepended to every metric name
     # @option [Array<String>|Hash] tags tags to be added to every metric
     # @option [Logger] logger for debugging
-    # @option [Integer] max_buffer_bytes max bytes to buffer when using #batch
+    # @option [Integer] buffer_max_payload_size max bytes to buffer
+    # @option [Integer] buffer_max_pool_size max messages to buffer
     # @option [String] socket_path unix socket path
     # @option [Float] default sample rate if not overridden
     def initialize(
       host = nil,
       port = nil,
+      socket_path: nil,
+
       namespace: nil,
       tags: nil,
-      max_buffer_bytes: DEFAULT_BUFFER_SIZE,
-      socket_path: nil,
-      logger: nil,
       sample_rate: nil,
-      disable_telemetry: false,
+
+      buffer_max_payload_size: nil,
+      buffer_max_pool_size: nil,
+      buffer_overflowing_stategy: :drop,
+
+      logger: nil,
+
+      telemetry_enable: true,
       telemetry_flush_interval: DEFAULT_TELEMETRY_FLUSH_INTERVAL
     )
       unless tags.nil? || tags.is_a?(Array) || tags.is_a?(Hash)
-        raise ArgumentError, 'tags must be a Array<String> or a Hash'
+        raise ArgumentError, 'tags must be an array of string tags or a Hash'
       end
 
       @namespace = namespace
       @prefix = @namespace ? "#{@namespace}.".freeze : nil
-
       @serializer = Serialization::Serializer.new(prefix: @prefix, global_tags: tags)
+      @sample_rate = sample_rate
 
-      transport_type = socket_path.nil? ? :udp : :uds
+      @forwarder = Forwarder.new(
+        host: host,
+        port: port,
+        socket_path: socket_path,
 
-      @telemetry = Telemetry.new(disable_telemetry, telemetry_flush_interval,
         global_tags: tags,
-        transport_type: transport_type
-      )
+        logger: logger,
 
-      @connection = case transport_type
-                    when :udp
-                      UDPConnection.new(host, port, logger, telemetry)
-                    when :uds
-                      UDSConnection.new(socket_path, logger, telemetry)
-                    end
+        buffer_max_payload_size: buffer_max_payload_size,
+        buffer_max_pool_size: buffer_max_pool_size,
+        buffer_overflowing_stategy: buffer_overflowing_stategy,
 
-      @logger = logger
-
-      @sample_rate = sample_rate
-
-      # we reduce max_buffer_bytes by a the rough estimate of the telemetry payload
-      @batch = Batch.new(connection, (max_buffer_bytes - telemetry.estimate_max_size))
+        telemetry_flush_interval: telemetry_enable ? telemetry_flush_interval : nil,
+      )
     end
 
     # yield a new instance to a block and close it when done
@@ -270,9 +269,9 @@ module Datadog
     # @example Report a critical service check status
     #   $statsd.service_check('my.service.check', Statsd::CRITICAL, :tags=>['urgent'])
     def service_check(name, status, opts = EMPTY_OPTIONS)
-      telemetry.sent(service_checks: 1)
+      telemetry.sent(service_checks: 1) if telemetry
 
-      send_stat(serializer.to_service_check(name, status, opts))
+      forwarder.send_message(serializer.to_service_check(name, status, opts))
     end
 
     # This end point allows you to post events to the stream. You can tag them, set priority and even aggregate them with other events.
@@ -290,17 +289,25 @@ module Datadog
     # @option opts [String, nil] :priority ('normal') Can be "normal" or "low"
     # @option opts [String, nil] :source_type_name (nil) Assign a source type to the event
     # @option opts [String, nil] :alert_type ('info') Can be "error", "warning", "info" or "success".
+    # @option opts [Boolean, false] :truncate_if_too_long (false) Truncate the event if it is too long
     # @option opts [Array<String>] :tags tags to be added to every metric
     # @example Report an awful event:
     #   $statsd.event('Something terrible happened', 'The end is near if we do nothing', :alert_type=>'warning', :tags=>['end_of_times','urgent'])
     def event(title, text, opts = EMPTY_OPTIONS)
-      telemetry.sent(events: 1)
+      telemetry.sent(events: 1) if telemetry
 
-      send_stat(serializer.to_event(title, text, opts))
+      forwarder.send_message(serializer.to_event(title, text, opts))
     end
 
-    # Send several metrics in the same UDP Packet
-    # They will be buffered and flushed when the block finishes
+    # Send several metrics in the same packet.
+    # They will be buffered and flushed when the block finishes.
+    #
+    # This method exists for compatibility with v4.x versions, it is not needed
+    # anymore since the batching is now automatically done internally.
+    # It also means that an automatic flush could occur if the buffer is filled
+    # during the execution of the batch block.
+    #
+    # This method is DEPRECATED and will be removed in future v6.x API.
     #
     # @example Send several metrics in one packet:
     #   $statsd.batch do |s|
@@ -308,19 +315,50 @@ module Datadog
     #      s.increment('page.views')
     #    end
     def batch
-      @batch.open do
-        yield self
-      end
+      yield self
+      flush(sync: true)
     end
 
     # Close the underlying socket
-    def close
-      connection.close
+    #
+    # @param [Boolean, true] flush Should we flush the metrics before closing
+    def close(flush: true)
+      flush(sync: true) if flush
+      forwarder.close
+    end
+
+    def sync_with_outbound_io
+      forwarder.sync_with_outbound_io
+    end
+
+    # Flush the buffer into the connection
+    def flush(flush_telemetry: false, sync: false)
+      forwarder.flush(flush_telemetry: flush_telemetry, sync: sync)
+    end
+
+    def telemetry
+      forwarder.telemetry
+    end
+
+    def host
+      forwarder.host
+    end
+
+    def port
+      forwarder.port
+    end
+
+    def socket_path
+      forwarder.socket_path
+    end
+
+    def transport_type
+      forwarder.transport_type
     end
 
     private
     attr_reader :serializer
-    attr_reader :telemetry
+    attr_reader :forwarder
 
     PROCESS_TIME_SUPPORTED = (RUBY_VERSION >= '2.1.0')
     EMPTY_OPTIONS = {}.freeze
@@ -336,22 +374,14 @@ module Datadog
     end
 
     def send_stats(stat, delta, type, opts = EMPTY_OPTIONS)
-      telemetry.sent(metrics: 1)
+      telemetry.sent(metrics: 1) if telemetry
 
       sample_rate = opts[:sample_rate] || @sample_rate || 1
 
       if sample_rate == 1 || rand <= sample_rate
         full_stat = serializer.to_stat(stat, delta, type, tags: opts[:tags], sample_rate: sample_rate)
 
-        send_stat(full_stat)
-      end
-    end
-
-    def send_stat(message)
-      if @batch.open?
-        @batch.add(message)
-      else
-        @connection.write(message)
+        forwarder.send_message(full_stat)
       end
     end
   end

data/lib/datadog/statsd/connection.rb

@@ -3,8 +3,9 @@
 module Datadog
   class Statsd
     class Connection
-      def initialize(telemetry)
+      def initialize(telemetry: nil, logger: nil)
         @telemetry = telemetry
+        @logger = logger
       end
 
       # Close the underlying socket
@@ -20,15 +21,11 @@ module Datadog
       def write(payload)
         logger.debug { "Statsd: #{payload}" } if logger
 
-        flush_telemetry = telemetry.flush?
-
-        payload += telemetry.flush if flush_telemetry
-
         send_message(payload)
 
-        telemetry.reset if flush_telemetry
+        telemetry.sent(packets: 1, bytes: payload.length) if telemetry
 
-        telemetry.sent(packets: 1, bytes: payload.length)
+        true
       rescue StandardError => boom
         # Try once to reconnect if the socket has been closed
         retries ||= 1
@@ -45,7 +42,7 @@ module Datadog
           end
         end
 
-        telemetry.dropped(packets: 1, bytes: payload.length)
+        telemetry.dropped(packets: 1, bytes: payload.length) if telemetry
         logger.error { "Statsd: #{boom.class} #{boom}" } if logger
         nil
       end

data/lib/datadog/statsd/forwarder.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class Forwarder
+      attr_reader :telemetry
+      attr_reader :transport_type
+
+      def initialize(
+        host: nil,
+        port: nil,
+        socket_path: nil,
+
+        buffer_max_payload_size: nil,
+        buffer_max_pool_size: nil,
+        buffer_overflowing_stategy: :drop,
+
+        telemetry_flush_interval: nil,
+        global_tags: [],
+
+        logger: nil
+      )
+        @transport_type = socket_path.nil? ? :udp : :uds
+
+        if telemetry_flush_interval
+          @telemetry = Telemetry.new(telemetry_flush_interval,
+            global_tags: global_tags,
+            transport_type: transport_type
+          )
+        end
+
+        @connection = case transport_type
+                      when :udp
+                        UDPConnection.new(host, port, logger: logger, telemetry: telemetry)
+                      when :uds
+                        UDSConnection.new(socket_path, logger: logger, telemetry: telemetry)
+                      end
+
+        # Initialize buffer
+        buffer_max_payload_size ||= (transport_type == :udp ? UDP_DEFAULT_BUFFER_SIZE : UDS_DEFAULT_BUFFER_SIZE)
+
+        if buffer_max_payload_size <= 0
+          raise ArgumentError, 'buffer_max_payload_size cannot be <= 0'
+        end
+
+        unless telemetry.nil? || telemetry.would_fit_in?(buffer_max_payload_size)
+          raise ArgumentError, "buffer_max_payload_size is not high enough to use telemetry (tags=(#{global_tags.inspect}))"
+        end
+
+        @buffer = MessageBuffer.new(@connection,
+          max_payload_size: buffer_max_payload_size,
+          max_pool_size: buffer_max_pool_size || DEFAULT_BUFFER_POOL_SIZE,
+          overflowing_stategy: buffer_overflowing_stategy,
+        )
+
+        @sender = Sender.new(buffer)
+        @sender.start
+      end
+
+      def send_message(message)
+        sender.add(message)
+
+        tick_telemetry
+      end
+
+      def sync_with_outbound_io
+        sender.rendez_vous
+      end
+
+      def flush(flush_telemetry: false, sync: false)
+        do_flush_telemetry if telemetry && flush_telemetry
+
+        sender.flush(sync: sync)
+      end
+
+      def host
+        return nil unless transport_type == :udp
+
+        connection.host
+      end
+
+      def port
+        return nil unless transport_type == :udp
+
+        connection.port
+      end
+
+      def socket_path
+        return nil unless transport_type == :uds
+
+        connection.socket_path
+      end
+
+      def close
+        sender.stop
+        connection.close
+      end
+
+      private
+      attr_reader :buffer
+      attr_reader :sender
+      attr_reader :connection
+
+      def do_flush_telemetry
+        telemetry_snapshot = telemetry.flush
+        telemetry.reset
+
+        telemetry_snapshot.each do |message|
+          sender.add(message)
+        end
+      end
+
+      def tick_telemetry
+        return nil unless telemetry
+
+        do_flush_telemetry if telemetry.should_flush?
+      end
+    end
+  end
+end

data/lib/datadog/statsd/message_buffer.rb

@@ -0,0 +1,88 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class MessageBuffer
+      PAYLOAD_SIZE_TOLERANCE = 0.05
+
+      def initialize(connection,
+        max_payload_size: nil,
+        max_pool_size: DEFAULT_BUFFER_POOL_SIZE,
+        overflowing_stategy: :drop
+      )
+        raise ArgumentError, 'max_payload_size keyword argument must be provided' unless max_payload_size
+        raise ArgumentError, 'max_pool_size keyword argument must be provided' unless max_pool_size
+
+        @connection = connection
+        @max_payload_size = max_payload_size
+        @max_pool_size = max_pool_size
+        @overflowing_stategy = overflowing_stategy
+
+        @buffer = String.new
+        @message_count = 0
+      end
+
+      def add(message)
+        message_size = message.bytesize
+
+        return nil unless message_size > 0 # to avoid adding empty messages to the buffer
+        return nil unless ensure_sendable!(message_size)
+
+        flush if should_flush?(message_size)
+
+        buffer << "\n" unless buffer.empty?
+        buffer << message
+
+        @message_count += 1
+
+        # flush when we're pretty sure that we won't be able
+        # to add another message to the buffer
+        flush if preemptive_flush?
+
+        true
+      end
+
+      def flush
+        return if buffer.empty?
+
+        connection.write(buffer)
+
+        buffer.clear
+        @message_count = 0
+      end
+
+      private
+      attr :max_payload_size
+      attr :max_pool_size
+
+      attr :overflowing_stategy
+
+      attr :connection
+      attr :buffer
+
+      def should_flush?(message_size)
+        return true if buffer.bytesize + 1 + message_size >= max_payload_size
+
+        false
+      end
+
+      def preemptive_flush?
+        @message_count == max_pool_size || buffer.bytesize > bytesize_threshold
+      end
+
+      def ensure_sendable!(message_size)
+        return true if message_size <= max_payload_size
+
+        if overflowing_stategy == :raise
+          raise Error, 'Message too big for payload limit'
+        end
+
+        false
+      end
+
+      def bytesize_threshold
+        @bytesize_threshold ||= (max_payload_size - PAYLOAD_SIZE_TOLERANCE * max_payload_size).to_i
+      end
+    end
+  end
+end

data/lib/datadog/statsd/sender.rb

@@ -0,0 +1,117 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class Sender
+      CLOSEABLE_QUEUES = Queue.instance_methods.include?(:close)
+
+      def initialize(message_buffer)
+        @message_buffer = message_buffer
+      end
+
+      def flush(sync: false)
+        # don't try to flush if there is no message_queue instantiated
+        return unless message_queue
+
+        message_queue.push(:flush)
+
+        rendez_vous if sync
+      end
+
+      def rendez_vous
+        # Initialize and get the thread's sync queue
+        queue = (Thread.current[:statsd_sync_queue] ||= Queue.new)
+        # tell sender-thread to notify us in the current
+        # thread's queue
+        message_queue.push(queue)
+        # wait for the sender thread to send a message
+        # once the flush is done
+        queue.pop
+      end
+
+      def add(message)
+        raise ArgumentError, 'Start sender first' unless message_queue
+
+        message_queue << message
+      end
+
+      def start
+        raise ArgumentError, 'Sender already started' if message_queue
+
+        # initialize message queue for background thread
+        @message_queue = Queue.new
+        # start background thread
+        @sender_thread = Thread.new(&method(:send_loop))
+      end
+
+      if CLOSEABLE_QUEUES
+        def stop(join_worker: true)
+          message_queue = @message_queue
+          message_queue.close if message_queue
+
+          sender_thread = @sender_thread
+          sender_thread.join if sender_thread && join_worker
+        end
+      else
+        def stop(join_worker: true)
+          message_queue = @message_queue
+          message_queue << :close if message_queue
+
+          sender_thread = @sender_thread
+          sender_thread.join if sender_thread && join_worker
+        end
+      end
+
+      private
+
+      attr_reader :message_buffer
+
+      attr_reader :message_queue
+      attr_reader :sender_thread
+
+      if CLOSEABLE_QUEUES
+        def send_loop
+          until (message = message_queue.pop).nil? && message_queue.closed?
+            # skip if message is nil, e.g. when message_queue
+            # is empty and closed
+            next unless message
+
+            case message
+            when :flush
+              message_buffer.flush
+            when Queue
+              message.push(:go_on)
+            else
+              message_buffer.add(message)
+            end
+          end
+
+          @message_queue = nil
+          @sender_thread = nil
+        end
+      else
+        def send_loop
+          loop do
+            message = message_queue.pop
+
+            next unless message
+
+            case message
+            when :close
+              break
+            when :flush
+              message_buffer.flush
+            when Queue
+              message.push(:go_on)
+            else
+              message_buffer.add(message)
+            end
+          end
+
+          @message_queue = nil
+          @sender_thread = nil
+        end
+      end
+    end
+  end
+end

data/lib/datadog/statsd/serialization/event_serializer.rb

@@ -48,7 +48,11 @@ module Datadog
             end
 
             if event.bytesize > MAX_EVENT_SIZE
-              raise "Event #{title} payload is too big (more that 8KB), event discarded"
+              if options[:truncate_if_too_long]
+                event.slice!(MAX_EVENT_SIZE..event.length)
+              else
+                raise "Event #{title} payload is too big (more that 8KB), event discarded"
+              end
             end
           end
         end

data/lib/datadog/statsd/serialization/stat_serializer.rb

@@ -6,34 +6,24 @@ module Datadog
       class StatSerializer
         def initialize(prefix, global_tags: [])
           @prefix = prefix
+          @prefix_str = prefix.to_s
           @tag_serializer = TagSerializer.new(global_tags)
         end
 
         def format(name, delta, type, tags: [], sample_rate: 1)
-          String.new.tap do |stat|
-            stat << prefix if prefix
-
-            # stat value
-            stat << formated_name(name)
-            stat << ':'
-            stat << delta.to_s
-
-            # stat type
-            stat << '|'
-            stat << type
-
-            # sample_rate
-            if sample_rate != 1
-              stat << '|'
-              stat << '@'
-              stat << sample_rate.to_s
-            end
+          name = formated_name(name)
 
-            # tags
+          if sample_rate != 1
+            if tags_list = tag_serializer.format(tags)
+              "#{@prefix_str}#{name}:#{delta}|#{type}|@#{sample_rate}|##{tags_list}"
+            else
+              "#{@prefix_str}#{name}:#{delta}|#{type}|@#{sample_rate}"
+            end
+          else
             if tags_list = tag_serializer.format(tags)
-              stat << '|'
-              stat << '#'
-              stat << tags_list
+              "#{@prefix_str}#{name}:#{delta}|#{type}|##{tags_list}"
+            else
+              "#{@prefix_str}#{name}:#{delta}|#{type}"
             end
           end
         end
@@ -43,18 +33,21 @@ module Datadog
         end
 
         private
+
         attr_reader :prefix
         attr_reader :tag_serializer
 
         def formated_name(name)
-          formated = name.is_a?(String) ? name.dup : name.to_s
-
-          formated.tap do |f|
-            # replace Ruby module scoping with '.'
-            f.gsub!('::', '.')
-            # replace reserved chars (: | @) with underscores.
-            f.tr!(':|@', '_')
+          if name.is_a?(String)
+            # DEV: gsub is faster than dup.gsub!
+            formated = name.gsub('::', '.')
+          else
+            formated = name.to_s
+            formated.gsub!('::', '.')
           end
+
+          formated.tr!(':|@', '_')
+          formated
         end
       end
     end

data/lib/datadog/statsd/serialization/tag_serializer.rb

@@ -13,18 +13,25 @@ module Datadog
 
           # Convert to tag list and set
           @global_tags = to_tags_list(global_tags)
+          if @global_tags.any?
+            @global_tags_formatted = @global_tags.join(',')
+          else
+            @global_tags_formatted = nil
+          end
         end
 
         def format(message_tags)
-          # fast return global tags if there's no message_tags
-          # to avoid more allocations
-          tag_list =  if message_tags && message_tags.any?
-                        global_tags + to_tags_list(message_tags)
-                      else
-                        global_tags
-                      end
+          if !message_tags || message_tags.empty?
+            return @global_tags_formatted
+          end
 
-          tag_list.join(',') if tag_list.any?
+          tags = if @global_tags_formatted
+                   [@global_tags_formatted, to_tags_list(message_tags)]
+                 else
+                   to_tags_list(message_tags)
+                 end
+
+          tags.join(',')
         end
 
         attr_reader :global_tags
@@ -51,19 +58,17 @@ module Datadog
         def to_tags_list(tags)
           case tags
           when Hash
-            tags.each_with_object([]) do |tag_pair, formatted_tags|
-              if tag_pair.last.nil?
-                formatted_tags << "#{tag_pair.first}"
+            tags.map do |name, value|
+              if value
+                escape_tag_content("#{name}:#{value}")
               else
-                formatted_tags << "#{tag_pair.first}:#{tag_pair.last}"
+                escape_tag_content(name)
               end
             end
           when Array
-            tags.dup
+            tags.map { |tag| escape_tag_content(tag) }
           else
             []
-          end.map! do |tag|
-            escape_tag_content(tag)
           end
         end
 

data/lib/datadog/statsd/telemetry.rb

@@ -11,10 +11,11 @@ module Datadog
       attr_reader :bytes_dropped
       attr_reader :packets_sent
       attr_reader :packets_dropped
-      attr_reader :estimate_max_size
 
-      def initialize(disabled, flush_interval, global_tags: [], transport_type: :udp)
-        @disabled = disabled
+      # Rough estimation of maximum telemetry message size without tags
+      MAX_TELEMETRY_MESSAGE_SIZE_WT_TAGS = 50 # bytes
+
+      def initialize(flush_interval, global_tags: [], transport_type: :udp)
         @flush_interval = flush_interval
         @global_tags = global_tags
         @transport_type = transport_type
@@ -27,15 +28,10 @@ module Datadog
           client_version: VERSION,
           client_transport: transport_type,
         ).format(global_tags)
+      end
 
-        # estimate_max_size is an estimation or the maximum size of the
-        # telemetry payload. Since we don't want our packet to go over
-        # 'max_buffer_bytes', we have to adjust with the size of the telemetry
-        # (and any tags used). The telemetry payload size will change depending
-        # on the actual value of metrics: metrics received, packet dropped,
-        # etc. This is why we add a 63bytes margin: 9 bytes for each of the 7
-        # telemetry metrics.
-        @estimate_max_size = disabled ? 0 : flush.length + 9 * 7
+      def would_fit_in?(max_buffer_payload_size)
+        MAX_TELEMETRY_MESSAGE_SIZE_WT_TAGS + serialized_tags.size < max_buffer_payload_size
       end
 
       def reset
@@ -63,27 +59,29 @@ module Datadog
         @packets_dropped += packets
       end
 
-      def flush?
+      def should_flush?
         @next_flush_time < now_in_s
       end
 
       def flush
-        return '' if @disabled
-
-        # using shorthand syntax to reduce the garbage collection
-        %Q(
-datadog.dogstatsd.client.metrics:#{@metrics}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.events:#{@events}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.service_checks:#{@service_checks}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.bytes_sent:#{@bytes_sent}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.bytes_dropped:#{@bytes_dropped}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.packets_sent:#{@packets_sent}|#{COUNTER_TYPE}|##{serialized_tags}
-datadog.dogstatsd.client.packets_dropped:#{@packets_dropped}|#{COUNTER_TYPE}|##{serialized_tags})
+        [
+          sprintf(pattern, 'metrics', @metrics),
+          sprintf(pattern, 'events', @events),
+          sprintf(pattern, 'service_checks', @service_checks),
+          sprintf(pattern, 'bytes_sent', @bytes_sent),
+          sprintf(pattern, 'bytes_dropped', @bytes_dropped),
+          sprintf(pattern, 'packets_sent', @packets_sent),
+          sprintf(pattern, 'packets_dropped', @packets_dropped),
+        ]
       end
 
       private
       attr_reader :serialized_tags
 
+      def pattern
+        @pattern ||= "datadog.dogstatsd.client.%s:%d|#{COUNTER_TYPE}|##{serialized_tags}"
+      end
+
       if Kernel.const_defined?('Process') && Process.respond_to?(:clock_gettime)
         def now_in_s
           Process.clock_gettime(Process::CLOCK_MONOTONIC, :second)

data/lib/datadog/statsd/udp_connection.rb

@@ -14,11 +14,11 @@ module Datadog
       # StatsD port. Defaults to 8125.
       attr_reader :port
 
-      def initialize(host, port, logger, telemetry)
-        super(telemetry)
+      def initialize(host, port, **kwargs)
+        super(**kwargs)
+
         @host = host || ENV.fetch('DD_AGENT_HOST', DEFAULT_HOST)
         @port = port || ENV.fetch('DD_DOGSTATSD_PORT', DEFAULT_PORT).to_i
-        @logger = logger
       end
 
       private

data/lib/datadog/statsd/uds_connection.rb

@@ -10,10 +10,10 @@ module Datadog
       # DogStatsd unix socket path
       attr_reader :socket_path
 
-      def initialize(socket_path, logger, telemetry)
-        super(telemetry)
+      def initialize(socket_path, **kwargs)
+        super(**kwargs)
+
         @socket_path = socket_path
-        @logger = logger
       end
 
       private

data/lib/datadog/statsd/version.rb

@@ -4,6 +4,6 @@ require_relative 'connection'
 
 module Datadog
   class Statsd
-    VERSION = '4.8.1'
+    VERSION = '5.1.0'
   end
 end

metadata

@@ -1,16 +1,17 @@
 --- !ruby/object:Gem::Specification
 name: dogstatsd-ruby
 version: !ruby/object:Gem::Version
-  version: 4.8.1
+  version: 5.1.0
 platform: ruby
 authors:
 - Rein Henrichs
-autorequire: 
+- Karim Bogtob
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-05-25 00:00:00.000000000 Z
+date: 2021-06-17 00:00:00.000000000 Z
 dependencies: []
-description: A Ruby DogStastd client
+description: A Ruby DogStatsd client
 email: code@datadoghq.com
 executables: []
 extensions: []
@@ -21,8 +22,10 @@ files:
 - LICENSE.txt
 - README.md
 - lib/datadog/statsd.rb
-- lib/datadog/statsd/batch.rb
 - lib/datadog/statsd/connection.rb
+- lib/datadog/statsd/forwarder.rb
+- lib/datadog/statsd/message_buffer.rb
+- lib/datadog/statsd/sender.rb
 - lib/datadog/statsd/serialization.rb
 - lib/datadog/statsd/serialization/event_serializer.rb
 - lib/datadog/statsd/serialization/serializer.rb
@@ -38,10 +41,10 @@ licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/DataDog/dogstatsd-ruby/issues
-  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v4.8.1/CHANGELOG.md
-  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/4.8.1
-  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v4.8.1
-post_install_message: 
+  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.1.0/CHANGELOG.md
+  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.1.0
+  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.1.0
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -56,8 +59,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.6
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.7.10
+signing_key:
 specification_version: 4
 summary: A Ruby DogStatsd client
 test_files: []

data/lib/datadog/statsd/batch.rb

@@ -1,56 +0,0 @@
-# frozen_string_literal: true
-
-module Datadog
-  class Statsd
-    class Batch
-      def initialize(connection, max_buffer_bytes)
-        @connection = connection
-        @max_buffer_bytes = max_buffer_bytes
-        @depth = 0
-        reset
-      end
-
-      def open
-        @depth += 1
-
-        yield
-      ensure
-        @depth -= 1
-        flush if !open?
-      end
-
-      def open?
-        @depth > 0
-      end
-
-      def add(message)
-        message_bytes = message.bytesize
-
-        unless @buffer_bytes == 0
-          if @buffer_bytes + 1 + message_bytes >= @max_buffer_bytes
-            flush
-          else
-            @buffer << "\n"
-            @buffer_bytes += 1
-          end
-        end
-
-        @buffer << message
-        @buffer_bytes += message_bytes
-      end
-
-      def flush
-        return if @buffer_bytes == 0
-        @connection.write(@buffer)
-        reset
-      end
-
-      private
-
-      def reset
-        @buffer = String.new
-        @buffer_bytes = 0
-      end
-    end
-  end
-end