dogstatsd-ruby 3.3.0 → 5.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 046a0af9f230451c91fdeb9ae128c550b1c20623fa017d326e63a2f053f7cc9a
-  data.tar.gz: e2ba838497de521e89d386645a0228f819fdd03c28acc05a4789aa7f8d979de8
+  metadata.gz: cfde2027b6fb73eee85ed0c612db51df707c2b01bec2266a3f2049acc34990f1
+  data.tar.gz: 5b1bb3263af1cbdde2bb84cf4d92f2e84ad7e0a0b3eaa5ad67f048750c062c46
 SHA512:
-  metadata.gz: f2048376477bd1af31a63b75d17638df999c9f6341f967b2c826177add2a13831a54a72729e2a5c62aa09fa4aca6dccbaba308c79d9adcd470808d3bb80cde59
-  data.tar.gz: 2e0ac5fc90f8a2359f35a754034bb6474c1169ac8c8f3301257555553e1aca4f708cecd4d9245ff835cbe4396561a89d3bd3700ef6cea6c92c343a1e41ca2055
+  metadata.gz: 2b563e322f2eaff18eeb07f786d6277290c032e94859d89d856a7bc2512a278424ab5ec9f94f8bf477a222e9ef00075ad2e0293b29dbb0021a5b6551f836b3c9
+  data.tar.gz: 787b04cc62a139289784a9b1992a152ca74fafa3d97afa68944dea95121ff6d3af80164b551b5430d50b263f0d0c8f702d22c1450faaaebdbc4bc69f09c331fe

data/README.md

@@ -1,93 +1,214 @@
+# dogstatsd-ruby
 
-dogstatsd-ruby
-==============
+A client for DogStatsD, an extension of the StatsD metric server for Datadog. Full API documentation is available in [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
 
-A client for DogStatsD, an extension of the StatsD metric server for Datadog.
+[![Build Status](https://secure.travis-ci.org/DataDog/dogstatsd-ruby.svg)](http://travis-ci.org/DataDog/dogstatsd-ruby)
 
-[![Build Status](https://secure.travis-ci.org/DataDog/dogstatsd-ruby.png)](http://travis-ci.org/DataDog/dogstatsd-ruby)
+See [CHANGELOG.md](CHANGELOG.md) for changes. To suggest a feature, report a bug, or general discussion, [open an issue](http://github.com/DataDog/dogstatsd-ruby/issues/).
 
-Quick Start Guide
------------------
+## Installation
 
 First install the library:
 
-    gem install dogstatsd-ruby
+```
+gem install dogstatsd-ruby
+```
 
-Then start instrumenting your code:
+## Configuration
 
-``` ruby
-# Load the dogstats module.
+To instantiate a DogStatsd client:
+
+```ruby
+# Import the library
 require 'datadog/statsd'
 
-# Create a stats instance.
+# Create a DogStatsD client instance
 statsd = Datadog::Statsd.new('localhost', 8125)
+# ...
+# release resources used by the client instance
+statsd.close()
+```
+Or if you want to connect over Unix Domain Socket:
+```ruby
+# Connection over Unix Domain Socket
+statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
+# ...
+# release resources used by the client instance
+statsd.close()
+```
+
+Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?code-lang=ruby#client-instantiation-parameters).
+
+### Migrating from v4.x to v5.x
 
-# you could also create a statsd class if you need a drop in replacement
-# class Statsd < Datadog::Statsd
-# end
+If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
+change concerning you is the new [threading model](#threading-model):
 
-# Increment a counter.
-statsd.increment('page.views')
+In practice, it means two things:
 
-# Record a gauge 50% of the time.
-statsd.gauge('users.online', 123, :sample_rate=>0.5)
+1. Now that the client is buffering metrics before sending them, you have to call `Datadog::Statsd#flush(sync: true)` if you want synchronous behavior. In most cases, this is not needed, as the sender thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
 
-# Sample a histogram
-statsd.histogram('file.upload.size', 1234)
+2. You have to make sure you are either:
 
-# Time a block of code
-statsd.time('page.render') do
-  render_page('home.html')
-end
+  * Using a singleton instance of the DogStatsD client instead of creating a new instance whenever you need one; this will let the buffering mechanism flush metrics regularly
+  * Or properly disposing of the DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close`
 
-# Send several metrics at the same time
-# All metrics will be buffered and sent in one packet when the block completes
-statsd.batch do |s|
-  s.increment('page.views')
-  s.gauge('users.online', 123)
-end
+If you have issues with the sender thread or the buffering mode, you can instantiate a client that behaves exactly as in v4.x (i.e. no sender thread and flush on every metric submission):
 
-# Tag a metric.
-statsd.histogram('query.time', 10, :tags => ["version:1"])
+```ruby
+# Create a DogStatsD client instance using UDP
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true, buffer_max_pool_size: 1)
+# ...
+statsd.close()
 ```
 
-You can also post events to your stream. You can tag them, set priority and even aggregate them with other events.
+or
+
+```ruby
+# Create a DogStatsD client instance using UDS
+statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file', single_thread: true, buffer_max_pool_size: 1)
+# ...
+statsd.close()
+```
+
+### v5.x Common Pitfalls
+
+Version v5.x of `dogstatsd-ruby` is using a sender thread for flushing. This provides better performance, but you need to consider the following pitfalls:
+
+1. Applications that use `fork` after having created the dogstatsd instance: the child process will automatically spawn a new sender thread to flush metrics.
+
+2. Applications that create multiple instances of the client without closing them: it is important to `#close` all instances to free the thread and the socket they are using otherwise you will leak those resources.
 
-Aggregation in the stream is made on hostname/event_type/source_type/aggregation_key.
+If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
 
-``` ruby
-# Post a simple message
-statsd.event("There might be a storm tomorrow", "A friend warned me earlier.")
+If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
 
-# Cry for help
-statsd.event("SO MUCH SNOW", "Started yesterday and it won't stop !!", :alert_type => "error", :tags => ["urgent", "endoftheworld"])
+Applications that run into issues but can't apply these recommendations should use the `single_thread` mode which disables the use of the sender thread.
+Here is how to instantiate a client in this mode:
+
+```ruby
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true)
+# ...
+# release resources used by the client instance and flush last metrics
+statsd.close()
+```
+
+### Origin detection over UDP
+
+Origin detection is a method to detect which pod DogStatsD packets are coming from, in order to add the pod's tags to the tag list.
+
+To enable origin detection over UDP, add the following lines to your application manifest:
+
+```yaml
+env:
+  - name: DD_ENTITY_ID
+    valueFrom:
+      fieldRef:
+        fieldPath: metadata.uid
 ```
 
+The DogStatsD client attaches an internal tag, `entity_id`. The value of this tag is the content of the `DD_ENTITY_ID` environment variable, which is the pod’s UID.
+
+## Usage
+
+In order to use DogStatsD metrics, events, and Service Checks the Datadog Agent must be [running and available](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby).
+
+### Metrics
+
+After the client is created, you can start sending custom metrics to Datadog. See the dedicated [Metric Submission: DogStatsD documentation](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby) to see how to submit all supported metric types to Datadog with working code examples:
+
+* [Submit a COUNT metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#count).
+* [Submit a GAUGE metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#gauge).
+* [Submit a SET metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#set)
+* [Submit a HISTOGRAM metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#histogram)
+* [Submit a DISTRIBUTION metric](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?code-lang=ruby#distribution)
+
+Some options are suppported when submitting metrics, like [applying a Sample Rate to your metrics](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-submission-options) or [tagging your metrics with your custom tags](https://docs.datadoghq.com/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-tagging). Find all the available functions to report metrics in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
+
+### Events
+
+After the client is created, you can start sending events to your Datadog Event Stream. See the dedicated [Event Submission: DogStatsD documentation](https://docs.datadoghq.com/events/guides/dogstatsd/?code-lang=ruby) to see how to submit an event to Datadog your Event Stream.
+
+### Service Checks
+
+After the client is created, you can start sending Service Checks to Datadog. See the dedicated [Service Check Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/?tab=ruby) to see how to submit a Service Check to Datadog.
+
+### Maximum packet size in high-throughput scenarios
+
+In order to have the most efficient use of this library in high-throughput scenarios,
+recommended values for the maximum packet size have already been set for both UDS (8192 bytes)
+and UDP (1432 bytes).
+
+However, if are in control of your network and want to use a different value for the maximum packet
+size, you can do it by setting the `buffer_max_payload_size` parameter:
+
+```ruby
+statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
+# ...
+statsd.close()
+```
+
+## Threading model
+
+Starting with version 5.0, `dogstatsd-ruby` employs a new threading model where one instance of `Datadog::Statsd` can be shared between threads and where data sending is non-blocking (asynchronous).
+
+When you instantiate a `Datadog::Statsd`, a sender thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class. You can make use of `single_thread: true` to disable this behavior.
+
+This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). Instantiating a lot of statsd clients without calling `#close` after they are not needed anymore will most likely lead to threads being leaked.
+
+The sender thread has the following logic (from `Datadog::Statsd::Sender#send_loop`):
+
+```
+while the sender message queue is not closed do
+  read message from sender message queue
+
+  if message is a Control message to flush
+    flush buffer in connection
+  else if message is a Control message to synchronize
+    synchronize with calling thread
+  else
+    add message to the buffer
+  end
+end while
+```
+
+There are three different kinds of messages:
+
+1. a control message to flush the buffer in the connection
+2. a control message to synchronize any thread with the sender thread
+3. a message to append to the buffer
+
+There is also an implicit message which closes the queue which will cause the sender thread to finish processing and exit.
+
+### Usual workflow
+
+You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
+
+### Flushing
+
+When calling `Datadog::Statsd#flush`, a specific control message (`:flush`) is sent to the sender thread. When the sender thread receives it, it flushes its internal buffer into the connection.
 
+### Rendez-vous
 
-Documentation
--------------
+It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronous flush using `Datadog::Statsd#flush(sync: true)`.
 
-Full API documentation is available
-[here](http://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/frames).
+Doing so means the caller thread is blocked and waiting until the data has been flushed by the sender thread.
 
+This is useful when preparing to exit the application or when checking unit tests.
 
-Feedback
---------
+### Thread-safety
 
-To suggest a feature, report a bug, or general discussion, head over
-[here](http://github.com/DataDog/dogstatsd-ruby/issues/).
+By default, instances of `Datadog::Statsd` are thread-safe and we recommend that a single instance be reused by all application threads (even in applications that employ forking). The sole exception is the `#close` method — this method is not yet thread safe (work in progress here [#209](https://github.com/DataDog/dogstatsd-ruby/pull/209)).
 
+When using the `single_thread: true` mode, instances of `Datadog::Statsd` are still thread-safe, but you may run into contention on heavily-threaded applications, so we don’t recommend (for performance reasons) reusing these instances.
 
-[Change Log](CHANGELOG.md)
-----------------------------
+## Versioning
 
+This Ruby gem is using [Semantic Versioning](https://guides.rubygems.org/patterns/#semantic-versioning) but please note that supported Ruby versions can change in a minor release of this library. As much as possible, we will add a "future deprecation" message in the minor release preceding the one dropping the support.
 
-Credits
--------
+## Credits
 
-dogstatsd-ruby is forked from Rien Henrichs [original Statsd
-client](https://github.com/reinh/statsd).
+dogstatsd-ruby is forked from Rein Henrichs' [original Statsd client](https://github.com/reinh/statsd).
 
 Copyright (c) 2011 Rein Henrichs. See LICENSE.txt for
 further details.

data/lib/datadog/statsd/connection.rb

@@ -0,0 +1,60 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class Connection
+      def initialize(telemetry: nil, logger: nil)
+        @telemetry = telemetry
+        @logger = logger
+      end
+
+      def reset_telemetry
+        telemetry.reset
+      end
+
+      # not thread safe: `Sender` instances that use this are required to properly synchronize or sequence calls to this method
+      def write(payload)
+        logger.debug { "Statsd: #{payload}" } if logger
+
+        send_message(payload)
+
+        telemetry.sent(packets: 1, bytes: payload.length) if telemetry
+
+        true
+      rescue StandardError => boom
+        # Try once to reconnect if the socket has been closed
+        retries ||= 1
+        if retries <= 1 &&
+          (boom.is_a?(Errno::ENOTCONN) or
+           boom.is_a?(Errno::ECONNREFUSED) or
+           boom.is_a?(IOError) && boom.message =~ /closed stream/i)
+          retries += 1
+          begin
+            close
+            connect
+            retry
+          rescue StandardError => e
+            boom = e
+          end
+        end
+
+        telemetry.dropped(packets: 1, bytes: payload.length) if telemetry
+        logger.error { "Statsd: #{boom.class} #{boom}" } if logger
+        nil
+      end
+
+      private
+
+      attr_reader :telemetry
+      attr_reader :logger
+
+      def connect
+        raise 'Should be implemented by subclass'
+      end
+
+      def close
+        raise 'Should be implemented by subclass'
+      end
+    end
+  end
+end

data/lib/datadog/statsd/forwarder.rb

@@ -0,0 +1,120 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class Forwarder
+      attr_reader :telemetry
+      attr_reader :transport_type
+
+      def initialize(
+        host: nil,
+        port: nil,
+        socket_path: nil,
+
+        buffer_max_payload_size: nil,
+        buffer_max_pool_size: nil,
+        buffer_overflowing_stategy: :drop,
+
+        telemetry_flush_interval: nil,
+        global_tags: [],
+
+        single_thread: false,
+
+        logger: nil
+      )
+        @transport_type = socket_path.nil? ? :udp : :uds
+
+        if telemetry_flush_interval
+          @telemetry = Telemetry.new(telemetry_flush_interval,
+            global_tags: global_tags,
+            transport_type: transport_type
+          )
+        end
+
+        @connection = case transport_type
+                      when :udp
+                        UDPConnection.new(host, port, logger: logger, telemetry: telemetry)
+                      when :uds
+                        UDSConnection.new(socket_path, logger: logger, telemetry: telemetry)
+                      end
+
+        # Initialize buffer
+        buffer_max_payload_size ||= (transport_type == :udp ? UDP_DEFAULT_BUFFER_SIZE : UDS_DEFAULT_BUFFER_SIZE)
+
+        if buffer_max_payload_size <= 0
+          raise ArgumentError, 'buffer_max_payload_size cannot be <= 0'
+        end
+
+        unless telemetry.nil? || telemetry.would_fit_in?(buffer_max_payload_size)
+          raise ArgumentError, "buffer_max_payload_size is not high enough to use telemetry (tags=(#{global_tags.inspect}))"
+        end
+
+        buffer = MessageBuffer.new(@connection,
+          max_payload_size: buffer_max_payload_size,
+          max_pool_size: buffer_max_pool_size || DEFAULT_BUFFER_POOL_SIZE,
+          overflowing_stategy: buffer_overflowing_stategy,
+        )
+        @sender = (single_thread ? SingleThreadSender : Sender).new(buffer, logger: logger)
+        @sender.start
+      end
+
+      def send_message(message)
+        sender.add(message)
+
+        tick_telemetry
+      end
+
+      def sync_with_outbound_io
+        sender.rendez_vous
+      end
+
+      def flush(flush_telemetry: false, sync: false)
+        do_flush_telemetry if telemetry && flush_telemetry
+
+        sender.flush(sync: sync)
+      end
+
+      def host
+        return nil unless transport_type == :udp
+
+        connection.host
+      end
+
+      def port
+        return nil unless transport_type == :udp
+
+        connection.port
+      end
+
+      def socket_path
+        return nil unless transport_type == :uds
+
+        connection.socket_path
+      end
+
+      def close
+        sender.stop
+        connection.close
+      end
+
+      private
+      attr_reader :sender
+      attr_reader :connection
+
+      def do_flush_telemetry
+        telemetry_snapshot = telemetry.flush
+        telemetry.reset
+
+        telemetry_snapshot.each do |message|
+          sender.add(message)
+        end
+      end
+
+      def tick_telemetry
+        return nil unless telemetry
+
+        do_flush_telemetry if telemetry.should_flush?
+      end
+    end
+  end
+end

data/lib/datadog/statsd/message_buffer.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+module Datadog
+  class Statsd
+    class MessageBuffer
+      PAYLOAD_SIZE_TOLERANCE = 0.05
+
+      def initialize(connection,
+        max_payload_size: nil,
+        max_pool_size: DEFAULT_BUFFER_POOL_SIZE,
+        overflowing_stategy: :drop
+      )
+        raise ArgumentError, 'max_payload_size keyword argument must be provided' unless max_payload_size
+        raise ArgumentError, 'max_pool_size keyword argument must be provided' unless max_pool_size
+
+        @connection = connection
+        @max_payload_size = max_payload_size
+        @max_pool_size = max_pool_size
+        @overflowing_stategy = overflowing_stategy
+
+        @buffer = String.new
+        clear_buffer
+      end
+
+      def add(message)
+        message_size = message.bytesize
+
+        return nil unless message_size > 0 # to avoid adding empty messages to the buffer
+        return nil unless ensure_sendable!(message_size)
+
+        flush if should_flush?(message_size)
+
+        buffer << "\n" unless buffer.empty?
+        buffer << message
+
+        @message_count += 1
+
+        # flush when we're pretty sure that we won't be able
+        # to add another message to the buffer
+        flush if preemptive_flush?
+
+        true
+      end
+
+      def reset
+        clear_buffer
+        connection.reset_telemetry
+      end
+
+      def flush
+        return if buffer.empty?
+
+        connection.write(buffer)
+        clear_buffer
+      end
+
+      private
+
+      attr :max_payload_size
+      attr :max_pool_size
+
+      attr :overflowing_stategy
+
+      attr :connection
+      attr :buffer
+
+      def should_flush?(message_size)
+        return true if buffer.bytesize + 1 + message_size >= max_payload_size
+
+        false
+      end
+
+      def clear_buffer
+        buffer.clear
+        @message_count = 0
+      end
+
+      def preemptive_flush?
+        @message_count == max_pool_size || buffer.bytesize > bytesize_threshold
+      end
+
+      def ensure_sendable!(message_size)
+        return true if message_size <= max_payload_size
+
+        if overflowing_stategy == :raise
+          raise Error, 'Message too big for payload limit'
+        end
+
+        false
+      end
+
+      def bytesize_threshold
+        @bytesize_threshold ||= (max_payload_size - PAYLOAD_SIZE_TOLERANCE * max_payload_size).to_i
+      end
+    end
+  end
+end