dogstatsd-ruby 4.0.0 → 5.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6d2b1b9e7ec2a48c5305f6acda103d0b7a09787801dd899a38fb665f5389e4f3
-  data.tar.gz: 3d974fffb4ace3bc248e69dcdaabb4c556e6783a25d65af98cb65bdccafc52ee
+  metadata.gz: 9abfaabd9be58a320dd8cc8d1d4998ac287daa08e68ab894054cf2910f5f3133
+  data.tar.gz: 88ebf4ef472f7663897c06fa3e5753a17c12722bc1a58f8aa4e9fb0a3ace07c6
 SHA512:
-  metadata.gz: 16c82cb62bfd324d5e1dc1c6293fa121b92bd0c88d00474210c5da67f2cb07ae350cc0b717b15739c7a6f09e9021323ba348cec70ec2d0ac66a362f15f81007b
-  data.tar.gz: 6e82cef56d746ecf7b10890d397e313c143611b18b4ae65db67eb2f252cb27acacb45de03f6447b7b4ebb1666209806d6b048fb2857c5c8790f1f6729f4ae806
+  metadata.gz: 199527bd5e9d39d94be3cabf28dc5f4d8913c4d3f746a6018a22c9ed13440c34d77fc11b363f47f9e2fd325974005fdc3567a223de4669b388ced99c3cd0750f
+  data.tar.gz: 0da25bafef540e4d36be8cae9c1fbb1f11c6aea94415828941aa9c34877db75166574f41b5e1a0829c71285f958162133d0f592a34619ce066e14c97ea86fd4d

data/README.md

@@ -1,97 +1,217 @@
+# dogstatsd-ruby
 
-dogstatsd-ruby
-==============
+A client for DogStatsD, an extension of the StatsD metric server for Datadog. Full API documentation is available in [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
 
-A client for DogStatsD, an extension of the StatsD metric server for Datadog.
+[![Build Status](https://secure.travis-ci.org/DataDog/dogstatsd-ruby.svg)](http://travis-ci.org/DataDog/dogstatsd-ruby)
 
-[![Build Status](https://secure.travis-ci.org/DataDog/dogstatsd-ruby.png)](http://travis-ci.org/DataDog/dogstatsd-ruby)
+See [CHANGELOG.md](CHANGELOG.md) for changes. To suggest a feature, report a bug, or general discussion, [open an issue](http://github.com/DataDog/dogstatsd-ruby/issues/).
 
-Quick Start Guide
------------------
+## Installation
 
 First install the library:
 
-    gem install dogstatsd-ruby
+```
+gem install dogstatsd-ruby
+```
+
+## Configuration
 
-Then start instrumenting your code:
+To instantiate a DogStatsd client:
 
-``` ruby
-# Load the dogstats module.
+```ruby
+# Import the library
 require 'datadog/statsd'
 
-# Create a stats instance.
+# Create a DogStatsD client instance.
 statsd = Datadog::Statsd.new('localhost', 8125)
+...
+# release resources used by the client instance
+statsd.close()
+```
+Or if you want to connect over Unix Domain Socket:
+```ruby
+# Connection over Unix Domain Socket
+statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
+...
+# release resources used by the client instance
+statsd.close()
+```
+
+Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby#client-instantiation-parameters).
+
+### Migrating from v4.x to v5.x
+
+If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
+change concerning you is the new threading model (please see section Threading model):
+
+In practice, it means two things:
+
+1. Now that the client is buffering metrics before sending them, you have to manually
+call the method `Datadog::Statsd#flush` if you want to force the sending of metrics. Note that the companion thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
+
+2. You have to make sure you are either:
+
+  * using singletons instances of the DogStatsD client and not allocating one each time you need one, letting the buffering mechanism flush metrics, it's still a bad solution if the process later forks (see related section below). Or,
+  * properly closing your DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close` to release the resources used by the instance and to close the socket
+
+If you have issues with the companion thread or the buffering mode, you can instantiate a client that behaves exactly as in v4.x (i.e. no companion thread and flush on every metric submission):
+
+```ruby
+# Import the library
+require 'datadog/statsd'
+
+# Create a DogStatsD client instance using UDP
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true, buffer_max_payload_size: 1)
+...
+# to close the instance is not necessary in this case since metrics are flushed on submission
+# but it is still a good practice and it explicitely closes the socket
+statsd.close()
+```
+
+or
+
+```ruby
+# Import the library
+require 'datadog/statsd'
+
+# Create a DogStatsD client instance using UDS
+statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file', single_thread: true, buffer_max_payload_size: 1)
+...
+# to close the instance is not necessary in this case since metrics are flushed on submission
+# but it is still a good practice and it explicitely closes the socket
+statsd.close()
+```
+
+### v5.x Common Pitfalls
 
-# you could also create a statsd class if you need a drop in replacement
-# class Statsd < Datadog::Statsd
-# end
+Version v5.x of `dogstatsd-ruby` is using a companion thread for preemptive flushing, it brings better performances for application having a high-throughput of statsd metrics, but it comes with new pitfalls:
 
-# Increment a counter.
-statsd.increment('page.views')
+    * Applications forking after having created the dogstatsd instance: forking a process can't duplicate the existing threads, meaning that one of the processes won't have a companion thread to flush the metrics and will lead to missing metrics.
+    * Applications creating a lot of different instances of the client without closing them: it is important to close the instance to free the thread and the socket it is using or it will lead to thread leaks.
 
-# Record a gauge 50% of the time.
-statsd.gauge('users.online', 123, :sample_rate=>0.5)
+If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
 
-# Sample a histogram
-statsd.histogram('file.upload.size', 1234)
+If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
 
-# Time a block of code
-statsd.time('page.render') do
-  render_page('home.html')
-end
+Applications that are in these situations but can't apply these recommendations should enable the `single_thread` mode which does not use a companion thread. Here is how to instantiate a client in this mode:
 
-# Send several metrics at the same time
-# All metrics will be buffered and sent in one packet when the block completes
-statsd.batch do |s|
-  s.increment('page.views')
-  s.gauge('users.online', 123)
-end
+```ruby
+# Import the library
+require 'datadog/statsd'
+
+# Create a DogStatsD client instance.
+statsd = Datadog::Statsd.new('localhost', 8125, single_thread: true)
+...
+# release resources used by the client instance and flush last metrics
+statsd.close()
+```
 
-# Tag a metric.
-statsd.histogram('query.time', 10, :tags => ["version:1"])
+### Origin detection over UDP
 
-# Auto-close socket after end of block
-Datadog::Statsd.open('localhost', 8125) do |s|
-  s.increment('page.views')
-end
+Origin detection is a method to detect which pod DogStatsD packets are coming from in order to add the pod's tags to the tag list.
+
+To enable origin detection over UDP, add the following lines to your application manifest
+```yaml
+env:
+  - name: DD_ENTITY_ID
+    valueFrom:
+      fieldRef:
+        fieldPath: metadata.uid
 ```
+The DogStatsD client attaches an internal tag, `entity_id`. The value of this tag is the content of the `DD_ENTITY_ID` environment variable, which is the pod’s UID.
+
+## Usage
+
+In order to use DogStatsD metrics, events, and Service Checks the Agent must be [running and available](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby).
+
+### Metrics
+
+After the client is created, you can start sending custom metrics to Datadog. See the dedicated [Metric Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby) to see how to submit all supported metric types to Datadog with working code examples:
+
+* [Submit a COUNT metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#count).
+* [Submit a GAUGE metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#gauge).
+* [Submit a SET metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#set)
+* [Submit a HISTOGRAM metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#histogram)
+* [Submit a DISTRIBUTION metric](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#distribution)
+
+Some options are suppported when submitting metrics, like [applying a Sample Rate to your metrics](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-submission-options) or [tagging your metrics with your custom tags](https://docs.datadoghq.com/developers/metrics/dogstatsd_metrics_submission/?tab=ruby#metric-tagging). Find all the available functions to report metrics in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd).
+
+### Events
+
+After the client is created, you can start sending events to your Datadog Event Stream. See the dedicated [Event Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/events/dogstatsd/?tab=ruby) to see how to submit an event to Datadog your Event Stream.
 
-You can also post events to your stream. You can tag them, set priority and even aggregate them with other events.
+### Service Checks
 
-Aggregation in the stream is made on hostname/event_type/source_type/aggregation_key.
+After the client is created, you can start sending Service Checks to Datadog. See the dedicated [Service Check Submission: DogStatsD documentation](https://docs.datadoghq.com/developers/service_checks/dogstatsd_service_checks_submission/?tab=ruby) to see how to submit a Service Check to Datadog.
 
-``` ruby
-# Post a simple message
-statsd.event("There might be a storm tomorrow", "A friend warned me earlier.")
+### Maximum packets size in high-throughput scenarios
 
-# Cry for help
-statsd.event("SO MUCH SNOW", "Started yesterday and it won't stop !!", :alert_type => "error", :tags => ["urgent", "endoftheworld"])
+In order to have the most efficient use of this library in high-throughput scenarios,
+default values for the maximum packets size have already been set for both UDS (8192 bytes)
+and UDP (1432 bytes) in order to have the best usage of the underlying network.
+However, if you perfectly know your network and you know that a different value for the maximum packets
+size should be used, you can set it with the parameter `buffer_max_payload_size`. Example:
+
+```ruby
+# Create a DogStatsD client instance.
+statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
 ```
 
+## Threading model
+
+On versions greater than 5.0, we changed the threading model of the library so that one instance of `Datadog::Statsd` could be shared between threads and so that the writes in the socket are non blocking.
+
+When you instantiate a `Datadog::Statsd`, a companion thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class. Please use `single_thread: true` while creating an instance if you don't want to or can't use a companion thread.
+
+This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). It also means that allocating a lot of statsd clients without closing them properly when not used anymore
+could lead to a thread leak (even though they will be sleeping, blocked on IO).
+The communication between the current thread is managed through a standard Ruby Queue.
+
+The sender thread has the following logic (Code present in the method `Datadog::Statsd::Sender#send_loop`):
+
+```
+while the sender message queue is not closed do
+  read message from sender message queue
+
+  if message is a Control message to flush
+    flush buffer in connection
+  else if message is a Control message to synchronize
+    synchronize with calling thread
+  else
+    add message to the buffer
+  end
+end while
+```
+
+Most of the time, the sender thread is blocked and sleeping when doing a blocking read from the sender message queue.
+
+We can see that there is 3 different kind of messages:
+
+* a control message to flush the buffer in the connection
+* a control message to synchronize any thread with the sender thread
+* a message to append to the buffer
 
+There is also an implicit message which is closing the queue as it will stop blocking read from the message queue (if happening) and thus, stop the sender thread.
 
-Documentation
--------------
+### Usual workflow
 
-Full API documentation is available
-[here](http://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/frames).
+You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
 
+### Flushing
 
-Feedback
---------
+When calling a flush, a specific control message (the `:flush` symbol) is sent to the sender thread. When finding it, it flushes its internal buffer into the connection.
 
-To suggest a feature, report a bug, or general discussion, head over
-[here](http://github.com/DataDog/dogstatsd-ruby/issues/).
+### Rendez-vous
 
+It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronized flush (calling `Datadog::Statsd#flush` with the `sync: true` option).
 
-[Change Log](CHANGELOG.md)
-----------------------------
+This means the current thread is going to sleep and wait for a Queue which is given to the sender thread. When the sender thread reads this queue from its own message queue, it puts a placeholder message in it so that it wakes up the calling thread.
 
+This is useful when closing the application or when checking unit tests.
 
-Credits
--------
+## Credits
 
-dogstatsd-ruby is forked from Rien Henrichs [original Statsd
+dogstatsd-ruby is forked from Rein Henrichs [original Statsd
 client](https://github.com/reinh/statsd).
 
 Copyright (c) 2011 Rein Henrichs. See LICENSE.txt for

data/lib/datadog/statsd.rb

@@ -1,6 +1,16 @@
 # frozen_string_literal: true
 require 'socket'
 
+require_relative 'statsd/version'
+require_relative 'statsd/telemetry'
+require_relative 'statsd/udp_connection'
+require_relative 'statsd/uds_connection'
+require_relative 'statsd/message_buffer'
+require_relative 'statsd/serialization'
+require_relative 'statsd/sender'
+require_relative 'statsd/single_thread_sender'
+require_relative 'statsd/forwarder'
+
 # = Datadog::Statsd: A DogStatsd client (https://www.datadoghq.com)
 #
 # @example Set up a global Statsd client for a server on localhost:8125
@@ -19,216 +29,101 @@ require 'socket'
 #   statsd = Datadog::Statsd.new 'localhost', 8125, tags: 'tag1:true'
 module Datadog
   class Statsd
-
-    class Connection
-      DEFAULT_HOST = '127.0.0.1'
-      DEFAULT_PORT = 8125
-
-      # StatsD host. Defaults to 127.0.0.1.
-      attr_reader :host
-
-      # StatsD port. Defaults to 8125.
-      attr_reader :port
-
-      # DogStatsd unix socket path. Not used by default.
-      attr_reader :socket_path
-
-      def initialize(host, port, socket_path, logger)
-        @host = host || DEFAULT_HOST
-        @port = port || DEFAULT_PORT
-        @socket_path = socket_path
-        @logger = logger
-      end
-
-      def write(message)
-        @logger.debug { "Statsd: #{message}" } if @logger
-        if @socket_path.nil?
-          socket.send(message, 0)
-        else
-          socket.sendmsg_nonblock(message)
-        end
-      rescue StandardError => boom
-        # Give up on this socket if it looks like it is bad
-        bad_socket = !@socket_path.nil? && (
-          boom.is_a?(Errno::ECONNREFUSED) ||
-          boom.is_a?(Errno::ECONNRESET) ||
-          boom.is_a?(Errno::ENOENT)
-        )
-        if bad_socket
-          @socket = nil
-          return
-        end
-
-        # Try once to reconnect if the socket has been closed
-        retries ||= 1
-        if retries <= 1 && boom.is_a?(IOError) && boom.message =~ /closed stream/i
-          retries += 1
-          begin
-            @socket = connect
-            retry
-          rescue StandardError => e
-            boom = e
-          end
-        end
-
-        @logger.error { "Statsd: #{boom.class} #{boom}" } if @logger
-        nil
-      end
-
-      # Close the underlying socket
-      def close
-        @socket && @socket.close
-      end
-
-      private
-
-      def socket
-        @socket ||= connect
-      end
-
-      def connect
-        if @socket_path.nil?
-          socket = UDPSocket.new
-          socket.connect(@host, @port)
-        else
-          socket = Socket.new(Socket::AF_UNIX, Socket::SOCK_DGRAM)
-          socket.connect(Socket.pack_sockaddr_un(@socket_path))
-        end
-        socket
-      end
+    class Error < StandardError
     end
 
-    class Batch
-      def initialize(connection, max_buffer_bytes)
-        @connection = connection
-        @max_buffer_bytes = max_buffer_bytes
-        @depth = 0
-        reset
-      end
-
-      def open
-        @depth += 1
-        yield
-      ensure
-        @depth -= 1
-        flush if !open?
-      end
-
-      def open?
-        @depth > 0
-      end
-
-      def add(message)
-        message_bytes = message.bytesize
-
-        unless @buffer_bytes == 0
-          if @buffer_bytes + 1 + message_bytes >= @max_buffer_bytes
-            flush
-          else
-            @buffer << NEW_LINE
-            @buffer_bytes += 1
-          end
-        end
-
-        @buffer << message
-        @buffer_bytes += message_bytes
-      end
-
-      def flush
-        return if @buffer_bytes == 0
-        @connection.write @buffer
-        reset
-      end
-
-      private
-
-      def reset
-        @buffer = String.new
-        @buffer_bytes = 0
-      end
-    end
-
-    # Create a dictionary to assign a key to every parameter's name, except for tags (treated differently)
-    # Goal: Simple and fast to add some other parameters
-    OPTS_KEYS = {
-      :date_happened     => :d,
-      :hostname          => :h,
-      :aggregation_key   => :k,
-      :priority          => :p,
-      :source_type_name  => :s,
-      :alert_type        => :t,
-    }
-
-    # Service check options
-    SC_OPT_KEYS = {
-      :timestamp  => 'd:'.freeze,
-      :hostname   => 'h:'.freeze,
-      :tags       => '#'.freeze,
-      :message    => 'm:'.freeze,
-    }
-
-    OK        = 0
-    WARNING   = 1
-    CRITICAL  = 2
-    UNKNOWN   = 3
-
-    MAX_EVENT_SIZE = 8 * 1024
-
-    COUNTER_TYPE = 'c'.freeze
-    GAUGE_TYPE = 'g'.freeze
-    HISTOGRAM_TYPE = 'h'.freeze
-    DISTRIBUTION_TYPE = 'd'.freeze
-    TIMING_TYPE = 'ms'.freeze
-    SET_TYPE = 's'.freeze
-    VERSION = "4.0.0".freeze
+    OK       = 0
+    WARNING  = 1
+    CRITICAL = 2
+    UNKNOWN  = 3
+
+    UDP_DEFAULT_BUFFER_SIZE = 1_432
+    UDS_DEFAULT_BUFFER_SIZE = 8_192
+    DEFAULT_BUFFER_POOL_SIZE = Float::INFINITY
+    MAX_EVENT_SIZE = 8 * 1_024
+    # minimum flush interval for the telemetry in seconds
+    DEFAULT_TELEMETRY_FLUSH_INTERVAL = 10
+
+    COUNTER_TYPE = 'c'
+    GAUGE_TYPE = 'g'
+    HISTOGRAM_TYPE = 'h'
+    DISTRIBUTION_TYPE = 'd'
+    TIMING_TYPE = 'ms'
+    SET_TYPE = 's'
 
     # A namespace to prepend to all statsd calls. Defaults to no namespace.
     attr_reader :namespace
 
     # Global tags to be added to every statsd call. Defaults to no tags.
-    attr_reader :tags
-
-    # Buffer containing the statsd message before they are sent in batch
-    attr_reader :buffer
-
-    # Maximum buffer size in bytes before it is flushed
-    attr_reader :max_buffer_bytes
+    def tags
+      serializer.global_tags
+    end
 
-    # Connection
-    attr_reader :connection
+    # Default sample rate
+    attr_reader :sample_rate
 
     # @param [String] host your statsd host
     # @param [Integer] port your statsd port
     # @option [String] namespace set a namespace to be prepended to every metric name
-    # @option [Array<String>] tags tags to be added to every metric
-    # @option [Loger] logger for debugging
-    # @option [Integer] max_buffer_bytes max bytes to buffer when using #batch
+    # @option [Array<String>|Hash] tags tags to be added to every metric
+    # @option [Logger] logger for debugging
+    # @option [Integer] buffer_max_payload_size max bytes to buffer
+    # @option [Integer] buffer_max_pool_size max messages to buffer
     # @option [String] socket_path unix socket path
+    # @option [Float] default sample rate if not overridden
+    # @option [Boolean] single_thread flushes the metrics on the main thread instead of in a companion thread
     def initialize(
       host = nil,
       port = nil,
+      socket_path: nil,
+
       namespace: nil,
       tags: nil,
-      max_buffer_bytes: 8192,
-      socket_path: nil,
-      logger: nil
+      sample_rate: nil,
+
+      buffer_max_payload_size: nil,
+      buffer_max_pool_size: nil,
+      buffer_overflowing_stategy: :drop,
+
+      logger: nil,
+
+      single_thread: false,
+
+      telemetry_enable: true,
+      telemetry_flush_interval: DEFAULT_TELEMETRY_FLUSH_INTERVAL
     )
-      @connection = Connection.new(host, port, socket_path, logger)
-      @logger = logger
+      unless tags.nil? || tags.is_a?(Array) || tags.is_a?(Hash)
+        raise ArgumentError, 'tags must be an array of string tags or a Hash'
+      end
 
       @namespace = namespace
       @prefix = @namespace ? "#{@namespace}.".freeze : nil
+      @serializer = Serialization::Serializer.new(prefix: @prefix, global_tags: tags)
+      @sample_rate = sample_rate
+
+      @forwarder = Forwarder.new(
+        host: host,
+        port: port,
+        socket_path: socket_path,
 
-      raise ArgumentError, 'tags must be a Array<String>' unless tags.nil? or tags.is_a? Array
-      @tags = (tags || []).compact.map! {|tag| escape_tag_content(tag)}
+        global_tags: tags,
+        logger: logger,
 
-      @batch = Batch.new @connection, max_buffer_bytes
+        single_thread: single_thread,
+
+        buffer_max_payload_size: buffer_max_payload_size,
+        buffer_max_pool_size: buffer_max_pool_size,
+        buffer_overflowing_stategy: buffer_overflowing_stategy,
+
+        telemetry_flush_interval: telemetry_enable ? telemetry_flush_interval : nil,
+      )
     end
 
     # yield a new instance to a block and close it when done
     # for short-term use-cases that don't want to close the socket manually
     def self.open(*args)
       instance = new(*args)
+
       yield instance
     ensure
       instance.close
@@ -242,10 +137,10 @@ module Datadog
     # @option opts [Array<String>] :tags An array of tags
     # @option opts [Numeric] :by increment value, default 1
     # @see #count
-    def increment(stat, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
+    def increment(stat, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
       incr_value = opts.fetch(:by, 1)
-      count stat, incr_value, opts
+      count(stat, incr_value, opts)
     end
 
     # Sends a decrement (count = -1) for the given stat to the statsd server.
@@ -256,10 +151,10 @@ module Datadog
     # @option opts [Array<String>] :tags An array of tags
     # @option opts [Numeric] :by decrement value, default 1
     # @see #count
-    def decrement(stat, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
+    def decrement(stat, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
       decr_value = - opts.fetch(:by, 1)
-      count stat, decr_value, opts
+      count(stat, decr_value, opts)
     end
 
     # Sends an arbitrary count for the given stat to the statsd server.
@@ -269,9 +164,9 @@ module Datadog
     # @param [Hash] opts the options to create the metric with
     # @option opts [Numeric] :sample_rate sample rate, 1 for always
     # @option opts [Array<String>] :tags An array of tags
-    def count(stat, count, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
-      send_stats stat, count, COUNTER_TYPE, opts
+    def count(stat, count, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
+      send_stats(stat, count, COUNTER_TYPE, opts)
     end
 
     # Sends an arbitary gauge value for the given stat to the statsd server.
@@ -287,9 +182,9 @@ module Datadog
     # @option opts [Array<String>] :tags An array of tags
     # @example Report the current user count:
     #   $statsd.gauge('user.count', User.count)
-    def gauge(stat, value, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
-      send_stats stat, value, GAUGE_TYPE, opts
+    def gauge(stat, value, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
+      send_stats(stat, value, GAUGE_TYPE, opts)
     end
 
     # Sends a value to be tracked as a histogram to the statsd server.
@@ -301,14 +196,11 @@ module Datadog
     # @option opts [Array<String>] :tags An array of tags
     # @example Report the current user count:
     #   $statsd.histogram('user.count', User.count)
-    def histogram(stat, value, opts=EMPTY_OPTIONS)
-      send_stats stat, value, HISTOGRAM_TYPE, opts
+    def histogram(stat, value, opts = EMPTY_OPTIONS)
+      send_stats(stat, value, HISTOGRAM_TYPE, opts)
     end
 
     # Sends a value to be tracked as a distribution to the statsd server.
-    # Note: Distributions are a beta feature of Datadog and not generally
-    # available. Distributions must be specifically enabled for your
-    # organization.
     #
     # @param [String] stat stat name.
     # @param [Numeric] value distribution value.
@@ -317,8 +209,8 @@ module Datadog
     # @option opts [Array<String>] :tags An array of tags
     # @example Report the current user count:
     #   $statsd.distribution('user.count', User.count)
-    def distribution(stat, value, opts=EMPTY_OPTIONS)
-      send_stats stat, value, DISTRIBUTION_TYPE, opts
+    def distribution(stat, value, opts = EMPTY_OPTIONS)
+      send_stats(stat, value, DISTRIBUTION_TYPE, opts)
     end
 
     # Sends a timing (in ms) for the given stat to the statsd server. The
@@ -331,9 +223,9 @@ module Datadog
     # @param [Hash] opts the options to create the metric with
     # @option opts [Numeric] :sample_rate sample rate, 1 for always
     # @option opts [Array<String>] :tags An array of tags
-    def timing(stat, ms, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
-      send_stats stat, ms, TIMING_TYPE, opts
+    def timing(stat, ms, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
+      send_stats(stat, ms, TIMING_TYPE, opts)
     end
 
     # Reports execution time of the provided block using {#timing}.
@@ -349,13 +241,12 @@ module Datadog
     # @see #timing
     # @example Report the time (in ms) taken to activate an account
     #   $statsd.time('account.activate') { @account.activate! }
-    def time(stat, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
-      start = (PROCESS_TIME_SUPPORTED ? Process.clock_gettime(Process::CLOCK_MONOTONIC) : Time.now.to_f)
-      return yield
+    def time(stat, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
+      start = now
+      yield
     ensure
-      finished = (PROCESS_TIME_SUPPORTED ? Process.clock_gettime(Process::CLOCK_MONOTONIC) : Time.now.to_f)
-      timing(stat, ((finished - start) * 1000).round, opts)
+      timing(stat, ((now - start) * 1000).round, opts)
     end
 
     # Sends a value to be tracked as a set to the statsd server.
@@ -367,9 +258,9 @@ module Datadog
     # @option opts [Array<String>] :tags An array of tags
     # @example Record a unique visitory by id:
     #   $statsd.set('visitors.uniques', User.id)
-    def set(stat, value, opts=EMPTY_OPTIONS)
-      opts = {:sample_rate => opts} if opts.is_a? Numeric
-      send_stats stat, value, SET_TYPE, opts
+    def set(stat, value, opts = EMPTY_OPTIONS)
+      opts = { sample_rate: opts } if opts.is_a?(Numeric)
+      send_stats(stat, value, SET_TYPE, opts)
     end
 
     # This method allows you to send custom service check statuses.
@@ -377,14 +268,16 @@ module Datadog
     # @param [String] name Service check name
     # @param [String] status Service check status.
     # @param [Hash] opts the additional data about the service check
-      # @option opts [Integer, nil] :timestamp (nil) Assign a timestamp to the event. Default is now when none
-      # @option opts [String, nil] :hostname (nil) Assign a hostname to the event.
+      # @option opts [Integer, String, nil] :timestamp (nil) Assign a timestamp to the service check. Default is now when none
+      # @option opts [String, nil] :hostname (nil) Assign a hostname to the service check.
       # @option opts [Array<String>, nil] :tags (nil) An array of tags
       # @option opts [String, nil] :message (nil) A message to associate with this service check status
     # @example Report a critical service check status
     #   $statsd.service_check('my.service.check', Statsd::CRITICAL, :tags=>['urgent'])
-    def service_check(name, status, opts=EMPTY_OPTIONS)
-      send_stat format_service_check(name, status, opts)
+    def service_check(name, status, opts = EMPTY_OPTIONS)
+      telemetry.sent(service_checks: 1) if telemetry
+
+      forwarder.send_message(serializer.to_service_check(name, status, opts))
     end
 
     # This end point allows you to post events to the stream. You can tag them, set priority and even aggregate them with other events.
@@ -394,23 +287,33 @@ module Datadog
     # it will be grouped with other events that don't have an event type.
     #
     # @param [String] title Event title
-    # @param [String] text Event text. Supports \n
+    # @param [String] text Event text. Supports newlines (+\n+)
     # @param [Hash] opts the additional data about the event
-    # @option opts [Integer, nil] :date_happened (nil) Assign a timestamp to the event. Default is now when none
+    # @option opts [Integer, String, nil] :date_happened (nil) Assign a timestamp to the event. Default is now when none
     # @option opts [String, nil] :hostname (nil) Assign a hostname to the event.
     # @option opts [String, nil] :aggregation_key (nil) Assign an aggregation key to the event, to group it with some others
     # @option opts [String, nil] :priority ('normal') Can be "normal" or "low"
     # @option opts [String, nil] :source_type_name (nil) Assign a source type to the event
     # @option opts [String, nil] :alert_type ('info') Can be "error", "warning", "info" or "success".
+    # @option opts [Boolean, false] :truncate_if_too_long (false) Truncate the event if it is too long
     # @option opts [Array<String>] :tags tags to be added to every metric
     # @example Report an awful event:
     #   $statsd.event('Something terrible happened', 'The end is near if we do nothing', :alert_type=>'warning', :tags=>['end_of_times','urgent'])
-    def event(title, text, opts=EMPTY_OPTIONS)
-      send_stat format_event(title, text, opts)
+    def event(title, text, opts = EMPTY_OPTIONS)
+      telemetry.sent(events: 1) if telemetry
+
+      forwarder.send_message(serializer.to_event(title, text, opts))
     end
 
-    # Send several metrics in the same UDP Packet
-    # They will be buffered and flushed when the block finishes
+    # Send several metrics in the same packet.
+    # They will be buffered and flushed when the block finishes.
+    #
+    # This method exists for compatibility with v4.x versions, it is not needed
+    # anymore since the batching is now automatically done internally.
+    # It also means that an automatic flush could occur if the buffer is filled
+    # during the execution of the batch block.
+    #
+    # This method is DEPRECATED and will be removed in future v6.x API.
     #
     # @example Send several metrics in one packet:
     #   $statsd.batch do |s|
@@ -418,140 +321,73 @@ module Datadog
     #      s.increment('page.views')
     #    end
     def batch
-      @batch.open { yield self }
+      yield self
+      flush(sync: true)
     end
 
     # Close the underlying socket
-    def close
-      @connection.close
+    #
+    # @param [Boolean, true] flush Should we flush the metrics before closing
+    def close(flush: true)
+      flush(sync: true) if flush
+      forwarder.close
     end
 
-    private
-
-    NEW_LINE = "\n".freeze
-    ESC_NEW_LINE = "\\n".freeze
-    COMMA = ",".freeze
-    PIPE = "|".freeze
-    DOT = ".".freeze
-    DOUBLE_COLON = "::".freeze
-    UNDERSCORE = "_".freeze
-    PROCESS_TIME_SUPPORTED = (RUBY_VERSION >= "2.1.0")
-    EMPTY_OPTIONS = {}.freeze
-
-    private_constant :NEW_LINE, :ESC_NEW_LINE, :COMMA, :PIPE, :DOT,
-      :DOUBLE_COLON, :UNDERSCORE, :EMPTY_OPTIONS
-
-    def format_service_check(name, status, opts=EMPTY_OPTIONS)
-      sc_string = "_sc|#{name}|#{status}".dup
-
-      SC_OPT_KEYS.each do |key, shorthand_key|
-        next unless opts[key]
-
-        if key == :tags
-          if tags_string = tags_as_string(opts)
-            sc_string << "|##{tags_string}"
-          end
-        elsif key == :message
-          message = remove_pipes(opts[:message])
-          escaped_message = escape_service_check_message(message)
-          sc_string << "|m:#{escaped_message}"
-        else
-          value = remove_pipes(opts[key])
-          sc_string << "|#{shorthand_key}#{value}"
-        end
-      end
-      sc_string
+    def sync_with_outbound_io
+      forwarder.sync_with_outbound_io
     end
 
-    def format_event(title, text, opts=EMPTY_OPTIONS)
-      escaped_title = escape_event_content(title)
-      escaped_text = escape_event_content(text)
-      event_string_data = "_e{#{escaped_title.length},#{escaped_text.length}}:#{escaped_title}|#{escaped_text}".dup
-
-      # We construct the string to be sent by adding '|key:value' parts to it when needed
-      # All pipes ('|') in the metadata are removed. Title and Text can keep theirs
-      OPTS_KEYS.each do |key, shorthand_key|
-        if key != :tags && opts[key]
-          value = remove_pipes(opts[key])
-          event_string_data << "|#{shorthand_key}:#{value}"
-        end
-      end
-
-      # Tags are joined and added as last part to the string to be sent
-      if tags_string = tags_as_string(opts)
-        event_string_data << "|##{tags_string}"
-      end
-
-      raise "Event #{title} payload is too big (more that 8KB), event discarded" if event_string_data.length > MAX_EVENT_SIZE
-      event_string_data
+    # Flush the buffer into the connection
+    def flush(flush_telemetry: false, sync: false)
+      forwarder.flush(flush_telemetry: flush_telemetry, sync: sync)
     end
 
-    def tags_as_string(opts)
-      if tag_arr = opts[:tags]
-        tag_arr = tag_arr.map { |tag| escape_tag_content(tag) }
-        tag_arr = tags + tag_arr # @tags are normalized when set, so not need to normalize them again
-      else
-        tag_arr = tags
-      end
-      tag_arr.join(COMMA) unless tag_arr.empty?
+    def telemetry
+      forwarder.telemetry
     end
 
-    def escape_event_content(msg)
-      msg.gsub NEW_LINE, ESC_NEW_LINE
+    def host
+      forwarder.host
     end
 
-    def escape_tag_content(tag)
-      tag = remove_pipes(tag.to_s)
-      tag.delete! COMMA
-      tag
+    def port
+      forwarder.port
     end
 
-    def remove_pipes(msg)
-      msg.delete PIPE
+    def socket_path
+      forwarder.socket_path
     end
 
-    def escape_service_check_message(msg)
-      escape_event_content(msg).gsub('m:'.freeze, 'm\:'.freeze)
+    def transport_type
+      forwarder.transport_type
     end
 
-    def send_stats(stat, delta, type, opts=EMPTY_OPTIONS)
-      sample_rate = opts[:sample_rate] || 1
-      if sample_rate == 1 or rand < sample_rate
-        full_stat = ''.dup
-        full_stat << @prefix if @prefix
-
-        stat = stat.is_a?(String) ? stat.dup : stat.to_s
-        # Replace Ruby module scoping with '.' and reserved chars (: | @) with underscores.
-        stat.gsub!(DOUBLE_COLON, DOT)
-        stat.tr!(':|@'.freeze, UNDERSCORE)
-        full_stat << stat
-
-        full_stat << ':'.freeze
-        full_stat << delta.to_s
-        full_stat << PIPE
-        full_stat << type
-
-        unless sample_rate == 1
-          full_stat << PIPE
-          full_stat << '@'.freeze
-          full_stat << sample_rate.to_s
-        end
-
-        if tags_string = tags_as_string(opts)
-          full_stat << PIPE
-          full_stat << '#'.freeze
-          full_stat << tags_string
-        end
-
-        send_stat(full_stat)
+    private
+    attr_reader :serializer
+    attr_reader :forwarder
+
+    PROCESS_TIME_SUPPORTED = (RUBY_VERSION >= '2.1.0')
+    EMPTY_OPTIONS = {}.freeze
+
+    if PROCESS_TIME_SUPPORTED
+      def now
+        Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      end
+    else
+      def now
+        Time.now.to_f
       end
     end
 
-    def send_stat(message)
-      if @batch.open?
-        @batch.add message
-      else
-        @connection.write(message)
+    def send_stats(stat, delta, type, opts = EMPTY_OPTIONS)
+      telemetry.sent(metrics: 1) if telemetry
+
+      sample_rate = opts[:sample_rate] || @sample_rate || 1
+
+      if sample_rate == 1 || rand <= sample_rate
+        full_stat = serializer.to_stat(stat, delta, type, tags: opts[:tags], sample_rate: sample_rate)
+
+        forwarder.send_message(full_stat)
       end
     end
   end