dogstatsd-ruby 5.0.1 → 5.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c77f82b5b9a858517a937a5b2db1a1f890a80c94ca63e60f12e256ab28f7192d
-  data.tar.gz: e95ee401174c084edb117068f73d1c72f08e602ea9b51363e57e53c176072472
+  metadata.gz: 264d78a169508453151e7a5c3260c2299ad05c727dc9c72cfa52f8be721809d0
+  data.tar.gz: 8e0726168a20f7efcb2bdcf72ab3d748184cd652de4fd25871f9fdf676020baf
 SHA512:
-  metadata.gz: 88d54866c8693d2dab18a1370d6ba5951a42ec4ed62615c6194548dd5faa109731840187e2a7ddaf591f08d07064c8e173df74728f182fe5cb5593c896503e19
-  data.tar.gz: d8021d0b6b21efe7ab14841665b01819c397d4419b247615d88b7685b332b1c8bb9a13d05fc22d21baf5d709d5452f9a9a3d932438addfbf1238e8ab5656a8e5
+  metadata.gz: 1875c615a043db68c69d855af2b454e45ef41af1d7e1f726a6067c35e260eb0dbc9b706a0588608e31bd5087e500b1b77b1b604f3dc22750f1376b6583b78c9a
+  data.tar.gz: 36f60d62312f6844efa9ce1b53a86cde380140582e009b5e5bf07d3a4a859e4d762e92a2ebcfe38a668145255f82f72c84504caabb93f25475e895a5166a9969

data/README.md

@@ -24,15 +24,49 @@ require 'datadog/statsd'
 
 # Create a DogStatsD client instance.
 statsd = Datadog::Statsd.new('localhost', 8125)
+...
+# release resources used by the client instance
+statsd.close()
 ```
 Or if you want to connect over Unix Domain Socket:
 ```ruby
 # Connection over Unix Domain Socket
 statsd = Datadog::Statsd.new(socket_path: '/path/to/socket/file')
+...
+# release resources used by the client instance
+statsd.close()
 ```
 
 Find a list of all the available options for your DogStatsD Client in the [DogStatsD-ruby rubydoc](https://www.rubydoc.info/github/DataDog/dogstatsd-ruby/master/Datadog/Statsd) or in the [Datadog public DogStatsD documentation](https://docs.datadoghq.com/developers/dogstatsd/?tab=ruby#client-instantiation-parameters).
 
+### Migrating from v4.x to v5.x
+
+If you are already using DogStatsD-ruby v4.x and you want to migrate to a version v5.x, the major
+change concerning you is the new threading model (please see section Threading model):
+
+In practice, it means two things:
+
+1. Now that the client is buffering metrics before sending them, you have to manually
+call the method `Datadog::Statsd#flush` if you want to force the sending of metrics. Note that the companion thread will automatically flush the buffered metrics if the buffer gets full or when you are closing the instance.
+
+2. You have to make sure you are either:
+
+  * using singletons instances of the DogStatsD client and not allocating one each time you need one, letting the buffering mechanism flush metrics, or,
+  * properly closing your DogStatsD client instance when it is not needed anymore using the method `Datadog::Statsd#close` to release the resources used by the instance and to close the socket
+
+### v5.x Common Pitfalls
+
+Version v5.x of `dogstatsd-ruby` is using a companion thread for preemptive flushing, it brings better performances for application having a high-throughput of statsd metrics, but it comes with new pitfalls:
+
+    * Applications forking after having created the dogstatsd instance: forking a process can't duplicate the existing threads, meaning that one of the processes won't have a companion thread to flush the metrics and will lead to missing metrics.
+    * Applications creating a lot of different instances of the client without closing them: it is important to close the instance to free the thread and the socket it is using or it will lead to thread leaks.
+
+If you are using [Sidekiq](https://github.com/mperham/sidekiq), please make sure to close the client instances that are instantiated. [See this example on using DogStatsD-ruby v5.x with Sidekiq](https://github.com/DataDog/dogstatsd-ruby/blob/master/examples/sidekiq_example.rb).
+
+If you are using [Puma](https://github.com/puma/puma) or [Unicorn](https://yhbt.net/unicorn.git), please make sure to create the instance of DogStatsD in the workers, not in the main process before it forks to create its workers. See [this comment for more details](https://github.com/DataDog/dogstatsd-ruby/issues/179#issuecomment-845570345).
+
+Applications that are in these situations and can't apply these recommendations should pin dogstatsd-ruby v4.x using `gem 'dogstatsd-ruby', '~> 4.0'`. Note that v4.x will continue to be maintained until a future v5.x version can more easily fit these use cases.
+
 ### Origin detection over UDP
 
 Origin detection is a method to detect which pod DogStatsD packets are coming from in order to add the pod's tags to the tag list.
@@ -84,6 +118,58 @@ size should be used, you can set it with the parameter `buffer_max_payload_size`
 statsd = Datadog::Statsd.new('localhost', 8125, buffer_max_payload_size: 4096)
 ```
 
+## Threading model
+
+On versions greater than 5.0, we changed the threading model of the library so that one instance of `Datadog::Statsd` could be shared between threads and so that the writes in the socket are non blocking.
+
+When you instantiate a `Datadog::Statsd`, a companion thread is spawned. This thread will be called the Sender thread, as it is modeled by the [Sender](../lib/datadog/statsd/sender.rb) class.
+
+This thread is stopped when you close the statsd client (`Datadog::Statsd#close`). It also means that allocating a lot of statsd clients without closing them properly when not used anymore
+could lead to a thread leak (even though they will be sleeping, blocked on IO).
+The communication between the current thread is managed through a standard Ruby Queue.
+
+The sender thread has the following logic (Code present in the method `Datadog::Statsd::Sender#send_loop`):
+
+```
+while the sender message queue is not closed do
+  read message from sender message queue
+
+  if message is a Control message to flush
+    flush buffer in connection
+  else if message is a Control message to synchronize
+    synchronize with calling thread
+  else
+    add message to the buffer
+  end
+end while
+```
+
+Most of the time, the sender thread is blocked and sleeping when doing a blocking read from the sender message queue.
+
+We can see that there is 3 different kind of messages:
+
+* a control message to flush the buffer in the connection
+* a control message to synchronize any thread with the sender thread
+* a message to append to the buffer
+
+There is also an implicit message which is closing the queue as it will stop blocking read from the message queue (if happening) and thus, stop the sender thread.
+
+### Usual workflow
+
+You push metrics to the statsd client which writes them quickly to the sender message queue. The sender thread receives those message, buffers them and flushes them to the connection when the buffer limit is reached.
+
+### Flushing
+
+When calling a flush, a specific control message (the `:flush` symbol) is sent to the sender thread. When finding it, it flushes its internal buffer into the connection.
+
+### Rendez-vous
+
+It is possible to ensure a message has been consumed by the sender thread and written to the buffer by simply calling a rendez-vous right after. This is done when you are doing a synchronized flush (calling `Datadog::Statsd#flush` with the `sync: true` option).
+
+This means the current thread is going to sleep and wait for a Queue which is given to the sender thread. When the sender thread reads this queue from its own message queue, it puts a placeholder message in it so that it wakes up the calling thread.
+
+This is useful when closing the application or when checking unit tests.
+
 ## Credits
 
 dogstatsd-ruby is forked from Rein Henrichs [original Statsd

data/lib/datadog/statsd.rb

@@ -320,7 +320,10 @@ module Datadog
     end
 
     # Close the underlying socket
-    def close
+    #
+    # @param [Boolean, true] flush Should we flush the metrics before closing
+    def close(flush: true)
+      flush(sync: true) if flush
       forwarder.close
     end
 

data/lib/datadog/statsd/sender.rb

@@ -10,7 +10,8 @@ module Datadog
       end
 
       def flush(sync: false)
-        raise ArgumentError, 'Start sender first' unless message_queue
+        # don't try to flush if there is no message_queue instantiated
+        return unless message_queue
 
         message_queue.push(:flush)
 

data/lib/datadog/statsd/version.rb

@@ -4,6 +4,6 @@ require_relative 'connection'
 
 module Datadog
   class Statsd
-    VERSION = '5.0.1'
+    VERSION = '5.1.0'
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: dogstatsd-ruby
 version: !ruby/object:Gem::Version
-  version: 5.0.1
+  version: 5.1.0
 platform: ruby
 authors:
 - Rein Henrichs
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-04-09 00:00:00.000000000 Z
+date: 2021-06-17 00:00:00.000000000 Z
 dependencies: []
 description: A Ruby DogStatsd client
 email: code@datadoghq.com
@@ -41,9 +41,9 @@ licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/DataDog/dogstatsd-ruby/issues
-  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.0.1/CHANGELOG.md
-  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.0.1
-  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.0.1
+  changelog_uri: https://github.com/DataDog/dogstatsd-ruby/blob/v5.1.0/CHANGELOG.md
+  documentation_uri: https://www.rubydoc.info/gems/dogstatsd-ruby/5.1.0
+  source_code_uri: https://github.com/DataDog/dogstatsd-ruby/tree/v5.1.0
 post_install_message:
 rdoc_options: []
 require_paths: